# LILPACK -- Package System for Lilush

## Overview

A convention for MNEME databases that serve as installable Lua module
packages for Lilush. A single `.lilpack` file carries Lua source code,
metadata, and shell integration descriptors.

## Convention version

The current convention version is **1**.

## Module loading

Lilush replaces the standard Lua filesystem-based module loading with
a three-stage searcher chain:

1. **Preload** -- built-in modules compiled into the binary
2. **LILPACK** -- modules from installed `.lilpack` databases
3. **Script-local** -- `./?.lua` from the current directory

Standard filesystem searchers (`package.path`, `package.cpath`) are
disabled entirely. This prevents accidental loading of system Lua
modules (e.g. LuaRocks packages) that might clash with Lilush modules.

## Manifest keyspace: `__lilpack`

A conforming LILPACK has a keyspace named `__lilpack` containing
well-known keys that describe the package. The `__lilpack` keyspace
name is reserved.

### Required keys

| Key | Value type | Description |
|-----|------------|-------------|
| `version` | integer | Convention version (currently `1`) |
| `name` | string | Unique package identifier (e.g. `"llm"`) |
| `pkg_version` | string | Semantic version (e.g. `"0.1.0"`) |
| `description` | string | Short human-readable description |
| `modules` | string (JSON) | Array of module names this package provides |
| `signature` | bytes (64B) | Ed25519 signature over canonical digest |
| `pubkey` | bytes (32B) | Signer's Ed25519 public key |

### Optional keys

| Key | Value type | Description |
|-----|------------|-------------|
| `author` | string | Author name or contact |
| `license` | string | License identifier |
| `min_lilush_version` | string | Minimum lilush runtime version required (`X.Y.Z`) |
| `dependencies` | string (JSON) | Array of dependency descriptors |
| `builtins` | string (JSON) | Array of builtin family descriptors |
| `modes` | string (JSON) | Array of shell mode descriptors |
| `cli` | string (JSON) | Array of CLI script command names |
| `theme` | string (JSON) | Theme section builders and/or theme definitions |

## Modules keyspace: `modules`

Lua source code is stored in a regular KV keyspace named `modules`.
Each key is a dot-separated module name matching what you pass to
`require()`. The value is the Lua source text.

Example keys for a package named `llm`:

    llm             -> source of llm.lua
    llm.oaic        -> source of llm/oaic.lua
    llm.tools.bash  -> source of llm/tools/bash.lua

Source is stored as text (not bytecode) to keep packages portable and
debuggable. Stack traces show `@lilpack:<pack>:<module>` chunk names.

## CLI scripts keyspace: `cli`

Standalone Lua scripts intended for installation under `~/.local/bin/`
are stored in a separate `cli` keyspace. Each key is a command name,
each value is the Lua source text.

Example keys for a package named `mytools`:

    mytool   -> source of cli/mytool.lua
    myutil   -> source of cli/myutil.lua

Scripts in the `cli` keyspace are not loadable via `require()`. They
are extracted as files during `lilpack install`.

## Dependencies

```json
[
    {"name": "llm", "version": ">=0.1.0"}
]
```

Each entry names another package and a version constraint. The `version`
constraint is a single clause using one of these operators (a bare
version is treated as `>=`):

| Constraint | Matches |
|------------|---------|
| `0.1.0` / `>=0.1.0` | `0.1.0` and anything newer |
| `>0.1.0` | strictly newer than `0.1.0` |
| `<=0.1.0` / `<0.1.0` | up to / below `0.1.0` |
| `=0.1.0` | exactly `0.1.0` |
| `~>0.1` | `>=0.1.0 <1.0.0` (pessimistic, pins major) |
| `~>0.1.2` | `>=0.1.2 <0.2.0` (pessimistic, pins minor) |
| (empty / omitted) | any version |

Versions are `X.Y.Z`; a missing minor/patch defaults to `0`, a leading
`v` is ignored, and a `-prerelease` suffix sorts below the same version
without one.

**At install time** dependencies are *enforced*: `lilpack install`
resolves the full graph, downloads each missing or unsatisfied
dependency from the repo, verifies its signature, checks the constraint,
and installs dependencies before the packages that need them. Resolution
is fully staged into temp files first, so a failure (missing dependency,
unsatisfiable constraint, bad signature, dependency cycle) leaves the
packages directory untouched.

**At load time** dependencies are only *diagnosed*: if an installed
package declares a dependency that is missing or whose installed version
does not satisfy the constraint, a warning is printed to stderr but
loading proceeds (a genuinely missing module still fails loudly when
`require()`d).

Because the repo serves only the latest version of each package (see
below), a constraint effectively means "is the latest published version
new enough"; if even the latest does not satisfy it, install fails with a
clear error.

## Minimum lilush version

A package may declare the oldest lilush runtime it supports via the
optional `min_lilush_version` manifest key (`X.Y.Z`). The running
runtime version is exposed to Lua as the global `LILUSH_VERSION` (set by
the binary at startup).

Enforcement mirrors [Dependencies](#dependencies):

- **At install time** it is *enforced*: `lilpack install` rejects a
  package (or any of its dependencies) whose `min_lilush_version` is
  newer than the running lilush, with a `requires lilush >= ...` error.
- **At load time** it is only *diagnosed*: a package requiring a newer
  lilush than the one running prints a warning to stderr but still loads.

The constraint uses the same `>=` semantics as a bare dependency version.

## Shell integration

### Builtin descriptors

A package can provide shell builtins by declaring them in the manifest:

```json
[{"module": "shell.builtins.k8s", "commands": ["ktl"]}]
```

The `module` field names the module within the package that returns a
builtin family table (same format as built-in families like
`shell.builtins.fs`). The module is loaded via `require()` during
shell startup.

### Mode descriptors

A package can provide shell modes:

```json
[{
    "name": "agent",
    "path": "agent.mode.agent",
    "history": true,
    "prompt": "agent.mode.agent.prompt",
    "completion": {
        "path": "agent.completion.slash",
        "sources": ["agent.completion.source.slash"]
    }
}]
```

Mode descriptors do NOT include shortcut assignments. Users map F-keys
to package modes via `~/.config/lilush/modes.json`:

```json
{
    "F4": "agent",
    "F5": "k8s-dashboard"
}
```

Each key is an F-key, each value is a LILPACK name whose first mode
descriptor is activated on that key.

### CLI script descriptors

A package can provide standalone CLI commands that are installed as
executable scripts in `~/.local/bin/`:

```json
["mytool", "myutil"]
```

Each entry is a command name. The corresponding Lua source must exist in
the `cli/` subdirectory of the source tree (e.g. `cli/mytool.lua`). On
install, each script is written to `~/.local/bin/<command>` with a
`#!/usr/bin/env lilush` shebang prepended (if not already present) and
made executable.

CLI scripts are independent files -- they do not participate in module
loading. Scripts that need LILPACK modules must initialise the searcher
chain themselves:

```lua
local lilpack = require("lilpack")
lilpack.init()
local mylib = require("mypackage.utils")
```

### Theme descriptors

A package can extend the theme system by providing section builders (styles
for a specific mode) and/or complete theme definitions (palette + overrides).

The `theme` manifest field is an object with two optional sub-fields:

**Section builders** — provide styles for a named theme section:

```json
{
    "theme": {
        "sections": [
            {"name": "agent", "builder": "agent.theme"}
        ]
    }
}
```

The `builder` field names a module that returns a function. The function
receives the active palette table and returns an RSS table for the section.
It is called each time the theme changes to rebuild section styles from the
new palette.

**Theme definitions** — provide a complete selectable theme:

```json
{
    "theme": {
        "definition": {
            "name": "solarized",
            "module": "solarized.theme"
        }
    }
}
```

The `module` field names a module that returns a table in the same format as
bundled catalog entries: `{ palette = {...}, shell = {...}, markdown = {...} }`.

A single package can declare both `sections` and `definition` if needed.

See [THEMES](THEMES.md) for the full theme architecture and API.

## LILPACK management

Packages are managed with `lilpack install`/`lilpack remove` builtins.
Packages are installed to `~/.local/share/lilush/packages/<name>.lilpack`.
Packages declaring CLI scripts also install executable files under
`~/.local/bin/`. Ensure `~/.local/bin` is in your `PATH`.

### Official Lilush Packages repo

By default `lilpack install` will try to install
remote packages from the official [Lilush Packages HTTP Repo](https://packages.lilush.link).
This can be overriden with the `LILPACK_REPO_URL` environment variable.

Packages in the official HTTP repo are built from the `lilpacks` repositories
listed at [git.lilush.link](https://git.lilush.link/).

### Repo layout and serving

The repo is a flat directory of static files served over HTTP. Each
package contributes exactly one file, `<name>.lilpack`, holding its
**latest** published version. `lilpack install <name>` is a plain
`GET <repo>/<name>.lilpack`. There is no index file and no per-version
history: publishing a new version overwrites the file. Dependency version
constraints are therefore satisfied against whatever is current — see
[Dependencies](#dependencies).

### Publishing (tag-driven)

Publishing is tied to git tags, not to commits. The published version is
taken **from the tag**, overriding the `version` in `lilpack.json` (which
then serves only as a dev/local default). Tags are bare `X.Y.Z` (no `v`
prefix).

`lilpack pack` supports two options for this:

- `--version <X.Y.Z>` — set the packed `pkg_version` (and thus the signed
  digest), overriding `lilpack.json`.
- `--publish <dir>` — after packing+signing, write the artifact to
  `<dir>/<name>.lilpack` (the name comes from the manifest), using the
  same atomic temp+rename as a normal pack.

A package repo couples tags to releases with a git `reference-transaction`
hook (git has no native post-tag hook). On creation of a bare semver tag
it packs and publishes; non-semver tags and branch updates are ignored.
The hook needs line parsing and a loop, which LSH does not provide, so it
is a lilush **Lua** script (lilush runs any file passed to it as Lua, with
the hook arguments in the global `arg` table):

```lua
#!/usr/bin/env lilush
-- .githooks/reference-transaction  (git config core.hooksPath .githooks)
-- arg[1] is the transaction state; ref updates arrive on stdin as
-- "<old> <new> <refname>" lines.
if arg[1] ~= "committed" then
    os.exit(0)
end
local site = os.getenv("LILPACK_SITE_DIR")
if not site then
    os.exit(0)
end
local lilpack = require("shell.builtins.lilpack")
for line in io.lines() do
    local _, _, ref = line:match("^(%S+)%s+(%S+)%s+(%S+)$")
    local tag = ref and ref:match("^refs/tags/(%d+%.%d+%.%d+%S*)$")
    if tag then
        lilpack["lilpack"].func("lilpack", { "pack", ".", "--version", tag, "--publish", site })
    end
end
```

`LILPACK_SITE_DIR` is the document root served at the repo URL, and
`LILPACK_SIGN_KEY` must be set for the hook's signing step. 

## Source manifest: `lilpack.json`

Each source directory contains a `lilpack.json` at its root:

```json
{
    "name": "llm",
    "version": "0.1.0",
    "description": "LLM client factory with backends and tools",
    "author": "Vladimir Zorin",
    "license": "LicenseRef-OWL-1.0-or-later",
    "min_lilush_version": "0.8.5",
    "dependencies": [],
    "builtins": [],
    "modes": [],
    "cli": []
}
```

The directory layout maps to module names. The top-level `cli/`,
`build/` and `tests/` subdirectories are reserved and excluded from
module scanning. `cli/` holds standalone CLI scripts; `build/` holds the
optional direct-build descriptor and entrypoint used to compile the pack
into a custom static binary (see [DEVELOPMENT](DEVELOPMENT.md)); `tests/`
holds the pack-local test suite (run with `lilush tests/run.lua`). Only
the top-level directories are excluded — nested directories of those
names are scanned normally:

```
llm/
  lilpack.json       <- manifest (not packed)
  llm.lua            <- module "llm"
  llm/
    oaic.lua         <- module "llm.oaic"
    tools/
      bash.lua       <- module "llm.tools.bash"
  cli/
    llmq.lua         <- CLI script "llmq" (not a module)
  build/             <- direct-build descriptor + entrypoint (not modules)
    llm.lua          <- build descriptor
    start.lua        <- entrypoint
  tests/             <- pack-local test suite (not modules)
```

## Digital signatures

All packages must be signed with Ed25519 SSH keys. Unsigned packages
are rejected at every stage: packing, installation, and load time.

Lilush ships with the author's public
[key](https://deviant.guru/keys) built in. Set `LILPACK_PUB_KEY` to
verify against a different trusted key. If a package fails
verification, it is rejected and none of its modules, builtins, modes,
or CLI scripts are made available.

### Environment variables

| Variable | Used by | Value |
|----------|---------|-------|
| `LILPACK_SIGN_KEY` | `lilpack pack` | **Required.** Path to unencrypted OpenSSH ed25519 private key |
| `LILPACK_PUB_KEY` | `lilpack install`, `lilpack verify` | Path to SSH `.pub` file (overrides built-in key) |

### Signing (`lilpack pack`)

`lilpack pack` requires `LILPACK_SIGN_KEY` to be set:

1. Reads the OpenSSH ed25519 private key (must be unencrypted)
2. Computes a canonical SHA-256 digest over all package content
3. Signs the digest with Ed25519
4. Stores the 64-byte `signature` and 32-byte `pubkey` in `__lilpack`

The embedded `pubkey` must exactly match the trusted verification key.
Packages with a mismatched embedded key are rejected.

### Verification (`lilpack install`)

`lilpack install` always verifies signatures before installing:

- **Verification passes** -- installed
- **Verification fails** -- rejected
- **No trusted key available** -- rejected

### Verification at load time (`lilpack.init`)

Every package is verified when the package system initialises. The
built-in public key is used by default; `LILPACK_PUB_KEY` overrides it.
Verification requires both:

- the embedded `pubkey` matches the trusted key
- the signature verifies against that key

- **Verification passes** -- loaded
- **Verification fails** -- skipped with error
- **No trusted key available** -- no packages loaded

### Canonical digest algorithm

The digest is a SHA-256 hash over a deterministic serialization of all
package content (excluding `signature` and `pubkey`):

1. Manifest fields in fixed order: `version`, `name`, `pkg_version`,
   `description`, `modules`, `author`, `license`, `dependencies`,
   `builtins`, `modes`, `cli`, `theme`, `min_lilush_version`. Each present
   field contributes `key=tostring(value)\0`.
2. Module sources in sorted name order. Each module contributes
   `module:name=source\0`.
3. CLI script sources in sorted name order. Each script contributes
   `cli:name=source\0`.
4. All parts are concatenated and hashed with SHA-256.

### Key fingerprints

Public key fingerprints are the first 16 hex characters of SHA-256 over
the raw 32-byte public key. Displayed by `lilpack info` and
`lilpack verify`.

## `lilpack` builtin

| Subcommand | Description |
|------------|-------------|
| `lilpack list` | List installed packages |
| `lilpack info <name>` | Show detailed package info (includes signature status) |
| `lilpack modules [-n name]` | List registered modules |
| `lilpack pack <dir> [-o path] [--version v] [--publish dir]` | Create `.lilpack` from source directory |
| `lilpack install <target>` | Install a `.lilpack` file or repo package (with deps) |
| `lilpack remove <name>` | Remove an installed package |
| `lilpack verify <path> [-k key]` | Verify package signature |

`lilpack install <target>` accepts a local file path or, if `target` has
no `/`, a package name to fetch from the repo. Either way its declared
dependencies are resolved and installed first.

### Examples

```lsh
# Pack and sign a source directory:
setenv LILPACK_SIGN_KEY ${HOME}/.ssh/lilpack_sign.ed
lilpack pack /path/to/lilpack/source

# Pack with a tag-derived version, publishing to the repo doc root:
lilpack pack . --version 0.3.0 --publish ${LILPACK_SITE_DIR}

# Install a package from the repo (pulls dependencies automatically):
lilpack install llm

# Install a local file (verified against the built-in key):
lilpack install llm-0.1.0.lilpack

# Install with a custom trusted key:
setenv LILPACK_PUB_KEY ${HOME}/.ssh/lilpack_sign.pub
lilpack install llm-0.1.0.lilpack

# Verify a package without installing:
lilpack verify llm-0.1.0.lilpack -k ~/.ssh/lilpack_sign.pub
```

## Lua API

```lua
local lilpack = require("lilpack")

-- Initialize the package system (called automatically at startup)
local extensions = lilpack.init()
-- extensions.builtins -- array of builtin descriptors
-- extensions.modes    -- table: mode_name -> mode config

-- Query the registry
local pkgs = lilpack.packages()       -- pack_name -> {db_path, manifest}
local mods = lilpack.modules()        -- module_name -> {db_path, pack_name}
local exts = lilpack.extensions()     -- {builtins = ..., modes = ..., cli = ..., theme = ...}
local manifest = lilpack.read_manifest(db_path)  -- read and validate manifest

-- Constants
lilpack.CONVENTION_VERSION  -- integer (current: 1)
lilpack.PACKAGES_DIR        -- packages directory path

-- Signature utilities
local digest = lilpack.compute_digest(db)               -- SHA-256 canonical digest from open MNEME db
local pubkey, err = lilpack.load_trusted_pubkey(path)    -- load trusted pubkey from file, env, or built-in
local fp = lilpack.pubkey_fingerprint(pubkey)            -- first 16 hex chars of SHA-256(pubkey)
local ok, err, pubkey = lilpack.verify_signature(path, trusted_pk)  -- verify .lilpack file
```