a9f64e7d39b67fd55b0cd7bd289b31d954190363
raw

LILPACK -- Package System for Lilush

Overview

A convention for MNEME databases that serve as installable Lua module packages for Lilush. A single .lilpack file carries Lua source code, metadata, and shell integration descriptors.

Convention version

The current convention version is 1.

Module loading

Lilush replaces the standard Lua filesystem-based module loading with a three-stage searcher chain:

  1. Preload -- built-in modules compiled into the binary

  2. LILPACK -- modules from installed .lilpack databases

  3. Script-local -- ./?.lua from the current directory

Standard filesystem searchers (package.path, package.cpath) are disabled entirely. This prevents accidental loading of system Lua modules (e.g. LuaRocks packages) that might clash with Lilush modules.

Manifest keyspace: __lilpack

A conforming LILPACK has a keyspace named __lilpack containing well-known keys that describe the package. The __lilpack keyspace name is reserved.

Required keys

KeyValue typeDescription
versionintegerConvention version (currently 1)
namestringUnique package identifier (e.g. "llm")
pkg_versionstringSemantic version (e.g. "0.1.0")
descriptionstringShort human-readable description
modulesstring (JSON)Array of module names this package provides
signaturebytes (64B)Ed25519 signature over canonical digest
pubkeybytes (32B)Signer's Ed25519 public key

Optional keys

KeyValue typeDescription
authorstringAuthor name or contact
licensestringLicense identifier
min_lilush_versionstringMinimum lilush runtime version required (X.Y.Z)
dependenciesstring (JSON)Array of dependency descriptors
builtinsstring (JSON)Array of builtin family descriptors
modesstring (JSON)Array of shell mode descriptors
clistring (JSON)Array of CLI script command names
themestring (JSON)Theme section builders and/or theme definitions

Modules keyspace: modules

Lua source code is stored in a regular KV keyspace named modules. Each key is a dot-separated module name matching what you pass to require(). The value is the Lua source text.

Example keys for a package named llm:

llm -> source of llm.lua llm.oaic -> source of llm/oaic.lua llm.tools.bash -> source of llm/tools/bash.lua

Source is stored as text (not bytecode) to keep packages portable and debuggable. Stack traces show @lilpack:<pack>:<module> chunk names.

CLI scripts keyspace: cli

Standalone Lua scripts intended for installation under ~/.local/bin/ are stored in a separate cli keyspace. Each key is a command name, each value is the Lua source text.

Example keys for a package named mytools:

mytool -> source of cli/mytool.lua myutil -> source of cli/myutil.lua

Scripts in the cli keyspace are not loadable via require(). They are extracted as files during lilpack install.

Dependencies

[
    {"name": "llm", "version": ">=0.1.0"}
]

Each entry names another package and a version constraint. The version constraint is a single clause using one of these operators (a bare version is treated as >=):

ConstraintMatches
0.1.0 / >=0.1.00.1.0 and anything newer
>0.1.0strictly newer than 0.1.0
<=0.1.0 / <0.1.0up to / below 0.1.0
=0.1.0exactly 0.1.0
~>0.1>=0.1.0 <1.0.0 (pessimistic, pins major)
~>0.1.2>=0.1.2 <0.2.0 (pessimistic, pins minor)
(empty / omitted)any version

Versions are X.Y.Z; a missing minor/patch defaults to 0, a leading v is ignored, and a -prerelease suffix sorts below the same version without one.

At install time dependencies are enforced: lilpack install resolves the full graph, downloads each missing or unsatisfied dependency from the repo, verifies its signature, checks the constraint, and installs dependencies before the packages that need them. Resolution is fully staged into temp files first, so a failure (missing dependency, unsatisfiable constraint, bad signature, dependency cycle) leaves the packages directory untouched.

At load time dependencies are only diagnosed: if an installed package declares a dependency that is missing or whose installed version does not satisfy the constraint, a warning is printed to stderr but loading proceeds (a genuinely missing module still fails loudly when require()d).

Because the repo serves only the latest version of each package (see below), a constraint effectively means "is the latest published version new enough"; if even the latest does not satisfy it, install fails with a clear error.

Minimum lilush version

A package may declare the oldest lilush runtime it supports via the optional min_lilush_version manifest key (X.Y.Z). The running runtime version is exposed to Lua as the global LILUSH_VERSION (set by the binary at startup).

Enforcement mirrors Dependencies:

The constraint uses the same >= semantics as a bare dependency version.

Shell integration

Builtin descriptors

A package can provide shell builtins by declaring them in the manifest:

[{"module": "shell.builtins.k8s", "commands": ["ktl"]}]

The module field names the module within the package that returns a builtin family table (same format as built-in families like shell.builtins.fs). The module is loaded via require() during shell startup.

Mode descriptors

A package can provide shell modes:

[{
    "name": "agent",
    "path": "agent.mode.agent",
    "history": true,
    "prompt": "agent.mode.agent.prompt",
    "completion": {
        "path": "agent.completion.slash",
        "sources": ["agent.completion.source.slash"]
    }
}]

Mode descriptors do NOT include shortcut assignments. Users map F-keys to package modes via ~/.config/lilush/modes.json:

{
    "F4": "agent",
    "F5": "k8s-dashboard"
}

Each key is an F-key, each value is a LILPACK name whose first mode descriptor is activated on that key.

CLI script descriptors

A package can provide standalone CLI commands that are installed as executable scripts in ~/.local/bin/:

["mytool", "myutil"]

Each entry is a command name. The corresponding Lua source must exist in the cli/ subdirectory of the source tree (e.g. cli/mytool.lua). On install, each script is written to ~/.local/bin/<command> with a #!/usr/bin/env lilush shebang prepended (if not already present) and made executable.

CLI scripts are independent files -- they do not participate in module loading. Scripts that need LILPACK modules must initialise the searcher chain themselves:

local lilpack = require("lilpack")
lilpack.init()
local mylib = require("mypackage.utils")

Theme descriptors

A package can extend the theme system by providing section builders (styles for a specific mode) and/or complete theme definitions (palette + overrides).

The theme manifest field is an object with two optional sub-fields:

Section builders — provide styles for a named theme section:

{
    "theme": {
        "sections": [
            {"name": "agent", "builder": "agent.theme"}
        ]
    }
}

The builder field names a module that returns a function. The function receives the active palette table and returns an RSS table for the section. It is called each time the theme changes to rebuild section styles from the new palette.

Theme definitions — provide a complete selectable theme:

{
    "theme": {
        "definition": {
            "name": "solarized",
            "module": "solarized.theme"
        }
    }
}

The module field names a module that returns a table in the same format as bundled catalog entries: { palette = {...}, shell = {...}, markdown = {...} }.

A single package can declare both sections and definition if needed.

See THEMES for the full theme architecture and API.

LILPACK management

Packages are managed with lilpack install/lilpack remove builtins. Packages are installed to ~/.local/share/lilush/packages/<name>.lilpack. Packages declaring CLI scripts also install executable files under ~/.local/bin/. Ensure ~/.local/bin is in your PATH.

Official Lilush Packages repo

By default lilpack install will try to install remote packages from the official Lilush Packages HTTP Repo. This can be overriden with the LILPACK_REPO_URL environment variable.

Packages in the official HTTP repo are built from the lilpacks repositories listed at git.lilush.link.

Repo layout and serving

The repo is a flat directory of static files served over HTTP. Each package contributes exactly one file, <name>.lilpack, holding its latest published version. lilpack install <name> is a plain GET <repo>/<name>.lilpack. There is no index file and no per-version history: publishing a new version overwrites the file. Dependency version constraints are therefore satisfied against whatever is current — see Dependencies.

Publishing (tag-driven)

Publishing is tied to git tags, not to commits. The published version is taken from the tag, overriding the version in lilpack.json (which then serves only as a dev/local default). Tags are bare X.Y.Z (no v prefix).

lilpack pack supports two options for this:

  • --version <X.Y.Z> — set the packed pkg_version (and thus the signed digest), overriding lilpack.json.

  • --publish <dir> — after packing+signing, write the artifact to <dir>/<name>.lilpack (the name comes from the manifest), using the same atomic temp+rename as a normal pack.

A package repo couples tags to releases with a git reference-transaction hook (git has no native post-tag hook). On creation of a bare semver tag it packs and publishes; non-semver tags and branch updates are ignored. The hook needs line parsing and a loop, which LSH does not provide, so it is a lilush Lua script (lilush runs any file passed to it as Lua, with the hook arguments in the global arg table):

#!/usr/bin/env lilush
-- .githooks/reference-transaction  (git config core.hooksPath .githooks)
-- arg[1] is the transaction state; ref updates arrive on stdin as
-- "<old> <new> <refname>" lines.
if arg[1] ~= "committed" then
    os.exit(0)
end
local site = os.getenv("LILPACK_SITE_DIR")
if not site then
    os.exit(0)
end
local lilpack = require("shell.builtins.lilpack")
for line in io.lines() do
    local _, _, ref = line:match("^(%S+)%s+(%S+)%s+(%S+)$")
    local tag = ref and ref:match("^refs/tags/(%d+%.%d+%.%d+%S*)$")
    if tag then
        lilpack["lilpack"].func("lilpack", { "pack", ".", "--version", tag, "--publish", site })
    end
end

LILPACK_SITE_DIR is the document root served at the repo URL, and LILPACK_SIGN_KEY must be set for the hook's signing step.

Source manifest: lilpack.json

Each source directory contains a lilpack.json at its root:

{
    "name": "llm",
    "version": "0.1.0",
    "description": "LLM client factory with backends and tools",
    "author": "Vladimir Zorin",
    "license": "LicenseRef-OWL-1.0-or-later",
    "min_lilush_version": "0.8.5",
    "dependencies": [],
    "builtins": [],
    "modes": [],
    "cli": []
}

The directory layout maps to module names. The top-level cli/, build/ and tests/ subdirectories are reserved and excluded from module scanning. cli/ holds standalone CLI scripts; build/ holds the optional direct-build descriptor and entrypoint used to compile the pack into a custom static binary (see DEVELOPMENT); tests/ holds the pack-local test suite (run with lilush tests/run.lua). Only the top-level directories are excluded — nested directories of those names are scanned normally:

llm/
  lilpack.json       <- manifest (not packed)
  llm.lua            <- module "llm"
  llm/
    oaic.lua         <- module "llm.oaic"
    tools/
      bash.lua       <- module "llm.tools.bash"
  cli/
    llmq.lua         <- CLI script "llmq" (not a module)
  build/             <- direct-build descriptor + entrypoint (not modules)
    llm.lua          <- build descriptor
    start.lua        <- entrypoint
  tests/             <- pack-local test suite (not modules)

Digital signatures

All packages must be signed with Ed25519 SSH keys. Unsigned packages are rejected at every stage: packing, installation, and load time.

Lilush ships with the author's public key built in. Set LILPACK_PUB_KEY to verify against a different trusted key. If a package fails verification, it is rejected and none of its modules, builtins, modes, or CLI scripts are made available.

Environment variables

VariableUsed byValue
LILPACK_SIGN_KEYlilpack packRequired. Path to unencrypted OpenSSH ed25519 private key
LILPACK_PUB_KEYlilpack install, lilpack verifyPath to SSH .pub file (overrides built-in key)

Signing (lilpack pack)

lilpack pack requires LILPACK_SIGN_KEY to be set:

  1. Reads the OpenSSH ed25519 private key (must be unencrypted)

  2. Computes a canonical SHA-256 digest over all package content

  3. Signs the digest with Ed25519

  4. Stores the 64-byte signature and 32-byte pubkey in __lilpack

The embedded pubkey must exactly match the trusted verification key. Packages with a mismatched embedded key are rejected.

Verification (lilpack install)

lilpack install always verifies signatures before installing:

  • Verification passes -- installed

  • Verification fails -- rejected

  • No trusted key available -- rejected

Verification at load time (lilpack.init)

Every package is verified when the package system initialises. The built-in public key is used by default; LILPACK_PUB_KEY overrides it. Verification requires both:

  • the embedded pubkey matches the trusted key

  • the signature verifies against that key

  • Verification passes -- loaded

  • Verification fails -- skipped with error

  • No trusted key available -- no packages loaded

Canonical digest algorithm

The digest is a SHA-256 hash over a deterministic serialization of all package content (excluding signature and pubkey):

  1. Manifest fields in fixed order: version, name, pkg_version, description, modules, author, license, dependencies, builtins, modes, cli, theme, min_lilush_version. Each present field contributes key=tostring(value)\0.

  2. Module sources in sorted name order. Each module contributes module:name=source\0.

  3. CLI script sources in sorted name order. Each script contributes cli:name=source\0.

  4. All parts are concatenated and hashed with SHA-256.

Key fingerprints

Public key fingerprints are the first 16 hex characters of SHA-256 over the raw 32-byte public key. Displayed by lilpack info and lilpack verify.

lilpack builtin

SubcommandDescription
lilpack listList installed packages
lilpack info <name>Show detailed package info (includes signature status)
lilpack modules [-n name]List registered modules
lilpack pack <dir> [-o path] [--version v] [--publish dir]Create .lilpack from source directory
lilpack install <target>Install a .lilpack file or repo package (with deps)
lilpack remove <name>Remove an installed package
lilpack verify <path> [-k key]Verify package signature

lilpack install <target> accepts a local file path or, if target has no /, a package name to fetch from the repo. Either way its declared dependencies are resolved and installed first.

Examples

# Pack and sign a source directory:
setenv LILPACK_SIGN_KEY ${HOME}/.ssh/lilpack_sign.ed
lilpack pack /path/to/lilpack/source

# Pack with a tag-derived version, publishing to the repo doc root:
lilpack pack . --version 0.3.0 --publish ${LILPACK_SITE_DIR}

# Install a package from the repo (pulls dependencies automatically):
lilpack install llm

# Install a local file (verified against the built-in key):
lilpack install llm-0.1.0.lilpack

# Install with a custom trusted key:
setenv LILPACK_PUB_KEY ${HOME}/.ssh/lilpack_sign.pub
lilpack install llm-0.1.0.lilpack

# Verify a package without installing:
lilpack verify llm-0.1.0.lilpack -k ~/.ssh/lilpack_sign.pub

Lua API

local lilpack = require("lilpack")

-- Initialize the package system (called automatically at startup)
local extensions = lilpack.init()
-- extensions.builtins -- array of builtin descriptors
-- extensions.modes    -- table: mode_name -> mode config

-- Query the registry
local pkgs = lilpack.packages()       -- pack_name -> {db_path, manifest}
local mods = lilpack.modules()        -- module_name -> {db_path, pack_name}
local exts = lilpack.extensions()     -- {builtins = ..., modes = ..., cli = ..., theme = ...}
local manifest = lilpack.read_manifest(db_path)  -- read and validate manifest

-- Constants
lilpack.CONVENTION_VERSION  -- integer (current: 1)
lilpack.PACKAGES_DIR        -- packages directory path

-- Signature utilities
local digest = lilpack.compute_digest(db)               -- SHA-256 canonical digest from open MNEME db
local pubkey, err = lilpack.load_trusted_pubkey(path)    -- load trusted pubkey from file, env, or built-in
local fp = lilpack.pubkey_fingerprint(pubkey)            -- first 16 hex chars of SHA-256(pubkey)
local ok, err, pubkey = lilpack.verify_signature(path, trusted_pk)  -- verify .lilpack file