# Completion

## Overview

This document specifies the completion engine architecture, provider
and source interfaces, and integration with the input object.

The completion system provides context-aware candidate suggestions
for interactive input.  It is built around three layers:

- **Engine** (`term.input.completion`) — candidate management,
  source registry, provider dispatch
- **Provider** — domain-specific search logic (e.g. shell commands,
  Lua symbols); injected into the engine at construction
- **Sources** — individual data backends (filesystem, binaries,
  environment) queried by the provider

Candidates come from the provider's `search()` (e.g. matching
filenames, commands) and are held in a single list alongside their
per-candidate metadata.

Rendering is handled by a pluggable **renderer** attached to the
input object.  The default renderer draws inline ghost text.

## Files

| Path | Role |
|------|------|
| `src/term/term/input/completion.lua` | Core engine |
| `src/term/term/input/renderer.lua` | Renderer interface and inline ghost renderer |
| `src/term/term/input.lua` | Input object integration |
| `src/shell/shell/completion/shell.lua` | Shell mode provider |
| `src/shell/shell/completion/lua.lua` | Lua mode provider |
| `src/shell/shell/completion/wiki.lua` | Wiki mode provider |
| `src/shell/shell/completion/utils.lua` | Shared helpers (`match_prefix`, `match_prefix_keys`) |
| `src/shell/shell/completion/source/` | Completion sources |
| `src/shell/shell/completion/source/learned.lua` | Learned subcommand source (MNEME-backed) |

## Engine API

### Construction

```lua
local completion = require("term.input.completion")
local theme_mod = require("theme")

local comp, err = completion.new({
    path = "shell.completion.shell",
    tss = theme_mod.subscribe("shell", "completion"),
    sources = {
        "shell.completion.source.bin",
        "shell.completion.source.builtins",
        "shell.completion.source.fs",
    },
    -- additional config fields are passed through as comp.cfg
})
```

The optional `tss` field is a theme-subscribe thunk (a closure returned
by `theme.subscribe()`).  When provided, the engine's default `get()`
uses it for metadata-driven styling — providers no longer need to
implement their own `get()` method.

The `path` module is loaded and its exported methods are injected
directly onto the completion object.  Each source module path is
loaded and instantiated via its `new()` constructor.

### Candidate methods

| Method | Returns | Description |
|--------|---------|-------------|
| `search(input, history, cursor_pos?)` | `bool` | Provider search; populates candidates via `provide()` + `set_meta()` |
| `available()` | `bool` | Whether candidates exist; auto-selects first |
| `count()` | `num` | Number of candidates |
| `get(promoted?)` | `str` | Current candidate; raw when `promoted=true`, styled otherwise |
| `chosen_index()` | `num` | Index of selected candidate (1-based, 0 = none) |
| `set_chosen_index(idx)` | | Set selected index (clamped to valid range) |
| `meta_at(idx)` | `tbl?` | Metadata for candidate at index |
| `common_prefix()` | `str?` | Longest common prefix across all candidates |
| `provide(candidates)` | | Replace candidate list |
| `set_meta(metadata)` | | Replace metadata table |
| `flush()` | | Clear all candidates |

### Source management

| Method | Description |
|--------|-------------|
| `source(name)` | Get registered source by name |
| `update()` | Call `update()` on all sources |
| `register_source(name, src)` | Add a source at runtime |
| `unregister_source(name)` | Remove a source; calls `close()` if available |
| `close()` | Call `close()` on all sources |

### Promotion

| Method | Returns | Description |
|--------|---------|-------------|
| `promote(candidate, metadata, line)` | `{line=, action=}` | Provider-defined; apply candidate to line |
| `should_promote_full()` | `bool` | Provider-defined; skip common-prefix narrowing? |
| `should_auto_promote()` | `bool` | Provider-defined; auto-promote on ENTER? |

When no `promote` is provided by the provider, the input object
falls back to simple append: `line = line .. candidate`.

## Provider interface

A provider is a Lua module that returns a table of methods.  These
are injected onto the completion object at construction time, so
inside all provider methods `self` refers to the completion instance.

Providers that need to keep state between calls should use the
`self.__provider` table rather than `self.__state` (which is
reserved for engine internals like candidates and sources).

### Required

```lua
-- Populate candidates for the given input.
-- Call self:provide() and self:set_meta() inside.
-- cursor_pos is passed when the input has eol_only = false.
search = function(self, input, history, cursor_pos)
    local candidates = self:source("fs"):search(input)
    self:provide(candidates)
    self:set_meta(metadata)
    return self:available()
end
```

### Optional

```lua
-- Custom styling for the current candidate.
-- Return raw text when promoted=true, styled text otherwise.
-- Not needed when config.tss is set — the engine's default get()
-- uses the tss thunk with the candidate's source metadata.
get = function(self, promoted)
    local l = self.__state.layer
    local variant = l.candidates[l.chosen]
    if promoted then return variant end
    return my_tss:apply(l.meta[l.chosen].source, variant).text
end

-- Apply a candidate to the current line.
-- Return {line = new_line, action = "execute" | nil}.
promote = function(self, candidate, metadata, line)
    return { line = line .. candidate }
end

-- Whether the current candidate must skip common-prefix narrowing.
should_promote_full = function(self)
    return false
end

-- Whether ENTER should auto-promote the current candidate.
should_auto_promote = function(self)
    return false
end
```

### Existing providers

**Shell** (`shell.completion.shell`) — command, path, env, history
completion with learned subcommands.

When the input has arguments, a dispatch table routes known
commands (`z`, `x`, `zx`, `cd`, `setenv`, `unsetenv`) to
dedicated handler functions.  All other commands fall through to
the generic chain: dedicated builtin handlers (`cmds` source),
learned subcommands (`learned` source), then filesystem fallback.
When a builtin handler or learned source returns empty candidates,
the engine falls through to filesystem completion instead of
returning nothing.

Learned subcommands are recorded after each successful command
execution (exit code 0).  Recording is **chain-aware**: the full
positional subcommand chain (up to `MAX_CHAIN_DEPTH = 3` levels) is
captured, so `git remote add` learns `remote` under `git`, `add`
under `git → remote`, and so on.  Each level is a sorted set in the
`completions` keyspace of `shell.mneme`, keyed by the binary name plus
preceding subcommands joined by a `\x1f` separator (depth-0 is just the
binary name).  The chain stops at the first flag, path-like token, or
non-subcommand-shaped operand; leading global flags are skipped.
Commands that primarily operate on file paths (`rm`, `ls`, `cp`, `mv`,
`stat`, `rsync`) are excluded entirely, and path-like arguments
(containing `/`, starting with `.` or `~`, having a file extension, or
resolving to an existing filesystem entry) are never recorded as
subcommands.

Ranking is **frecency**, not raw frequency: each use increments the
member's score by `exp((now - T0) / τ)` (τ = 30 days), an ever-growing
weight so recently-used subcommands sort ahead of stale high-frequency
ones.  Reverse iteration over the sorted set yields that order
directly.

**Flags** are learned too: flag tokens (`-m`, `--amend`, with any
`=value` stripped, never the value itself) are stored in a separate
`flag_completions` keyspace, keyed by the same positional context.
When the partial token starts with `-`, the learned source suggests
flags for the current context, falling back to shallower contexts down
to the bare command.

Implements `promote` with `replace_prompt` / `exec_on_prom` /
`trim_promotion` / `reduce_spaces` metadata.

**Lua** (`shell.completion.lua`) — keyword and symbol completion
for Lua mode.

## Source interface

Each source is a module that exports `new()`:

```lua

local new = function(config)
    return {
        search = function(self, query, ...)
            -- Return array of candidate strings
            return { "match_a", "match_b" }
        end,
        update = function(self)
            -- Optional: refresh cached data
        end,
        close = function(self)
            -- Optional: release resources
        end,
    }
end

return { new = new }
```

### Source utilities

The `shell.completion.utils` module provides helpers for the
common prefix-match-and-sort pattern used by most sources:

```lua
local cu = require("shell.completion.utils")

-- Match array items starting with prefix, return suffixes
cu.match_prefix(items, prefix, opts?)
-- Match table keys starting with prefix, return suffixes
cu.match_prefix_keys(tbl, prefix, opts?)
```

`opts` is an optional table: `suffix` (string to append, default
`""`) and `sort` (sort by string length, default `true`).

### Built-in sources (shell)

| Source | Module suffix | Candidates |
|--------|--------------|------------|
| `bin` | `source.bin` | Executables from `$PATH` |
| `builtins` | `source.builtins` | Shell builtins and aliases |
| `fs` | `source.fs` | Filesystem paths |
| `env` | `source.env` | Environment variables |
| `cmds` | `source.cmds` | Builtin-specific args (secrets, theme, mneme, lilpack, ...) |
| `learned` | `source.learned` | Learned subcommand chains and flags from successful executions (frecency-ranked) |
| `lua_keywords` | `source.lua_keywords` | Lua language keywords |
| `lua_symbols` | `source.lua_symbols` | Lua symbols and members |

### Built-in sources (wiki)

| Source | Module suffix | Candidates |
|--------|--------------|------------|
| `wiki_commands` | `source.wiki_commands` | Wiki mode commands (`wiki`, `search`, `browse`, `view`, `info`) |
| `wiki_dbs` | `source.wiki_dbs` | `.mneme` files from `~/.local/share/lilush/wiki/` |
| `wiki_entries` | `source.wiki_entries` | Entry keys from the open wiki database |
| `wiki_indexes` | `source.wiki_indexes` | Browsing index names from the wiki manifest |
| `wiki_results` | `source.wiki_results` | Search result keys for the `view` command |

## Renderer interface

A renderer controls how completion candidates are displayed.  It is
attached to the input object via the `renderer` config field.

```lua
local renderer = require("term.input.renderer")

-- Use the built-in inline ghost renderer (default)
local r = renderer.new_inline_ghost()
```

### Required methods

```lua
-- Render completion candidates at the cursor position.
-- Returns the number of characters drawn (for clearing).
draw = function(self, completion, cursor_col, available_width, tss)
    -- completion: the completion object
    -- cursor_col: cursor column within the visible area
    -- available_width: max drawable characters
    -- tss: TSS instance for styling
    return draw_length
end

-- Erase previously drawn completion.
clear = function(self, draw_length, blank_char)
end
```

### Inline ghost renderer

The default renderer draws a single candidate as dimmed text after
the cursor, styled as `"completion"`.

TSS rules used:

| Rule | Purpose | Default |
|------|---------|---------|
| `completion` | Candidate text | Gray (`{146, 153, 167}`) |

## Metadata

Metadata is a table indexed by candidate position (1-based).
Providers can use plain tables or metatables with `__index` for
lazy generation:

```lua
local meta_with_index = function(fn)
    local m = {}
    setmetatable(m, { __index = fn })
    return m
end

self:set_meta(meta_with_index(function(t, key)
    if key <= builtins_count then
        return { source = "builtin" }
    end
    return { source = "bin" }
end))
```

### Common fields

| Field | Type | Used by | Description |
|-------|------|---------|-------------|
| `source` | `str` | All providers | TSS rule name for styling |

### Shell-specific fields

These are handled by the shell provider's `promote()` method, not
by the generic engine.

| Field | Type | Description |
|-------|------|-------------|
| `replace_prompt` | `str` | Replace entire line with this prefix + candidate |
| `exec_on_prom` | `bool` | Execute immediately after promotion |
| `trim_promotion` | `bool` | Strip leading whitespace from candidate |
| `reduce_spaces` | `bool` | Collapse consecutive spaces |

## Input integration

The input object (`term.input`) drives the completion lifecycle.

### Configuration

```lua
local input = require("term.input")

local inp = input.new({
    completion = comp,        -- completion object
    history = hist,           -- history object (passed to search)
    renderer = my_renderer,   -- custom renderer (default: inline ghost)
    eol_only = true,          -- complete only at end-of-line (default)
})
```

### Key bindings

| Key | Condition | Action |
|-----|-----------|--------|
| Any character | At EOL, no trailing space | `search_completion()` + draw |
| TAB | Candidates available | `promote_completion()` — common prefix or full |
| UP | At EOL, candidates available | `promote_completion_full()` |
| ESC | Buffer not empty | `scroll_completion("up")` — cycle candidates |
| ENTER | `should_auto_promote()` | `promote_completion_full()` — auto-execute |

### Completion lifecycle

1. User types a character at EOL
2. `search_completion()` calls `completion:search(line, history)`
3. Provider queries sources, calls `provide()` + `set_meta()`
4. `draw_completion()` delegates to renderer
5. User presses TAB:
   - Single candidate or `should_promote_full()`: full promotion
   - Multiple candidates: append common prefix, re-search

### eol_only flag

When `eol_only = false`, completion triggers at any cursor position.
The cursor position is passed to the provider's `search()` as the
`cursor_pos` parameter.  The provider is responsible for determining
what to complete based on cursor position.  The `promote()` method
handles how the candidate is inserted (append, replace token, etc.).

Note: the default inline ghost renderer draws at the cursor, which
may overwrite text after the cursor in mid-line mode.  A popup
renderer is recommended for mid-line completion.

## Writing a new provider

Minimal provider for a hypothetical "tags" completion:

```lua
-- my_app/completion/tags.lua
local search = function(self, input)
    self:flush()
    local prefix = input:match("#(%w*)$")
    if not prefix then return false end

    local tag_source = self:source("tags")
    local candidates = tag_source:search(prefix)
    self:provide(candidates)
    self:set_meta(setmetatable({}, {
        __index = function() return { source = "tag" } end,
    }))
    return self:available()
end

return { search = search }
```

Usage:

```lua
local theme_mod = require("theme")

local comp = completion.new({
    path = "my_app.completion.tags",
    tss = theme_mod.subscribe("my_app", "completion"),
    sources = { "my_app.completion.source.tags" },
})

local inp = input.new({ completion = comp })
```

The `tss` thunk handles styling via the `source` metadata field,
so a custom `get()` is only needed for non-standard rendering.