# Embedded LLM Module (`agent_smith/llm`)

## Overview

This document describes the current implemented LLM contract used by Agent Smith.

## 1. Scope and boundary

`agent_smith/llm` owns protocol mechanics only:

- request shaping for Chat Completions-compatible APIs
- HTTP/SSE transport calls
- stream assembly
- tool schema/call/result helpers
- model catalog fetch and discovery helpers

It does **not** own turn orchestration, role policy, dispatch, UI, or persistence.

---

## 2. Public surface

Top-level entry: `agent_smith/llm.lua`

- `new(cfg)`
- `chat(cfg, request)`
- `chat_stream(cfg, request, handlers)`
- `model` namespace (`agent_smith.llm.model`)
- `tool` namespace (`agent_smith.llm.tool`)

Client constructor/export (`llm/client.lua`) also exposes helper methods:

- `normalize_api_url`
- `build_headers`
- `build_endpoint`
- `build_chat_request`
- `serialize_request_body`
- `normalize_usage`
- `normalize_error`

---

## 3. Request contract

### Required core fields

- `model`
- `messages` (array)

### Optional fields currently passed through

- `tools`
- `tool_choice`
- `stream`
- `temperature`
- `max_new_tokens`
- `cancel` (runtime cancellation token/handle)

### Message validation and normalization

`sanitize_messages` enforces sequence correctness:

- assistant tool calls must have IDs
- tool results must reference known pending tool call IDs
- no duplicate or dangling tool call IDs
- tool result content is stringified when needed

Invalid ordering/shape returns schema/protocol errors before transport.

---

## 4. Response contract

Non-streaming `chat` returns normalized payload:

- `message` (`role`, `content`, optional `tool_calls`)
- `finish_reason`
- `usage` normalized to `input_tokens` / `output_tokens` / `total_tokens`
- `raw` response body/chunk context

Errors are normalized to `agent_error` with stable `kind`/`code` when possible.

No implicit retry loop is performed inside the raw client.

Transport and empty-response retries are handled by the turn layer (`core/turn.lua`).
---

## 5. Streaming contract

`chat_stream(request, handlers)` supports handlers:

- `on_text(delta)`
- `on_reasoning(delta)`
- `on_tool_delta(delta)`
- `on_usage(usage)`
- `on_error(err)`
- `on_done(final_response)`

Accepted SSE message payload forms in `llm/client.lua`:

- decoded table chunk (`event.data` already object)
- JSON string chunk (decoded before assembly)
- `[DONE]` sentinel string (marks stream completion)
- heartbeat/comment/empty frames (ignored)

`llm/stream.lua` assembler:

- accumulates text and reasoning deltas
- merges incremental tool-call deltas
- tracks finish reason and usage
- builds final normalized response

SSE polling checks cancellation between `sse:update()` iterations.

`update() == false` while still connected is treated as transient up to a bounded threshold;
if it persists, streaming fails with explicit `stream_stalled` error instead of silently finalizing.

`on_done` is called only on successful stream completion (explicit done/disconnect path).
Protocol/transport failures surface as errors and do not synthesize success.
---

## 6. Tool payload contracts

`agent_smith.llm.tool` exports:

- `build_tool_schema(tool_spec)`
- `extract_tool_call(message)`
- `extract_tool_calls(message)`
- `extract_stream_tool_calls(delta)`
- `merge_stream_delta(state, delta_calls)`
- `build_tool_result_message(call_id, output)`

### Tool schema input expectations

`build_tool_schema` expects:

- `tool_spec.name` (string)
- `tool_spec.input_schema` (table)

It emits Chat Completions wire shape:

```json
{
  "type": "function",
  "function": {
    "name": "...",
    "description": "...",
    "parameters": { "...": "..." }
  }
}
```

### Naming mapping

- LLM call field: `id`
- runtime field: `call_id`
- tool message wire field: `tool_call_id`

---

## 7. Model helpers

### Catalog fetch (`llm/model/catalog.lua`)

- Fetches `GET /models` from Chat Completions-compatible API base.
- Normalizes entries into descriptors (`id`, `name`, context/output token hints, `raw`).

### Discovery (`llm/model/discovery.lua`)

Implemented strategies:

- `first_available`
- `prefer_prefix`
- `prefer_capability_tag`
- `fallback_chain`

Each returns `{ model_id, reason }` on success.

---

## 8. Limitations

1. Protocol support is Chat Completions-compatible only.
2. No provider abstraction or multi-provider negotiation.
3. No automatic retries inside the raw LLM client.
4. Non-stream cancellation is checked before/after request boundaries.
5. Stream cancellation depends on SSE poll/update cadence and transport timeouts.
6. Discovery helpers are available but not automatically used by runtime model selection.