bfe060ceb51638f00fb6b9e6eb46773cd87eaa76

raw

Embedded LLM Module (agent_smith/llm)

Overview

This document describes the current implemented LLM contract used by Agent Smith.

1. Scope and boundary

agent_smith/llm owns protocol mechanics only:

request shaping for Chat Completions-compatible APIs
HTTP/SSE transport calls
stream assembly
tool schema/call/result helpers
model catalog fetch and discovery helpers

It does not own turn orchestration, role policy, dispatch, UI, or persistence.

2. Public surface

Top-level entry: agent_smith/llm.lua

new(cfg)
chat(cfg, request)
chat_stream(cfg, request, handlers)
model namespace (agent_smith.llm.model)
tool namespace (agent_smith.llm.tool)

Client constructor/export (llm/client.lua) also exposes helper methods:

normalize_api_url
build_headers
build_endpoint
build_chat_request
serialize_request_body
normalize_usage
normalize_error

3. Request contract

Required core fields

model
messages (array)

Optional fields currently passed through

tools
tool_choice
stream
temperature
max_new_tokens
cancel (runtime cancellation token/handle)

Message validation and normalization

sanitize_messages enforces sequence correctness:

assistant tool calls must have IDs
tool results must reference known pending tool call IDs
no duplicate or dangling tool call IDs
tool result content is stringified when needed

Invalid ordering/shape returns schema/protocol errors before transport.

4. Response contract

Non-streaming chat returns normalized payload:

message (role, content, optional tool_calls)
finish_reason
usage normalized to input_tokens / output_tokens / total_tokens
raw response body/chunk context

Errors are normalized to agent_error with stable kind/code when possible.

No implicit retry loop is performed inside the raw client.

Transport and empty-response retries are handled by the turn layer (core/turn.lua).

5. Streaming contract

chat_stream(request, handlers) supports handlers:

on_text(delta)
on_reasoning(delta)
on_tool_delta(delta)
on_usage(usage)
on_error(err)
on_done(final_response)

Accepted SSE message payload forms in llm/client.lua:

decoded table chunk (event.data already object)
JSON string chunk (decoded before assembly)
[DONE] sentinel string (marks stream completion)
heartbeat/comment/empty frames (ignored)

llm/stream.lua assembler:

accumulates text and reasoning deltas
merges incremental tool-call deltas
tracks finish reason and usage
builds final normalized response

SSE polling checks cancellation between sse:update() iterations.

update() == false while still connected is treated as transient up to a bounded threshold; if it persists, streaming fails with explicit stream_stalled error instead of silently finalizing.

on_done is called only on successful stream completion (explicit done/disconnect path). Protocol/transport failures surface as errors and do not synthesize success.

6. Tool payload contracts

agent_smith.llm.tool exports:

build_tool_schema(tool_spec)
extract_tool_call(message)
extract_tool_calls(message)
extract_stream_tool_calls(delta)
merge_stream_delta(state, delta_calls)
build_tool_result_message(call_id, output)

Tool schema input expectations

build_tool_schema expects:

tool_spec.name (string)
tool_spec.input_schema (table)

It emits Chat Completions wire shape:

{
  "type": "function",
  "function": {
    "name": "...",
    "description": "...",
    "parameters": { "...": "..." }
  }
}

Naming mapping

LLM call field: id
runtime field: call_id
tool message wire field: tool_call_id

7. Model helpers

Catalog fetch (`llm/model/catalog.lua`)

Fetches GET /models from Chat Completions-compatible API base.
Normalizes entries into descriptors (id, name, context/output token hints, raw).

Discovery (`llm/model/discovery.lua`)

Implemented strategies:

first_available
prefer_prefix
prefer_capability_tag
fallback_chain

Each returns { model_id, reason } on success.

8. Limitations

Protocol support is Chat Completions-compatible only.
No provider abstraction or multi-provider negotiation.
No automatic retries inside the raw LLM client.
Non-stream cancellation is checked before/after request boundaries.
Stream cancellation depends on SSE poll/update cadence and transport timeouts.
Discovery helpers are available but not automatically used by runtime model selection.