docs/features/introspection.md

opencoti runtime introspection & control surface (#676/#677)

Complete reference for observing and controlling the opencoti agentic-serving features (PolyKV, SharedKVPool, rolling-KV residency, DCA, MTP/spec-decode, session KV-reuse, retention eviction, sparse-attn, RYS) — what is configurable at boot, what a client can set per request, and what the server reports back at runtime.

Status: implemented 2026-07-15 (tasks #676 /props echo + core KV-info API, #677 /slots enrichment). Vendored-source additive edits; host-only (no CUDA).

---

1. The three planes

Plane	Mechanism	When
Boot knobs	CLI flags (common/arg.cpp) ← typed TS adapter fields ← OPENCOTI_LLAMAFILE_* env	server launch
Per-request control	3 JSON body fields (session_id, shared_pool_slot, shared_prefix_n_tokens)	every completion request
Runtime introspection	/props → "opencoti" (boot-state echo + effective KV), /slots → per-slot "opencoti" (live state), completion timings fields	any time

---

2. Boot knobs — by feature

Typed fields live in packages/opencoti-llamafile/src/config.ts (each maps to env OPENCOTI_LLAMAFILE_<SNAKE_CASE>); CLI flags are registered in vendors/sources/llamafile/llama.cpp/common/arg.cpp. Flags marked (extraArgs only) have no typed adapter field yet.

PolyKV — KV-quant tiers (M6)

• ctypeK / ctypeV → -ctk / -ctv — window KV types; scalar (f16,bf16,q8_0,q6_0,q5_1,q5_0,q4_0) + turbo (turbo2/3/4/8) + TCQ (turbo2_tcq,turbo3_tcq).
• ctypeKTail / ctypeVTail → -ctkt / -ctvt — spilled-tail types (#622 P2 mixed-KV); unset = same as window.
• kvAutoTier → env OPENCOTI_KV_AUTO_TIER=1 — boot policy picks the least-compressing scalar pair that stays fully resident (#621). Explicit -ctv disables auto entirely; explicit -ctk holds K, walks V only. Dense full-attention models only; iSWA keeps f16.
• env OPENCOTI_KV_AUTO_TIER_TAIL=1 — when auto-tier still spills, also auto-pick a q4_0 tail (separate opt-in).

SharedKVPool (M6-S1) — per-request only, see §3. No boot flag.

Rolling-KV residency / spill (M7, #582)

• vramTargetMiB → --vram-target — VRAM budget cap (MiB); 0 = free-VRAM auto.
• kvResidencyMode → --kv-residency-mode — residency tactic knob (0 = auto).
• headinferGpuHeadsFrac → --headinfer-gpu-heads-frac (default auto).

Session KV-reuse (M0) + host-state guard

• nParallel → --parallel, slotPromptSimilarity → -sps, cacheReuse → --cache-reuse, ctxSize → -c.
• cacheRamMiB → --cache-ram, ctxCheckpoints → --ctx-checkpoints (bounded host-RAM defaults, bug-1325).
• --kv-unified — required for assistant-MTP with --parallel > 1 (bug-2110 guard).

Retention eviction (M1)

• restKvEviction → --rest-kv-eviction, restKvRecent → --rest-kv-recent, restKvLayer → --rest-kv-layer. Boot-global (no per-request override).

DCA — Dual Chunk Attention (0078)

• dcaEnabled → --dca on|off, dcaChunkSize → --dca-chunk-size, dcaYarnFactor → --dca-yarn-factor.

Spec-decode / MTP (M6-S4, bug-858)

• mtpHead → --mtp-head (alias of -md; gemma4-assistant GGUF), specType → --spec-type (draft-assistant | draft-mtp | …), draftNGpuLayers → -ngld, --spec-draft-n-max / --spec-draft-n-min, --draft-block-size, draft KV types -ctkd / -ctvd.

Sparse attention (#551) (extraArgs only)

• --sparse-attn, --sparse-attn-block-size/-topk/-recent/-sink/-refresh/-mode.

RYS layer duplication (#656)

• repeatLayers → --repeat-layers; probe: --rys-probe* (extraArgs only).

Lazy slot context (F4 M3) (server: extraArgs only; typed on embedder)

• --slot-initial-ctx, --slot-shrink-idle-ms.

---

3. Per-request control (JSON body fields)

Parsed in tools/server/server-task.cpp (~line 275):

Field	Default	Feature	Effect
session_id	""	M0 session KV-reuse	Slot affinity: the same session routes back to the slot that holds its KV (prevents cross-session eviction at --parallel > 1). opencode gets this injected automatically by withKvReuseFetch (which also forces cache_prompt: true).
shared_pool_slot	-1	SharedKVPool (M6-S1)	Bit-share this request's prefix KV with pool slot N (read-only).
shared_prefix_n_tokens	0	SharedKVPool	Length of the shared prefix in tokens.

Everything else (tier, residency, DCA, retention) is boot-global by design — changing it requires a relaunch.

---

4. Runtime introspection

4.1 GET /props → "opencoti" object (#676)

Boot-state echo, computed once at model load (sleep-safe — never touches the live context at request time). Shape:

"opencoti": {
  "kv": {
    "cache_type_k": "q8_0",          // configured (-ctk)
    "cache_type_v": "q4_0",          // configured (-ctv)
    "cache_type_k_tail": null,       // configured (-ctkt); null = same as window
    "cache_type_v_tail": null,
    "auto_tier": false,              // OPENCOTI_KV_AUTO_TIER set
    "auto_tier_tail": false,         // OPENCOTI_KV_AUTO_TIER_TAIL set
    "effective": {                   // READ BACK from the live KV cache —
      "type_k": "q8_0",              // reflects auto-tier + residency sizing,
      "type_v": "q4_0",              // not just what was asked for
      "type_k_tail": "q8_0",         // == window type when no distinct tail
      "type_v_tail": "q4_0",
      "n_cells": 524288,             // allocated KV cells per stream
      "n_cells_resident": 524288,    // device-resident window (min over spilling layers)
      "n_layers_spilling": 0,        // layers with an engaged rolling-KV window
      "fully_resident": true,
      "is_iswa": true                // reported from the iSWA base (full-attn) cache
    }
  },
  "residency":  { "kv_residency_mode": 0, "vram_target_mib": 0, "headinfer_gpu_heads_frac": 1.0 },
  "dca":        { "enabled": true, "chunk_size": 0, "yarn_factor": 1.0 },
  "sparse_attn":{ "enabled": false, "block_size": 128, "topk": 0, "recent": 0, "sink": 1, "refresh": 8, "mode": 0 },
  "speculative":{ "types": ["none", "draft-assistant"],   // "none" is always slot 0 of the type list
                  "draft_model": "…assistant-Q8_0.gguf",
                  "n_max": 3, "n_min": 0, "draft_block_size": 0, "n_gpu_layers": 99,
                  "cache_type_k": "f16", "cache_type_v": "f16" },
  "kv_reuse":   { "n_parallel": 4, "kv_unified": true,    // kv_unified auto-forced for assistant-MTP (bug-2110)
                  "cache_reuse": 0, "slot_prompt_similarity": 0.1,
                  "ctx_checkpoints": 32, "cache_ram_mib": 8192 },
  "rest_kv":    { "eviction": false, "recent": 256, "layer": -1 },
  "repeat_layers": null
}

(Verified live 2026-07-15 against the v7-coder server on :8231 — values above are the actual output for -ctk q8_0 -ctv q4_0 -c 524288 --dca on --spec-type draft-assistant.)

The kv.effective block is the only authoritative source for the auto-tier decision — the decision happens inside the KV-cache constructor and is read back through the new core API (§5). configured != effective is expected whenever auto-tier engaged. Drafter note: on dual-ctx MTP the target context is reported; the drafter's own KV layer is f16 by design.

4.2 GET /slots → per-slot "opencoti" object (#677)

Live state per slot (requires --slots, i.e. params.endpoint_slots):

{
  "id": 3, "n_ctx": 131072, "speculative": true, "is_processing": false,
  "opencoti": {
    "draft_n_total": 114,            // LIFETIME draft tokens generated (this slot) —
    "draft_n_accepted": 62,          //   survives task end; resets only on server restart
    "draft_acceptance": 0.544,       // lifetime ratio; null before any draft ran
    "session_id": "ses_09f4…",       // current/last task's session binding (M0)
    "shared_pool_slot": -1,          // current/last task's SharedKVPool binding
    "shared_prefix_n_tokens": 0
  }
}

The draft counters are lifetime accumulators (deliberately not cleared by the slot's per-task reset — the per-task counters feed timings.draft_n instead). To watch a single long turn, poll twice and diff. The session/pool fields appear only once the slot has served at least one task (they come from the current-or-last task's params); a freshly booted idle slot shows just the three draft fields.

Practical use: draft_acceptance ≳ 0.95 sustained at turn end is the rumination/loop tell (healthy agentic decode sits ~0.4–0.9); this is now pollable instead of log-scraping.

4.3 Per-completion timings (pre-existing, unchanged)

When timings are requested, each response carries: cache_n (KV prefix-reuse hits — the M0/PolyKV effect), and draft_n / draft_n_accepted for that response's spec rounds.

4.4 Still log-only (known gaps)

PolyKV pool share/reject lines (poly-kv-pool: shared …), retention-eviction discards, rolling-KV tactic selection detail, auto-tier WARN line. Engine events do not feed opencoti-server /v1/tier-events (that store records tier-routing only).

---

5. Core C API (for embedders / tools)

Added in include/llama.h:

struct llama_opencoti_kv_info {
    enum ggml_type type_k, type_v;            // effective window types (post auto-tier)
    enum ggml_type type_k_tail, type_v_tail;  // spilled-tail types (== window when none)
    uint32_t n_cells;                         // allocated KV cells per stream
    uint32_t n_cells_resident;                // min resident window across spilling layers
    uint32_t n_layers_spilling;               // 0 = fully resident
    bool     is_iswa;                         // reported from iSWA base cache
};
LLAMA_API bool llama_memory_opencoti_kv_info(llama_memory_t mem,
                                             struct llama_opencoti_kv_info * info);

Dispatch: virtual on llama_memory_i (default false → pure-recurrent memories report nothing); llama_kv_cache reads its live tensors; llama_kv_cache_iswa forwards to its non-SWA base; llama_memory_hybrid forwards to its attention child. Boot-static — safe to read once after context creation and cache (the server does exactly that in get_meta()).

Touched files: include/llama.h, src/llama-memory.h, src/llama-kv-cache.{h,cpp}, src/llama-kv-cache-iswa.{h,cpp}, src/llama-memory-hybrid.{h,cpp}, src/llama-context.cpp, tools/server/server-context.{h,cpp}, plus llamafile/BUILD.mk (bug-2186: the llamafile-mode server.cpp.o rule had no header deps — a stale object linked an old inline update_meta with a mismatched server_context_meta layout, silently yielding "opencoti": null; the rule now depends on tools/server/*.h). All additive; off-path behavior byte-identical (pure read-side additions to two JSON endpoints).

6. Quick recipes

# What is this server actually running?
curl -s :8231/props | jq .opencoti

# Did auto-tier change my KV types?
curl -s :8231/props | jq '.opencoti.kv | {configured: {k:.cache_type_k, v:.cache_type_v}, effective}'

# Is anything spilling?
curl -s :8231/props | jq .opencoti.kv.effective.fully_resident

# Live loop-tell: per-slot draft acceptance
curl -s :8231/slots | jq '.[] | {id, processing:.is_processing, acc:.opencoti.draft_acceptance}'

Related docs: advanced_kv.md (M0-M3), poly_kv.md (M6/SharedKVPool), rolling_kv.md (M7 residency), gemma4_dca.md, mtp.md, tstar_auto_spill.md (auto-tier/auto-spill policy).