# opencoti runtime introspection & control surface (#676/#677)

Complete reference for **observing and controlling the opencoti agentic-serving
features** (PolyKV, SharedKVPool, rolling-KV residency, DCA, MTP/spec-decode,
session KV-reuse, retention eviction, sparse-attn, RYS) — what is configurable
at boot, what a client can set per request, and what the server reports back at
runtime.

Status: implemented 2026-07-15 (tasks #676 `/props` echo + core KV-info API,
#677 `/slots` enrichment). Vendored-source additive edits; host-only (no CUDA).

---

## 1. The three planes

| Plane | Mechanism | When |
|---|---|---|
| **Boot knobs** | CLI flags (`common/arg.cpp`) ← typed TS adapter fields ← `OPENCOTI_LLAMAFILE_*` env | server launch |
| **Per-request control** | 3 JSON body fields (`session_id`, `shared_pool_slot`, `shared_prefix_n_tokens`) | every completion request |
| **Runtime introspection** | `/props → "opencoti"` (boot-state echo + effective KV), `/slots → per-slot "opencoti"` (live state), completion `timings` fields | any time |

---

## 2. Boot knobs — by feature

Typed fields live in `packages/opencoti-llamafile/src/config.ts` (each maps to
env `OPENCOTI_LLAMAFILE_<SNAKE_CASE>`); CLI flags are registered in
`vendors/sources/llamafile/llama.cpp/common/arg.cpp`. Flags marked *(extraArgs
only)* have no typed adapter field yet.

### PolyKV — KV-quant tiers (M6)
- `ctypeK` / `ctypeV` → `-ctk` / `-ctv` — window KV types; scalar
  (`f16,bf16,q8_0,q6_0,q5_1,q5_0,q4_0`) + turbo (`turbo2/3/4/8`) + TCQ
  (`turbo2_tcq,turbo3_tcq`).
- `ctypeKTail` / `ctypeVTail` → `-ctkt` / `-ctvt` — **spilled-tail** types
  (#622 P2 mixed-KV); unset = same as window.
- `kvAutoTier` → env **`OPENCOTI_KV_AUTO_TIER=1`** — boot policy picks the
  least-compressing scalar pair that stays fully resident (#621). Explicit
  `-ctv` disables auto entirely; explicit `-ctk` holds K, walks V only. Dense
  full-attention models only; iSWA keeps f16.
- env **`OPENCOTI_KV_AUTO_TIER_TAIL=1`** — when auto-tier still spills, also
  auto-pick a q4_0 tail (separate opt-in).

### SharedKVPool (M6-S1) — per-request only, see §3. No boot flag.

### Rolling-KV residency / spill (M7, #582)
- `vramTargetMiB` → `--vram-target` — VRAM budget cap (MiB); 0 = free-VRAM auto.
- `kvResidencyMode` → `--kv-residency-mode` — residency tactic knob (0 = auto).
- `headinferGpuHeadsFrac` → `--headinfer-gpu-heads-frac` (default `auto`).

### Session KV-reuse (M0) + host-state guard
- `nParallel` → `--parallel`, `slotPromptSimilarity` → `-sps`,
  `cacheReuse` → `--cache-reuse`, `ctxSize` → `-c`.
- `cacheRamMiB` → `--cache-ram`, `ctxCheckpoints` → `--ctx-checkpoints`
  (bounded host-RAM defaults, bug-1325).
- `--kv-unified` — required for assistant-MTP with `--parallel > 1` (bug-2110 guard).

### Retention eviction (M1)
- `restKvEviction` → `--rest-kv-eviction`, `restKvRecent` → `--rest-kv-recent`,
  `restKvLayer` → `--rest-kv-layer`. Boot-global (no per-request override).

### DCA — Dual Chunk Attention (0078)
- `dcaEnabled` → `--dca on|off`, `dcaChunkSize` → `--dca-chunk-size`,
  `dcaYarnFactor` → `--dca-yarn-factor`.

### Spec-decode / MTP (M6-S4, bug-858)
- `mtpHead` → `--mtp-head` (alias of `-md`; gemma4-assistant GGUF),
  `specType` → `--spec-type` (`draft-assistant` | `draft-mtp` | …),
  `draftNGpuLayers` → `-ngld`, `--spec-draft-n-max` / `--spec-draft-n-min`,
  `--draft-block-size`, draft KV types `-ctkd` / `-ctvd`.

### Sparse attention (#551) *(extraArgs only)*
- `--sparse-attn`, `--sparse-attn-block-size/-topk/-recent/-sink/-refresh/-mode`.

### RYS layer duplication (#656) 
- `repeatLayers` → `--repeat-layers`; probe: `--rys-probe*` *(extraArgs only)*.

### Lazy slot context (F4 M3) *(server: extraArgs only; typed on embedder)*
- `--slot-initial-ctx`, `--slot-shrink-idle-ms`.

---

## 3. Per-request control (JSON body fields)

Parsed in `tools/server/server-task.cpp` (~line 275):

| Field | Default | Feature | Effect |
|---|---|---|---|
| `session_id` | `""` | M0 session KV-reuse | Slot affinity: the same session routes back to the slot that holds its KV (prevents cross-session eviction at `--parallel > 1`). opencode gets this injected automatically by `withKvReuseFetch` (which also forces `cache_prompt: true`). |
| `shared_pool_slot` | `-1` | SharedKVPool (M6-S1) | Bit-share this request's prefix KV with pool slot N (read-only). |
| `shared_prefix_n_tokens` | `0` | SharedKVPool | Length of the shared prefix in tokens. |

Everything else (tier, residency, DCA, retention) is **boot-global by design** —
changing it requires a relaunch.

---

## 4. Runtime introspection

### 4.1 `GET /props` → `"opencoti"` object (#676)

Boot-state echo, computed once at model load (sleep-safe — never touches the
live context at request time). Shape:

```jsonc
"opencoti": {
  "kv": {
    "cache_type_k": "q8_0",          // configured (-ctk)
    "cache_type_v": "q4_0",          // configured (-ctv)
    "cache_type_k_tail": null,       // configured (-ctkt); null = same as window
    "cache_type_v_tail": null,
    "auto_tier": false,              // OPENCOTI_KV_AUTO_TIER set
    "auto_tier_tail": false,         // OPENCOTI_KV_AUTO_TIER_TAIL set
    "effective": {                   // READ BACK from the live KV cache —
      "type_k": "q8_0",              // reflects auto-tier + residency sizing,
      "type_v": "q4_0",              // not just what was asked for
      "type_k_tail": "q8_0",         // == window type when no distinct tail
      "type_v_tail": "q4_0",
      "n_cells": 524288,             // allocated KV cells per stream
      "n_cells_resident": 524288,    // device-resident window (min over spilling layers)
      "n_layers_spilling": 0,        // layers with an engaged rolling-KV window
      "fully_resident": true,
      "is_iswa": true                // reported from the iSWA base (full-attn) cache
    }
  },
  "residency":  { "kv_residency_mode": 0, "vram_target_mib": 0, "headinfer_gpu_heads_frac": 1.0 },
  "dca":        { "enabled": true, "chunk_size": 0, "yarn_factor": 1.0 },
  "sparse_attn":{ "enabled": false, "block_size": 128, "topk": 0, "recent": 0, "sink": 1, "refresh": 8, "mode": 0 },
  "speculative":{ "types": ["none", "draft-assistant"],   // "none" is always slot 0 of the type list
                  "draft_model": "…assistant-Q8_0.gguf",
                  "n_max": 3, "n_min": 0, "draft_block_size": 0, "n_gpu_layers": 99,
                  "cache_type_k": "f16", "cache_type_v": "f16" },
  "kv_reuse":   { "n_parallel": 4, "kv_unified": true,    // kv_unified auto-forced for assistant-MTP (bug-2110)
                  "cache_reuse": 0, "slot_prompt_similarity": 0.1,
                  "ctx_checkpoints": 32, "cache_ram_mib": 8192 },
  "rest_kv":    { "eviction": false, "recent": 256, "layer": -1 },
  "repeat_layers": null
}
```

(Verified live 2026-07-15 against the v7-coder server on :8231 — values above are the
actual output for `-ctk q8_0 -ctv q4_0 -c 524288 --dca on --spec-type draft-assistant`.)

The `kv.effective` block is the **only authoritative source for the auto-tier
decision** — the decision happens inside the KV-cache constructor and is read
back through the new core API (§5). `configured != effective` is expected
whenever auto-tier engaged. Drafter note: on dual-ctx MTP the target context is
reported; the drafter's own KV layer is f16 by design.

### 4.2 `GET /slots` → per-slot `"opencoti"` object (#677)

Live state per slot (requires `--slots`, i.e. `params.endpoint_slots`):

```jsonc
{
  "id": 3, "n_ctx": 131072, "speculative": true, "is_processing": false,
  "opencoti": {
    "draft_n_total": 114,            // LIFETIME draft tokens generated (this slot) —
    "draft_n_accepted": 62,          //   survives task end; resets only on server restart
    "draft_acceptance": 0.544,       // lifetime ratio; null before any draft ran
    "session_id": "ses_09f4…",       // current/last task's session binding (M0)
    "shared_pool_slot": -1,          // current/last task's SharedKVPool binding
    "shared_prefix_n_tokens": 0
  }
}
```

The draft counters are **lifetime accumulators** (deliberately not cleared by the
slot's per-task reset — the per-task counters feed `timings.draft_n` instead). To
watch a single long turn, poll twice and diff. The session/pool fields appear only
once the slot has served at least one task (they come from the current-or-last
task's params); a freshly booted idle slot shows just the three draft fields.

Practical use: **draft_acceptance ≳ 0.95 sustained at turn end is the
rumination/loop tell** (healthy agentic decode sits ~0.4–0.9); this is now
pollable instead of log-scraping.

### 4.3 Per-completion `timings` (pre-existing, unchanged)

When timings are requested, each response carries: `cache_n` (KV prefix-reuse
hits — the M0/PolyKV effect), and `draft_n` / `draft_n_accepted` for that
response's spec rounds.

### 4.4 Still log-only (known gaps)

PolyKV pool share/reject lines (`poly-kv-pool: shared …`), retention-eviction
discards, rolling-KV tactic selection detail, auto-tier WARN line. Engine
events do **not** feed opencoti-server `/v1/tier-events` (that store records
tier-*routing* only).

---

## 5. Core C API (for embedders / tools)

Added in `include/llama.h`:

```c
struct llama_opencoti_kv_info {
    enum ggml_type type_k, type_v;            // effective window types (post auto-tier)
    enum ggml_type type_k_tail, type_v_tail;  // spilled-tail types (== window when none)
    uint32_t n_cells;                         // allocated KV cells per stream
    uint32_t n_cells_resident;                // min resident window across spilling layers
    uint32_t n_layers_spilling;               // 0 = fully resident
    bool     is_iswa;                         // reported from iSWA base cache
};
LLAMA_API bool llama_memory_opencoti_kv_info(llama_memory_t mem,
                                             struct llama_opencoti_kv_info * info);
```

Dispatch: virtual on `llama_memory_i` (default false → pure-recurrent memories
report nothing); `llama_kv_cache` reads its live tensors; `llama_kv_cache_iswa`
forwards to its non-SWA base; `llama_memory_hybrid` forwards to its attention
child. Boot-static — safe to read once after context creation and cache (the
server does exactly that in `get_meta()`).

Touched files: `include/llama.h`, `src/llama-memory.h`,
`src/llama-kv-cache.{h,cpp}`, `src/llama-kv-cache-iswa.{h,cpp}`,
`src/llama-memory-hybrid.{h,cpp}`, `src/llama-context.cpp`,
`tools/server/server-context.{h,cpp}`, plus `llamafile/BUILD.mk` (bug-2186: the
llamafile-mode `server.cpp.o` rule had no header deps — a stale object linked an
old inline `update_meta` with a mismatched `server_context_meta` layout, silently
yielding `"opencoti": null`; the rule now depends on `tools/server/*.h`). All
additive; off-path behavior byte-identical (pure read-side additions to two JSON
endpoints).

## 6. Quick recipes

```sh
# What is this server actually running?
curl -s :8231/props | jq .opencoti

# Did auto-tier change my KV types?
curl -s :8231/props | jq '.opencoti.kv | {configured: {k:.cache_type_k, v:.cache_type_v}, effective}'

# Is anything spilling?
curl -s :8231/props | jq .opencoti.kv.effective.fully_resident

# Live loop-tell: per-slot draft acceptance
curl -s :8231/slots | jq '.[] | {id, processing:.is_processing, acc:.opencoti.draft_acceptance}'
```

Related docs: [advanced_kv.md](advanced_kv.md) (M0-M3), [poly_kv.md](poly_kv.md)
(M6/SharedKVPool), [rolling_kv.md](rolling_kv.md) (M7 residency),
[gemma4_dca.md](gemma4_dca.md), [mtp.md](../evaluations/mtp.md),
[tstar_auto_spill.md](tstar_auto_spill.md) (auto-tier/auto-spill policy).