Upload folder using huggingface_hub

docs/features/advanced_kv.md

@@ -0,0 +1,909 @@
+# F5 — Advanced KV Cache Techniques
+
+> Parent: [../MASTER_PLAN.md](../MASTER_PLAN.md)
+> Status: **planning**
+> Owner: TBD
+
+## Problem
+
+The vendored llamafile (F2) handles a single model + a single
+context window competently, but the KV cache is the dominant
+memory consumer at long contexts and the dominant time consumer
+across agentic turns that share a prefix. The research
+landscape has produced several techniques that attack different
+parts of this problem:
+
+- **Context-aware eviction** — drop tokens the model is least
+  likely to attend to again, instead of allocating capacity for
+  every token in the window.
+- **Per-head GPU/CPU split** — selective attention heads stay
+  on GPU, the rest stream from host RAM, dramatically
+  shrinking the GPU memory footprint.
+- **Asymmetric GPU-CPU pipelining** — load-aware scheduling
+  that keeps both compute resources busy.
+- **Layer-ahead pre-computation** — the CPU starts the next
+  layer's attention while the GPU finishes the current one.
+- **Shared compressed pool** — a single asymmetrically-
+  compressed KV pool across concurrent agents.
+
+opencoti adopts these as a *sequence of patches* against the
+vendored llamafile/llama.cpp tree, each developed and benched in
+isolation, then validated as a stack.
+
+## Goals
+
+- **G1.** Land a series of numbered patches under
+  `vendors/patches/llamafile/` that implement (or port the
+  ideas of) **ReST-KV**, **HeadInfer**, **NEO**,
+  **ScoutAttention**, and **PolyKV**.
+- **G2.** Each technique gets its own per-patch bench in
+  `perf/llamafile/<technique>.bench.ts` measuring the
+  documented claim against baseline.
+- **G3.** A glue milestone (`0050-glue-bench.patch`) validates
+  the stack composes and provides a standing bench harness for
+  all techniques + the full stack.
+- **G4.** Patches stay rebasable on upstream pin bumps. Each
+  patch ships with a header naming its bench file, milestone,
+  and (when applicable) the upstreaming PR.
+
+## Non-goals (for now)
+
+- **LMCache** — vLLM/SGLang-bound, llama.cpp portability 1/5.
+  Tracked as M7 (deferred). Re-evaluate once M0–M6 are in.
+- Full multi-tenant serving infrastructure. PolyKV ships in
+  single-agent mode at M6 (degenerates to a 1-tenant pool);
+  multi-tenant value lights up only when opencoti adds
+  multi-agent fan-out on Tier 0 (future roadmap).
+- A single "advanced features" toggle in user config. Each
+  technique is its own opt-in flag with sensible defaults
+  (most default off pending stability).
+
+## Sequencing rationale
+
+ReST-KV first (memory savings reduce load on every later
+technique). HeadInfer second (per-head GPU residency is a
+clean orthogonal axis on top of eviction). NEO third (load-
+aware GPU-CPU scheduling assumes per-head granularity to
+schedule against). ScoutAttention fourth (layer-ahead pre-
+compute sits naturally on top of NEO's CPU compute path).
+Glue fifth. PolyKV **immediately after glue** (keep momentum
+from the preceding patch series — deferring loses the
+implementation context and forces a costly cold-context
+re-visit later).
+
+llama.cpp-portability scores from research:
+
+| Technique | Score | License | Reference |
+| --- | --- | --- | --- |
+| ReST-KV | 3/5 | unknown | arXiv 2605.09649 (Make Each Token Count) — no public code yet |
+| HeadInfer | 2/5 | MIT | arXiv 2502.12574 — [github.com/wdlctc/headinfer](https://github.com/wdlctc/headinfer) |
+| NEO | 2/5 | Apache-2.0 | arXiv 2411.01142 — [github.com/NEO-MLSys25/NEO](https://github.com/NEO-MLSys25/NEO) |
+| ScoutAttention | 2/5 | unknown | arXiv 2603.27138 — code pending (DAC 2026) |
+| PolyKV | 1/5 | MIT | arXiv 2604.24971 — [github.com/ishan1410/PolyKV](https://github.com/ishan1410/PolyKV) |
+| LMCache *(deferred)* | 1/5 | Apache-2.0 | arXiv 2510.09665 — [github.com/lmcache/lmcache](https://github.com/lmcache/lmcache) |
+
+## Patch number reservations
+
+```
+0001..0005-*.patch                   F4 M3 lazy-slot (these are F4's, not F5's)
+0006-kv-reuse-prefix.patch           F5 M0 (was F2 M4; SHIPPED 2026-05-24)
+0007-build-header-deps.patch         build infra (SHIPPED 2026-05-27; not a feature — see below)
+0010-rest-kv-eviction.patch          F5 M1 (SHIPPED 2026-05-27)
+0011-rest-kv-config.patch            F5 M1 (optional split; NOT built — single-patch v1)
+0020-headinfer-per-head.patch        F5 M2 (SHIPPED 2026-05-28)
+0030-neo-pipeline.patch              F5 M3 (SHIPPED 2026-05-28)
+0031-per-stream-split.patch          F4 M3 Phase 4 (SHIPPED 2026-05-29; task #109)
+0032-m2-state-io.patch               F5 M2 hardening / bug-222 fix (SHIPPED 2026-05-29)
+0033-cpy-tie-blck-align.patch        F5 bug-225 fix (SHIPPED 2026-05-29)
+0034-m2-pinned-host.patch            F5 M7-A foundation — pinned host KV (SHIPPED 2026-05-29)
+0035-concat-q-block-aware.patch      F5 bug-226 fix (SHIPPED 2026-05-30)
+0036-pcie-probe-consume.patch        F5-opt W1 — PCIe probe consume #293 (SHIPPED 2026-05-30)
+0040-iqk-flash-attn.patch            F5-opt W2 — wholesale iqk CPU-FA import #290 (SHIPPED 2026-05-30)
+0041-q8kv.patch                      RETIRED 2026-05-30 — Q8_KV dropped (#291); redundant 8-bit (q8_0 covers it), TurboQuant M6 is the KV-quant driver
+0042-fused-moe.patch                 F5-opt W3 — op-level fused MoE up+gate+GLU #292 (SHIPPED 2026-05-30; separate-up/gate MoE only — N/A to Gemma-4 fused gate_up)
+(M5 glue+bench shipped as TS — perf/llamafile/, NOT a patch; old 0050 slot retired)
+0070-rolling-kv.patch                F5 M7 Rolling KV #296 (was LMCache; redesigned — see rolling_kv.md) (SHIPPED)
+0071-state-io-cosmocc.patch          F5 M7 cosmocc state-IO open fix #349 / bug-269 (SHIPPED; applies after 0070)
+0072-poly-kv-pool.patch              F5 M6 PolyKV S1 SharedKVPool #370 (SHIPPED 2026-06-06; applies after 0071)
+0073-turboquant-kv.patch             F5 M6 PolyKV S2 TurboQuant tiers+InnerQ+Level-A+Level-B #371 (SHIPPED 2026-06-06; GGML_OP_TURBO_WHT; applies after 0071)
+0090-scout-layer-ahead.patch         F5 M4 ScoutAttention (DEFERRED Aug 2026, #264 — moved off the 0040 slot)
+```
+
+Gaps (0002–0009, 0012–0019, ...) are intentional — they leave
+room for per-technique refinements without renumbering
+downstream patches.
+
+## Milestones
+
+### M0 — KV-cache reuse across agentic turns *(absorbed from F2 M4; shipped 2026-05-24)*
+
+Reuse the resident KV across agentic turns instead of re-prefilling the
+shared prefix (system prompt + tool defs + history) every turn.
+
+**Key finding (shaped the design).** The vendored llama.cpp server
+**already** reuses KV for a matching token-prefix: `cache_prompt` defaults
+to `true`, and `get_available_slot()` (server-context.cpp) picks the idle
+slot whose cached prefix best matches the incoming prompt (above
+`--slot-prompt-similarity`, then LRU). So single-session turn-to-turn reuse
+at the default `--parallel 1` works out of the box — the original "add a
+server-side prefix-hash cache" plan predates that capability. What upstream
+lacks is **session→slot affinity**: under `--parallel > 1`, a concurrent
+opencode session can be routed to another session's slot and evict a prefix
+that would otherwise be reused.
+
+**What shipped.**
+- `vendors/patches/llamafile/0006-kv-reuse-prefix.patch` (touches only
+  `tools/server/{server-task.h,server-task.cpp,server-context.cpp}`, disjoint
+  from the F4 0001–0005 lazy-slot patches): an optional `session_id` request
+  field + a `session_id → slot.id` affinity map. `get_available_slot()`
+  prefers the session's own idle slot (its resident KV *is* that session's
+  prefix) before the existing LCP→LRU path; `launch_slot_with_task()` records
+  the binding, dropping any stale session on that slot. **Regression shield:**
+  empty `session_id` (every non-opencoti client) skips the affinity phase, so
+  upstream slot selection is byte-for-byte unchanged.
+- `@opencoti/llamafile`: a session-aware `fetch` wrapper
+  (`withKvReuseFetch`) tags each chat-completions body with the session's id
+  + `cache_prompt: true`; `TierStreamRequest.sessionID` already carries it, so
+  no `runtime.ts` / surgical-hook change. Typed, env-backed server flags
+  (`--parallel`/`-sps`/`--cache-reuse`/`-c`), omitted when unset.
+- "(session_id, prefix) reuse" falls out: affinity routes to the slot, the
+  server's existing token-LCP supplies the prefix match — **no separate
+  prefix hash needed**.
+
+**Bench** (`perf/llamafile/turn-2-latency.bench.ts`, live on solidPC,
+Qwen2.5-Coder-0.5B, CPU): scenario A turn-2 prefill collapses **2061 → 20
+tokens** (202ms → 12ms; `cache_n` 2076) vs a no-cache control at 2096 tokens;
+scenario B (`--parallel 2`, two sessions sharing a system prefix) each reuse
+their own KV under affinity (`cache_n` 1990 each). 5/5 checks pass.
+
+**Deferred (optional `0007`, evidence-driven):** an explicit `prefix_hash`
++ cross-restart disk persistence (`--slot-save-path` keyed by
+`session_id+hash`). Within a server lifetime the resident-token LCP already
+supplies the prefix match, so the hash adds nothing until restart-durable or
+distributed reuse is wanted. No plugin, no HTTP surface, no TS client, no
+opencode surgical hook — the surgical-hook grep count stays 18 (the vendored
+C++ marker is `// opencoti F5 M0`, a vendored-source tag, not a counted hook).
+
+**Build gotcha (buglog bug-167):** `server-task.h` is included by
+`server-context.cpp`, `server-task.cpp` **and** `server-queue.cpp`. Editing
+that header and rebuilding with `bun run build:llamafile:make` leaves
+`server-queue.cpp.o` compiled against the old `task_params` layout → ABI
+skew → `bad_array_new_length` on every generation request. After a vendored
+**header** change, force a consistent recompile (`rm -f
+vendors/sources/llamafile/o//llama.cpp/tools/server/*.o` then `:make`, or the
+full reproducible `build:llamafile`).
+
+### M1 — ReST-KV / retention-aware eviction *(shipped 2026-05-27)*
+
+When a slot exceeds `n_ctx`, upstream context-shift keeps the first `n_keep`
+tokens and blindly discards the **positional middle** `n_discard` chunk. M1
+makes the choice **retention-aware**: discard the lowest-value contiguous
+window instead, so long-context generation keeps the evidence that matters.
+
+**Reference decision (resolved the open question).** The named paper
+(arXiv 2605.09649) is **TRIM-KV** — *learned* retention gates trained by
+distillation, needing a per-model pipeline + published gates; not a drop-in
+runtime patch. **SnapKV / H2O** score tokens from the materialized attention
+matrix, which **Flash Attention never produces** (the vendored pin defaults
+`flash_attn_type = AUTO`) — incompatible with the FA path opencoti wants on.
+So M1 substitutes a **score-free, FA-compatible** criterion: **KeyDiff**
+(arXiv 2504.15364) — evict the keys least distinctive by cosine similarity
+(distinctiveness = mean over heads of `1 - cos(key_head, mean_head)`; higher =
+keep). `domvox/triattention-ggml` is a working ggml reference but scores every
+decode interval (K copied GPU→CPU each `--tri-interval`, "the main
+bottleneck"); M1 avoids that by scoring **only at the infrequent
+context-shift trigger**, amortizing the K read.
+
+**Depth decision: server-layer window (contiguous).** Eviction stays
+contiguous so the server's CPU token mirror (`slot.prompt.tokens`) stays 1:1
+with KV positions and all prefix-reuse / sampler logic is reused untouched.
+Only *which* width-`n_discard` window is discarded changes — the
+minimum-score one over `[n_keep, n_tokens − rest_kv_recent)`, not the
+positional middle. Full per-token **scattered** KeyDiff (a core
+`evict_to_budget` + a server rework to tolerate holes) is **deferred** —
+revisit only if M1's bench shows window granularity costs meaningful quality.
+
+**What shipped.**
+- `vendors/patches/llamafile/0010-rest-kv-eviction.patch` (build-time; no
+  opencode upstream file touched):
+  - `src/llama-kv-cache.{h,cpp}`: concrete `seq_key_scores(seq_id,
+    layer_hint)` — enumerates the seq's cells in position order, reads each
+    post-RoPE K row to host, reshapes into `n_head_kv` heads, returns the
+    per-position KeyDiff score over a representative layer (default mid).
+  - **ABI-safe interface choice:** `src/llama-memory.h` gains a **non-pure**
+    virtual `seq_key_scores` with an empty default body — no backend is forced
+    to implement it and no vtable churn/RTTI is needed (the pure-virtual
+    `llama_memory_i` methods are left untouched).
+  - `src/llama-kv-cache-iswa.{h,cpp}`: delegate to the base cache.
+  - `include/llama.h` + `src/llama-context.cpp`: C shim
+    `llama_memory_seq_key_scores(...)` that returns 0 for non-KV memory (or a
+    key type with no float converter) → the caller falls back to positional.
+  - `tools/server/server-context.cpp`: inside the **existing** shift block,
+    when `rest_kv_eviction` is on, build prefix sums over the scores and pick
+    the min-sum window start `w0`; run the existing `seq_rm`/`seq_add`/mirror
+    rewrite parameterized by `w0` instead of `n_keep`. **Off (default) =
+    byte-for-byte upstream** (the off-path never calls the scorer).
+  - `common/common.h` + `common/arg.cpp`: `--rest-kv-eviction` /
+    `--rest-kv-recent N` / `--rest-kv-layer N` (server example).
+- `@opencoti/llamafile` (`config.ts`/`launch.ts`): typed, env-backed
+  (`OPENCOTI_LLAMAFILE_REST_KV_*`) fields → `buildServerArgs` emits the flags
+  when set, omits when unset. Global flag, not per-request → no fetch wrapper.
+
+**Bench** (`perf/llamafile/rest-kv-eviction.bench.ts`, needle-in-the-middle,
+live on solidPC, Qwen2.5-Coder-0.5B, CPU): **hard checks 3/3** — a context
+shift fired, the rest-kv eviction path executed **only** with eviction on, and
+the off-path never logged it (proving the off-path is upstream-identical).
+These deterministic, model-independent checks gate the exit code. The
+quality-delta **headline is soft and model-dependent**: a 0.5B model either
+states the needle code *before* the shift fires (generous headroom → both runs
+recover) or degrades so far *after* the shift that neither restates a random
+code even when the needle survives in KV — demonstrating the delta needs a
+stronger model. `0011` stays reserved (single-patch v1, mirroring M0's `0006`).
+
+**Surgical-hook count stays 18** — the C++ marker is `// opencoti F5 M1
+rest-kv-eviction`, a vendored-source tag, not a counted opencode hook.
+
+**Build-infra prerequisite (`0007`, buglog bug-170).** Iterating M1's vendored
+C++ surfaced that the build had **no working header-dependency tracking**
+(mkdeps' `o//depend` can't resolve llama.cpp's `-iquote` includes → zero src
+edges → header edits never recompiled → forced from-scratch rebuilds) **and**
+that **ccache cached nothing** (a `.d`-name mismatch made it abort every
+compile with "internal error"). Patch `0007-build-header-deps.patch` fixes both
+(`-MMD -MF $@.d` in the compile rules + `-include` the per-object `.o.d`), and
+`build-pipeline` runs the full reset-build with `CCACHE_RECACHE=1` so a warm
+cache can't orphan a cosmocc fat object's `.aarch64` twin. See
+`docs/features/llamafile_build.md`. This is what made M1 (and every later F5
+milestone) tractable to iterate.
+
+**Deferred / not done:** scattered per-token KeyDiff; quantized-K cosine
+scoring (falls back to positional when K is quantized); the optional `0011`
+per-request override; the full **reset-based reproducible build** verification
+(would `git clean -fdx` the in-tree vendored state — run when a reset window is
+authorized; `0010`/`0007` are reverse-verified against pristine).
+
+### M2 — HeadInfer (head-wise GPU/CPU split) *(shipped 2026-05-28)*
+
+When a model is GPU-offloaded, the *entire* KV cache for offloaded layers lives
+in VRAM and caps how much context fits on the card. M2 keeps a configurable
+fraction of attention heads' KV resident on GPU and offloads the rest to host
+memory, reassembling the full head set per attention step via `ggml_concat` on
+the head axis. Resident VRAM drops proportionally to the offloaded fraction;
+correctness is preserved.
+
+**Reference decision (resolved at impl time).** The named paper (arXiv
+2502.12574, [wdlctc/headinfer](https://github.com/wdlctc/headinfer)) is
+HF-Transformers; not portable. A parallel investigation of `/shared/dev/lightseek`
+— which has deep, working llama.cpp head-splitting surgery — confirmed it is
+**distributed tensor-parallelism** (heads split across MPI/RPC ranks, each on
+its own GPU) on a different llama.cpp lineage (vanilla upstream, not the
+Mozilla-Ocho llamafile fork). lightseek informed the design (GQA-ratio guard,
+"logical dims full / physical alloc partial" framing) but its backend-level
+machinery is **reference only, not code to port** — porting it wholesale would
+blow past opencoti's minimal-additive-patch rule.
+
+**Depth decision: head-RESIDENCY split, not head-COMPUTE split (foundation-first).**
+M2 delivers offload + scheduler-driven stream-back for correctness; the per-head
+attention COMPUTE split (run CPU-resident heads' attention on the CPU backend to
+avoid the per-step stream-back) is the headline perf win and is **deferred to
+optional `0021`** — revisit only if the bench shows the stream-back cost
+dominates. The "heavy" depth option, consciously deferred (mirrors M1's
+window-vs-scattered).
+
+**What shipped.**
+- `vendors/patches/llamafile/0020-headinfer-per-head.patch` (build-time;
+  no opencode upstream file touched; 14 files / 28 hunks / 520 lines):
+  - `src/llama-kv-cache.{h,cpp}`: per-layer `kv_layer` gains optional
+    `k_cpu`/`v_cpu` sub-tensors + a `gpu_heads` count (0/absent = no split).
+    Constructor allocates the GPU half (gpu-head rows) in the device buffer-type
+    and the CPU half (cpu-head rows) in a CPU buffer-type via the existing
+    `ctx_for_buft(ggml_backend_cpu_buffer_type())` — a CPU sub-tensor on an
+    otherwise-GPU layer just adds a CPU ctx/buffer pair, **no allocator
+    rework**. GQA-safe split: clamp `gpu_heads` to a whole number of KV-head
+    groups (lightseek lesson). Phase-1/2/3 hooks (`ensure_cleared`,
+    `shrink_if_idle`) loop the extra sub-tensor.
+  - `get_k`/`get_v`: when split, build the gpu/cpu views and
+    `ggml_concat(ctx, kg_v, kc_v, /*dim=*/1)` — the head axis is dim 1, so the
+    concat produces the same `[head_dim, n_head_kv, n_kv, ns]` shape the
+    non-split path returns. Backend scheduler auto-inserts the cross-backend
+    copy. Off path (no split) is unchanged.
+  - `cpy_k`/`cpy_v`: split branch slices `k_cur` by head → two `ggml_set_rows`
+    + a dependency-tying `ggml_add` so the two scatter nodes propagate in one
+    graph node. Off path unchanged.
+  - `ggml/src/ggml-cuda/ggml-cuda.cu`: **one-line fix** to CUDA `supports_op`
+    for `GGML_OP_CONCAT` — was over-broad (any non-I32/I16) while the kernel
+    `ggml_cuda_op_concat` is F32-only (three asserts at `concat.cu:158-160`).
+    Tightened to `op->src[0]->type == GGML_TYPE_F32` so F16 concat correctly
+    routes to the CPU backend (which has a real F16 path at
+    `ggml-cpu/ops.cpp:1980 concat_f16`). This is a genuine upstream bug —
+    `supports_op` was lying about what the kernel implements.
+  - Plumbing: `common/common.h` + `common/arg.cpp` add
+    `--headinfer-gpu-heads-frac F` (server example), threaded through
+    `cparams`/`llama-context.cpp`/`llama-cparams.h`/`llama-model.cpp` into the
+    cache constructor (mirroring the `--no-kv-unified` wiring template). Both
+    iswa and hybrid memory ctors forward `1.0f` (off) — they don't thread
+    headinfer themselves; the split is unified-mode only by design.
+- `@opencoti/llamafile` (`config.ts`/`launch.ts`): typed, env-backed
+  (`OPENCOTI_LLAMAFILE_HEADINFER_GPU_HEADS_FRAC`) `headinferGpuHeadsFrac` field
+  → `buildServerArgs` emits `--headinfer-gpu-heads-frac` only when set
+  (omitted = server default = off). Three new launch-args tests
+  (omit/emit/precede-extraArgs); typecheck clean, **78/0 tests**.
+
+**Bench** (`perf/llamafile/headinfer-residency.bench.ts`, live on solidPC,
+RTX 3090, Qwen2.5-Coder-0.5B-IQ4_XS, `-ngl 99 -fa on --parallel 1`, ctx 4096):
+**6/6 PASS** —
+| | baseline `frac=1.0` | split `frac=0.5` |
+| --- | --- | --- |
+| `CUDA0` KV | **48 MiB** | **24 MiB** (-50%) |
+| `CPU` KV | 0 (off-path identity) | **24 MiB** (+ host half) |
+| greedy decode (temp=0) | reference | **byte-identical to baseline** |
+
+Identical-text is the strongest correctness check possible (implies identical
+token ids; stronger than a cosine ≥ 0.999 bar). Reproducible against
+Qwen2.5-1.5B-Instruct at the same frac (linearly scaled, 112 → 56 / 56). The
+deferred-perf cost is the GPU→CPU→GPU concat bounce per step; option (a)
+two-`ggml_cpy`-into-preallocated-dest can replace concat for steady-state perf
+in a follow-up (post-M2 or `0021`).
+
+**Off-path identity:** `frac=1.0` (default, every non-opencoti client) — `k_cpu`
+is never allocated, `get_k`/`get_v`/`cpy_k`/`cpy_v` take the non-split branch;
+the binary's KV path is byte-for-byte upstream-equivalent.
+
+**Surgical-hook count stays 18** — the in-tree marker is `// opencoti F5 M2
+headinfer`, a vendored-source tag, not a counted opencode hook.
+
+**Build-infra prerequisite that surfaced (buglog `bug-178`).** Iterating M2's
+vendored C++ surfaced that an earlier `CCACHE_RECACHE=1` from-scratch rebuild
+(M1-E / `#244` lineage) had silently produced a binary with **broken C++
+exception unwinding** — any `throw` aborts with `ud2` / SIGILL. This is a
+distinct variant of `bug-176`: that defect's known symptom was a *link failure*
+(`concomitant .aarch64 file missing`); this variant **links successfully** but
+the consolidated `.eh_frame` in the APE is corrupt, so the unwinder traps —
+masked from M1-E because `--version` (the only thing M1-E verified) never
+exercises unwinding. Disambiguated from non-causes: NOT app/M2 source
+(gated-off `headinfer_gpu_heads_frac=1.0` AND pure-upstream `std::stoi` both
+trap); NOT toolchain or flags (a 5-line `throw`/`catch` compiled with the exact
+build flags via `cosmoc++` catches correctly). Fix: rebuild non-destructively
+with **`OPENCOTI_NO_CCACHE=1 bun run build:llamafile:make`** (the `make`
+subcommand calls `runMake()` which never git-resets the submodule;
+`OPENCOTI_NO_CCACHE=1` drops the ccache `CC=/CXX=` wrapper so `cosmocc` itself
+writes both fat-object arches in one invocation, no stale x86-only twin).
+**Lesson:** every llamafile build verification MUST run a real CPU inference
+or a throw-probe (`-n notanum`), never just `--version`.
+
+**Deferred / not done:**
+- **Per-stream device buffers + host-pageable** (`#109` Phase 4): the M2 plan's
+  gated extension. Gate fired — landing per-stream now would expand the
+  `cpy_k`/`cpy_v` multi-stream scatter surface (the plan's flagged hot-path
+  regression risk) with no validated need; M2-A/B/C delivered the head-residency
+  headline win on unified mode cleanly without it. **Carries to optional `0021`**.
+- **Per-head attention COMPUTE split** (run CPU-resident heads' attention on the
+  CPU backend to eliminate the per-step stream-back). The headline perf win;
+  consciously deferred per the foundation-first depth decision above. Revisit
+  only if a future bench shows the stream-back cost dominates.
+- **Quantized-K head-split** edge cases; multi-GPU head residency — later tuning.
+- **Steady-state concat optimization** (two `ggml_cpy` into a pre-allocated F16
+  destination on GPU): replaces the GPU→CPU→GPU concat bounce with a single
+  CPU→GPU copy + a GPU-side memcpy. Larger surgery, post-M2 follow-up.
+
+### M3 — NEO (asymmetric GPU/CPU attention pipelining) *(shipped 2026-05-29)*
+
+**Reference:** arXiv 2411.01142, Apache-2.0 at
+[github.com/NEO-MLSys25/NEO](https://github.com/NEO-MLSys25/NEO).
+Reference is swiftLLM (PyTorch). Like M2's lightseek precedent,
+**not directly portable** — NEO ports the structural split + concurrent
+dispatch concept into ggml's decode loop rather than wrapping the
+upstream code. The published design covers three pieces (per-head
+COMPUTE split + async overlap + load-aware scheduling); M3 bundled
+all three into one `0030-neo-pipeline.patch` per locked decision
+(absorbing the `0021` "per-head compute split" reservation from M2).
+
+**Depth decision:** full structural-split + orchestrator + load-aware
+controller was the planned scope; the orchestrator and structural
+split shipped; the EWMA load-aware controller (M3-C) was **deferred**
+after the M3-D gate fired (see "What shipped" below).
+
+**Patch:** [`vendors/patches/llamafile/0030-neo-pipeline.patch`](../../vendors/patches/llamafile/0030-neo-pipeline.patch).
+Sits on top of `0020` (M2 head-residency split). The historical
+"per-head compute split" reservation at `0021` was absorbed by `0030`
+(M3 bundled it in). **F4 M3 Phase 4** (per-stream KV tensor split,
+SHIPPED 2026-05-29 — see `docs/decisions/0001-lazy-slot-context.md
+§Phase 4 — shipped notes` and the patches README row for
+`0031-per-stream-split`) lands at `0031`, *after* `0030`, because
+Phase 4-D lifts M2's dedicated `get_k_gpu`/`get_v_gpu` accessors that
+`0030` introduces for NEO's two-FA dispatch. The originally-reserved
+`0021` slot stays vacant.
+
+**What shipped:**
+
+- **M3-A structural two-FA split.** New `llm_graph_context::build_attn_mha_neo`
+  in `llama.cpp/src/llama-graph.cpp` is invoked from `build_attn`'s
+  flash-attn branch when `cparams.neo_pipeline_mode != 0 &&
+  mctx_cur->headinfer_split_active(il)`. It slices Q on dim 1 at the
+  GQA-clamped `gpu_heads * gqa_ratio` boundary (a clean `ggml_view_4d`
+  with no extra permute), slices mask and sinks correspondingly, builds
+  TWO `ggml_flash_attn_ext` ops on disjoint head ranges, and concats
+  the outputs on the head axis. The K/V halves come from new
+  `llama_memory_context` accessors `get_k_gpu`/`_cpu`/`get_v_gpu`/`_cpu`
+  + `headinfer_split_active(il)` (added in `llama-memory.h`,
+  implemented in `llama-kv-cache.{h,cpp}`, delegated through
+  `llama_kv_cache_context` wrappers). The scheduler auto-routes each
+  FA op to the backend hosting its K/V views (CUDA for `cur_g`, CPU
+  for `cur_c`) — no scheduler surgery needed. This **eliminates M2's
+  per-step `O(n_kv × head_dim_cpu × 2 B)` CPU→GPU stream-back** and
+  replaces it with an `O(n_tokens × head_dim_cpu)` bounce on the FA
+  output. The original `get_k`/`get_v` paths are unchanged so
+  `--neo-pipeline off` runs the M2 concat path byte-identically.
+
+- **M3-B NEO orchestrator (wired-but-no-observable-win).** New vendored
+  source files `vendors/sources/llamafile/llama.cpp/ggml/include/ggml-neo-pipeline.h`
+  + `ggml/src/ggml-neo-pipeline.cpp` provide a registry of `(cur_g, cur_c)`
+  pairs + a `cudaEvent` slot per `cur_g`. The graph builder calls
+  `ggml_neo_pipeline_register_pair(cur_g, cur_c)` after constructing each
+  pair. The CUDA backend's `ggml_backend_cuda_graph_compute` consults the
+  registry: if the split's first node is a registered `cur_g`, it sets
+  `cuda_ctx->curr_stream_no = 1` so all kernels dispatch on the alternate
+  CUDA stream, records a `cudaEvent` on stream 1 at end-of-graph, then
+  restores `curr_stream_no = 0`. When a downstream split reads a
+  registered `cur_g` (typically the concat join), the hook inserts a
+  `cudaStreamWaitEvent(cuda_ctx->stream(), event, 0)` so the consuming
+  stream waits for the stream-1 work. CUDA graph capture is disabled for
+  NEO-engaged splits; off-path graphs keep the fast `cuda_graph` path.
+
+- **M3-D adapter.** `--neo-pipeline off|on|auto` in `common/arg.cpp`
+  (env: `LLAMA_ARG_NEO_PIPELINE`). `cparams.neo_pipeline_mode` field
+  plumbed through `include/llama.h`, `common/common.{h,cpp}`,
+  `llama-cparams.h`, `llama-context.cpp`. `@opencoti/llamafile` typed
+  field `neoPipelineMode: "off"|"on"|"auto"|undefined` + env
+  `OPENCOTI_LLAMAFILE_NEO_PIPELINE` mirror the M2 frac template; 3 new
+  launch-args tests pass (13 total).
+
+- **Build system.** New `ggml-neo-pipeline.cpp` listed in
+  `llama.cpp/BUILD.mk` `GGML_SRCS_CPP` (for the main binary) and in
+  `llamafile/build-functions.sh` `compile_ggml_core` source list (for the
+  CUDA DSO). Both rebuild cleanly; 14 `ggml_neo_pipeline_*` symbols
+  exported from `ggml-cuda.so`.
+
+**Bench numbers — `perf/llamafile/neo-pipeline.bench.ts`** (solidPC RTX 3090
++ Qwen2.5-Coder-0.5B IQ4_XS, ctx=4096, predict=128, frac=0.5):
+
+| Run | tps | Δ vs M2 |
+| --- | --- | --- |
+| B baseline (no split) | 326.69 | +75% |
+| M m2-only (split, M2 path) | 186.96 | — |
+| **N neo-on (split, M3 path)** | **187.04** | **+0.04%** |
+| O off-identity (`--neo-pipeline off`) | 151.65 | — |
+
+Hard checks:
+- **C1 PASS** — N completion === B completion (correctness intact).
+- **C2 FAIL** — N tps - M tps < 5% (observed 0.04%).
+- **C3 PASS** — N GPU KV (24 MiB) === M GPU KV (24 MiB) — residency preserved.
+- **C4 PASS** — O completion === M completion (`--neo-pipeline off` byte-identical).
+
+Soft check S1 (NEO ≥ 80% of baseline tps): **WARN** at 57.3% — the
+CPU FA on half-heads is the wall-time bottleneck; no overlap can
+recover the baseline because the baseline runs zero CPU work.
+
+**Why C2 failed:** the M3-B stream-swap mechanism is correct (M3-A
+`cur_g` is in fact dispatched on CUDA stream 1, downstream consumers
+correctly wait on the stream-1 event, `cur_c` runs concurrently on
+the CPU backend, the GPU FA finishes long before the CPU FA), but
+for Qwen-Coder-0.5B at ctx=4096 the per-step GPU FA on half-heads
+finishes in single-digit μs while the CPU FA on half-heads takes
+~10× longer. Total wall time is bounded by `max(GPU FA, CPU FA) + concat`
+which is dominated by the CPU FA. Even at long context (ctx=16384,
+predict=256 with a 4480-char prompt) the speedup stayed marginal
+(+1.4%, still below C2's 5% threshold). The overlap WOULD pay off
+on workloads where GPU FA cost approaches CPU FA cost — multi-GPU
+splits, much larger models with smaller `frac`, or quantized-K
+paths where the GPU side does more work per head. M3-B is correct
+infrastructure for those regimes and for M4 (ScoutAttention) to
+reuse.
+
+**Per the M3-D gate decision:** C2 failed but the orchestrator is
+**correct** (no crashes, no deadlocks, C1+C3+C4 pass at the
+short-ctx bench) and the off-path is byte-identical to the
+M2-shipped binary. The orchestrator ships **wired-but-no-observable-win**
+(not dormant-stub as the original plan text described) — the CUDA
+hook is live and registered pairs route to stream 1 — but the
+mechanism produces no measurable throughput improvement on this
+hardware/model. The M3-A structural win (eliminating the per-step
+CPU→GPU K/V stream-back) is the actual M3 deliverable.
+
+**M3-C (EWMA load-aware controller) deferred.** The plan called for
+a per-layer `neo_layer_state` measuring GPU/CPU FA times and
+adjusting `gpu_heads_active` toward load balance. Without an
+observable M3-B overlap win this is cosmetic — there's nothing for
+the controller to optimize toward. Revisit when M3-B's mechanism
+finds a regime where it actually wins (multi-GPU, much larger
+models, quantized-K), or when M4 ScoutAttention reuses the same
+orchestrator and the load-balance trade-off becomes meaningful.
+
+**Off-path identity proven.** M2's `perf/llamafile/headinfer-residency.bench.ts`
+re-runs **6/6 PASS** against the M3-built binary — the M2 concat
+path is byte-equivalent to the 0020-shipped binary. `0030` does not
+regress M2. Surgical-hook count stays **18** (all M3 changes are
+in vendored llama.cpp source captured into `0030`, not in upstream
+opencode `packages/`).
+
+**Build / capture rules carried forward:** `OPENCOTI_NO_CCACHE=1
+bun run build:llamafile:make` for the main binary; `bun run
+build:llamafile:cuda` for the CUDA DSO (~24 min build on solidPC).
+Patch captured via **snapshot-diff** under `.opencoti/snap-m3-pre/`
+(NOT `git diff HEAD`; bug-121). Apply-tested clean against the
+snapshot. Vendored submodule's dirty state intentionally not staged
+on the substance commit.
+
+**Deferred / non-goals (carry forward):**
+- **M3-B win on this hardware** — see Why C2 failed; revisit when
+  workload reshapes.
+- **M3-C EWMA load-aware controller** — see above.
+- **Multi-GPU NEO** (orchestrating multiple CUDA devices alongside
+  CPU) — single-CUDA-device + CPU is the current scope.
+- **Vulkan / ROCm backends** — CUDA-only; the orchestrator's CUDA
+  hook is unreachable on other GPU backends, so M3-A's structural
+  split runs without M3-B's stream-swap there.
+- **Per-stream device buffers + host-pageable** (#109) — still
+  carries from M2 / F4 Phase 4.
+- **Persistence of EWMA state across server restarts** — moot since
+  M3-C deferred.
+- **Reallocating `gpu_heads` mid-decode** — the runtime controller
+  is deferred; physical `gpu_heads` is still ctor-set.
+
+### M4 — ScoutAttention (layer-ahead CPU pre-compute) *(deferred 2026-05-29)*
+
+- **Reference:** arXiv 2603.27138 — paper published 2026-03-28, accepted
+  DAC '26 (July 26-29, Long Beach). Authors: Qiuyang Zhang, Kai Zhou,
+  Ding Tang, Kai Lu, Cheng Li, Zhenyu Yang, Peng Xu, Jiguang Wan (HUST +
+  Huawei + Zhejiang Lab).
+- **Reference-code status (verified 2026-05-29):** **ABSENT.** Authoritative
+  search via `gh search repos`/`gh search code` for `ScoutAttention`,
+  `scout-attention layer-ahead`, `layer-ahead pre-computation`,
+  `asynchronous periodic recall sparse attention` returns zero
+  implementation hits. HF Hub model + space search empty. The full paper
+  HTML has no GitHub/GitLab/artifact link and no code-availability
+  statement. Authors are HUST + Huawei + Zhejiang Lab — academic
+  publication, code not advertised.
+- **Algorithmic substance (from paper Algorithm 1, recorded so a future
+  picker-up has the spec cold):**
+  1. Predict next-layer query `Q_pred^(i+1) ← W_Q^(i+1) · X^i` (residual-
+     similarity trick, cosine > 0.93 across Gemma 3, Llama 3.1, Mistral,
+     GLM 4 tested in the paper)
+  2. Block-wise top-k: `B_pred^(i+1) ← TopK(Q_pred^(i+1) · K_digest^(i+1)^T)`
+  3. Identify CPU-resident blocks: `B_cpu^(i+1) ← B_pred^(i+1) \ B_gpu^(i+1)`
+  4. Async spawn `CPUAttn(B_cpu^(i+1))`
+  5. GPU runs full layer i (attn + FFN + QKV projections) overlapped
+  6. `A^i ← Merge(A_gpu^i, A_cpu^i)` via FlashAttention's online-softmax
+     merge (the previously-spawned CPU work joins here)
+
+  Plus asynchronous periodic recall (β = 12% threshold → avg ~8.7 layers
+  between recalls, CPU compute ratio ~8.2%). Hyperparameters tested: block
+  size 32 default, sparse budget 1024-2048, ctx up to 64k, batches 16-64,
+  Qwen3-8B/14B/32B.
+- **Why portability is harder than M2/M3.** Paper ships on SGLang +
+  FlashInfer + IPEX. M2 (lightseek) and M3 (NEO/swiftLLM) were paper-ports
+  onto **existing** ggml primitives. M4 requires **new primitives**:
+  a block-wise top-k CUDA kernel (FlashInfer-equivalent), a block-sparse
+  variant of `ggml_flash_attn_ext`, a `K_digest` side tensor + its
+  computation kernel, and an IPEX-equivalent CPU worker accepting a
+  selected-blocks input (llama.cpp's `flash_attn_ext` CPU kernel does
+  full-K/V only). Multi-week kernel work with C1 byte-identity risk on
+  every downstream check.
+- **Decision (user 2026-05-29):** defer until reference code surfaces.
+  Re-evaluation August 2026 (post-DAC). Progression updates to M5 Glue
+  ("compose-validate M0+M1+M2+M3", M4 removed from the compose set until
+  it lands) → M6 PolyKV (has reference code) → revisit M4.
+- **If/when M4 lights up later, options remain on the table:** (a)
+  reimplement from spec, (b) substitute a simpler layer-ahead
+  K_pred-prefetch-only approximation that skips the block-sparse FA
+  refactor, or (c) compose with M3's NEO orchestrator (run NEO's GPU
+  half dense, run the CPU half block-sparse) — the registry surface
+  added in `0030-neo-pipeline.patch` is the natural integration point.
+- **Config flag (when M4 ships):** `--scout-attention on|off` (default off).
+
+### M5 — Glue + bench harness *(shipped 2026-05-29)*
+
+**Goal.** Compose-validate the F5 stack — M0 (session-keyed KV reuse) +
+M1 (retention-aware eviction) + M2 (head-residency split) + M3 (NEO
+concurrent FA). Each prior milestone landed its own bench in isolation;
+M5 is the first that lights all four up simultaneously. M4 is removed
+from the compose set per its 2026-05-29 deferral.
+
+**What shipped.**
+
+- `perf/llamafile/_bench-lib.ts` — first shared helper module under
+  `perf/llamafile/` (293 lines). Exports `resolveBin`, `resolveModel`
+  (`"qwen" | "gemma" | "auto" | <abs-path>`), `pickPort`, `probeGpu`
+  (enforces the 2026-05-28 Cerebrum free-VRAM rule), `spawnServer`
+  (returns `{baseURL, stop, log}`; `stop()` is the bug-191
+  SIGTERM→2 s race→SIGKILL reap with `await proc.exited` in both
+  branches), `waitForHealth`, `metric`, `parseKvBuffers`,
+  `callCompletion`, `tps`. Underscore prefix keeps the file out of
+  `*.bench.ts` glob runners. **Existing four benches stay on their
+  inlined helpers** — no regression risk to landed work; future
+  benches (M6 PolyKV onwards) inherit the lib.
+
+- `perf/llamafile/advanced-kv-stack.bench.ts` — the compose bench
+  (510 lines). Six configurations × one multi-turn needle workload ×
+  9 hard checks + 2 soft checks. CLI: `--model qwen|gemma|<abs-path>`
+  defaulting to auto→qwen for fast CI; `--ctx`, `--predict`,
+  `--filler`, `--frac` for one-off tuning. Self-skips on no bin /
+  no model / no GPU+DSO / insufficient free VRAM. Results JSON to
+  `.opencoti/m5-stack-bench.json`.
+
+**Configurations.**
+
+| ID | Flags                                                                                                | Tests                       |
+|----|------------------------------------------------------------------------------------------------------|-----------------------------|
+| B  | `-ngl 99 -fa on --parallel 1 -c 1024`                                                                | Baseline                    |
+| A0 | B + `--kv-unified --parallel 2 -c 2048 --cache-reuse 256`                                            | M0 alone                    |
+| A1 | B + `--rest-kv-eviction --rest-kv-recent 256 --rest-kv-layer -1`                                     | M1 alone                    |
+| A2 | B + `--headinfer-gpu-heads-frac 0.5`                                                                 | M2 alone                    |
+| A3 | B + `--headinfer-gpu-heads-frac 0.5 --neo-pipeline on`                                               | M3 alone                    |
+| S  | B + ALL of A0+A1+A2+A3 simultaneously (parallel=2, ctx=2048, kv-unified, all four flag groups)       | **The compose validation**  |
+
+**Live bench (solidPC RTX 3090, Qwen2.5-Coder-0.5B IQ4_XS, ctx=1024 per
+slot, n_predict=256, filler=35, frac=0.5):**
+
+```
+B  baseline    : t1_tps=328  cuda_mib=12  (per cell 0.0117 KiB)
+A0 M0 alone    : t1_tps=350  cuda_mib=24  (per cell 0.0117 KiB; -kv-unified at parallel=2)
+A1 M1 alone    : t1_tps=337  cuda_mib=12  (per cell 0.0117 KiB)
+A2 M2 alone    : t1_tps=159  cuda_mib=6 + cpu_mib=6  (per cell 0.0059 KiB — 50% saving)
+A3 M3 alone    : t1_tps=160  cuda_mib=6 + cpu_mib=6  (NEO on; identical residency to A2)
+S  STACK       : t1_tps=159  cuda_mib=12 + cpu_mib=12  (per cell 0.0059 KiB — M2 saving preserved)
+
+Hard checks: 9/9 PASS  (C3, C7 cleanly SKIPPED — workload didn't force
+                       context shift on this model + ctx + filler combo;
+                       the bench mirrors M1's skip-not-fail pattern.)
+Soft checks: 2/2 PASS
+```
+
+**Key findings.**
+
+1. **M1 + M2 compounding cost is negligible.** Phase-1 explore flagged
+   a soft compose risk — M1's `seq_key_scores` dequantizes K rows at
+   context-shift; M2's `get_k` concatenates `k_cpu` on every step. The
+   bench measured **S t1_tps 158.55 vs A2 alone 158.90 — within noise**
+   (soft check W1 PASS). No follow-up needed.
+
+2. **M2's per-cell GPU saving is exactly preserved in the stack.** Soft
+   check W2: S per-cell CUDA = A2 per-cell CUDA = 0.0059 KiB/cell. The
+   M2 split is identically engaged in S as in A2 alone.
+
+3. **`--kv-unified` must be explicit when `--parallel > 1`.** First
+   compose-conflict surfaced: llamafile's `llama_context_params`
+   defaults `kv_unified = false`, so `--parallel N > 1` → `n_stream = N`,
+   and M2's split gate (patch 0020 line 176, `n_stream == 1`) silently
+   fails. M0 + M2 wouldn't have composed otherwise. The bench encodes
+   the fix: A0 and S pass `--kv-unified` explicitly. **This is exactly
+   the kind of finding M5's compose-validation exists to surface.**
+   A future M5 follow-up could make `--kv-unified` the default when
+   `--headinfer-gpu-heads-frac < 1.0`, or document a hard error if both
+   `--parallel > 1` and frac are set without `--kv-unified`. Tracked as
+   a soft TODO in the bench's config block.
+
+4. **M3 stack equality.** Hard check C5: A3 turn-1 completion ===
+   A2 turn-1 completion (byte-identical). The 0030 patch's off-or-on
+   structural equality on M2 is preserved when other flags are on.
+
+5. **No pathological combined slowdown.** C9 floor at 40% of baseline
+   (A2 alone runs at 48% of B — the cost of M2's CPU stream-back).
+   Stack tps lands at A2 alone's level (158 vs 159 — no additional
+   M0/M1/M3 cost stacked on M2's floor).
+
+**Deferred / non-goals.**
+
+- **Helper-module extraction in M0/M1/M2/M3 benches.** Per the M5
+  scope-locking decision, leave the four existing benches on their
+  inlined copies to protect landed work. Follow-up task can refactor
+  them when convenient; the lib API was designed compatible.
+- **Force shift on Qwen workload.** Tuning `(filler, predict)` to
+  cross `n_ctx` mid-decode under Qwen2.5-Coder IQ4_XS hit a BPE
+  merge-breakpoint at filler ≈ 36-37 that jumps prompt tokenization
+  from ~750 → ~1050. Filler=35 leaves prompt at ~740 + decode 256 =
+  996 < 1024 — under the shift threshold. C3 / C7 skip cleanly
+  rather than false-fail; override with `--filler N --ctx N
+  --predict N` from the CLI for one-off shift-forced runs. The
+  M1 bench already exercises shift in isolation, so M5's skip is
+  fine for compose-validation.
+- **CI runner / fixed GPU image** (the F2 M6 dependency from the
+  original M5 sketch) — out of scope; the bench self-skips when no
+  GPU is present, so it's CI-safe today.
+- **Make `--kv-unified` the auto-default when M2 is engaged.** Could
+  land in a tiny `0030`-tier follow-up patch; deferred until either
+  M6 PolyKV or a user request makes it worth touching the cparam
+  wiring.
+- **Gemma profile run.** Bench supports `--model gemma` (probes for
+  14 GB free VRAM); Qwen-default run is the CI path. Realistic-perf
+  Gemma numbers can land in a future one-off study.
+
+**Hook count unchanged** (`git grep "opencoti-hook:" packages/ docs/`
+returns 19 — same as before M5, where the 19th is a docs-text mention
+from the M3-H ship, not a real new surgical hook). No new marker in
+opencode source, no llama.cpp source change, no patch.
+`0050-glue-bench.patch` reservation slot in
+`vendors/patches/llamafile/README.md` stays empty — M5 is pure
+application-side perf harness.
+
+### M6 — PolyKV (shared compressed KV pool)
+
+> **Child plan: [poly_kv.md](poly_kv.md)** — **M6 SHIPPED 2026-06-06.** Staged fast-first: **S0**
+> asymmetric KV-compression via stock `-ctk q8_0 -ctv q4_0` through M7's dequant-on-lift (zero new
+> kernel, banked correct at 4k) → **S1** shared read-only prefix pool (multi-tenant core, O(1) in
+> agents, 6.9× fan-out memory win) → **S2** TurboQuant `TURBO{2,3,4,8}_0` family + InnerQ + Level-A
+> dequant-on-lift + Level-B fused FA-VEC (`GGML_OP_TURBO_WHT`; 4/4 tiers logit-equiv) → **S3** glue
+> bench + compose gate + patches `0072`/`0073` (captured + byte-identical-proven). Key insight: M7's
+> dequant-on-lift (S3d) lets opencoti decompress the pool **in-attention**, a win the PyTorch
+> reference (decompress-then-attend) can't get. Remaining: **S4** MTP draft head (#373).
+
+- Reference: arXiv 2604.24971, MIT at
+  [github.com/ishan1410/PolyKV](https://github.com/ishan1410/PolyKV).
+  Single asymmetrically-compressed KV pool across multiple
+  concurrent agents; ~97.7% memory reduction at 3–4+
+  concurrent agents.
+- **Sequenced immediately after M5**, not deferred. Reasoning:
+  deferring loses the implementation context from the
+  preceding patch series — the same code paths (slot KV
+  management, allocator hooks, attention kernel
+  modifications) get re-visited with cold context if PolyKV
+  lands in a separate work unit later. Keeping it in this
+  work unit reuses the mental model. The fact that opencoti's
+  multi-agent fan-out on Tier 0 isn't shipped yet is *not* a
+  blocker — PolyKV is a memory-pool architecture that can
+  land in single-agent mode (degenerates to a 1-tenant pool)
+  and light up its full value once fan-out arrives.
+- Smoke at M6: single-tenant correctness. Multi-tenant bench
+  follows whenever multi-agent fan-out lands.
+- Compose-with-glue: re-run M5's `advanced-kv-stack.bench.ts`
+  with PolyKV on the stack; document any conflict resolution.
+
+### M7 — LMCache *(DEFERRED)*
+
+- Reference: arXiv 2510.09665, Apache-2.0 at
+  [github.com/lmcache/lmcache](https://github.com/lmcache/lmcache).
+  Cache offloading + prefix-decode disaggregation; modular
+  connector for vLLM/SGLang.
+- **Deferred because**: vLLM/SGLang-bound architecture,
+  llama.cpp portability 1/5. Re-evaluate when M0–M6 are
+  landed and there's a clear marginal-win case.
+- **Superseded 2026-05-29**: the M7 slot is now **Rolling KV**
+  (streaming double-buffered KV pipeline, M2-as-runtime-tactic),
+  designed in [`rolling_kv.md`](rolling_kv.md), patch `0070`. LMCache
+  remains an external reference only.
+
+## F5-opt — concurrency=1 optimization round *(task #287)*
+
+> **RESOLVED 2026-06-04 — superseded by M7.** A fresh concurrency=1 re-baseline of the
+> **shipped M7 binary** (default `--kv-residency-mode auto`) vs vanilla-0.10.1 on the
+> prefill-bound regime that defined this task lands at **parity**: prefill 4k/32k ratio
+> 1.00/1.01, decode 1.03/0.94 — all in [0.94, 1.03]. The old worst case (niah@32k prefill,
+> **3.6× = 0.28**) is now **1.01**. The cause is structural: M7's GPU_RESIDENT-by-default
+> retired the always-on M2 CPU-half this round was created to optimize (server log:
+> `GPU_RESIDENT=5/25, POSITION_WINDOW=0, CPU_FA_TAIL=0` at every tested ctx). The lone
+> residual — decode@32k 0.94 (~6%, per-layer tactic-table bookkeeping on the resident path)
+> — is a future micro-opt, not a blocker. Full matrix + artifact note:
+> [`.opencoti/m7-c1-rebaseline/RESULT.md`](../../.opencoti/m7-c1-rebaseline/RESULT.md).
+> The four workstreams below shipped en route and remain the building blocks (W1 probe,
+> W2 CPU-FA fallback, W3 fused-MoE, W4 the Rolling KV residency redesign itself).
+
+The 2026-05-29 RULER F5 comparison held quality at 100% (4k/32k) but ran
+2.6–4.7× slower than vanilla at concurrency=1, dominated by the M2
+CPU-half. Four workstreams attack compute / transfer / residency before
+the deferred RULER 256k re-run (see the program plan). Shipped so far:
+
+- **W1 (`0036`, #293) — PCIe/ReBAR probe consumption.** Boot-time
+  `pcie_profile` reader + `--pcie-autodetect`/`--pcie-bw-gbps`; feeds W4's
+  tile sizing. SHIPPED 2026-05-30.
+  **Measured proof (#367, 2026-06-04, RTX 3090, `perf/llamafile/rebar-probe.cu`).**
+  Pinned vs pageable host↔device bandwidth (GB/s), pinned/pageable ratio:
+
+  | transfer | pinned H2D/D2H | pageable H2D/D2H | pinned/pageable H2D · D2H | ReBAR verdict |
+  |----------|----------------|------------------|---------------------------|---------------|
+  | 1 MiB    | 6.397 / 6.361  | 5.545 / 4.437    | **1.154× · 1.434×**       | partial/degraded |
+  | 16 MiB   | 6.656 / 6.590  | 6.400 / 6.334    | 1.040× · 1.040×           | active |
+  | 64 MiB   | 6.669 / 6.593  | 6.537 / 6.468    | 1.020× · 1.019×           | active |
+
+  Link = **x8 @ 8.0 GT/s (PCIe 3.0 x8**; iGPU absorbs 8 of 16 lanes), ReBAR active
+  (max x16 @ 16.0 GT/s). The pinned win is **size-dependent**: large at small,
+  latency-bound transfers (+15% H2D / +43% D2H at 1 MiB), negligible once the x8 link
+  saturates at 64 MiB (~6.67 GB/s, the cached `effective_bw_gbps`). This validates the
+  **#289/M7-A pinned-host KV residency**: M7 streams the spill tail in *small* tiles —
+  exactly the regime where pinned staging earns +15–43%, not the 2% a bandwidth-bound
+  read would suggest. `effective_bw_gbps` (pinned plateau, 6.67) is the M7 tile-sizing
+  input (`tile_bytes_max = compute_ms × eff_bw × 0.8`). Raw JSONs:
+  `.opencoti/rebar-probe-solidpc-dev0-{1,16,64}MiB-20260604-*.json`.
+- **W2 (`0040`, #290) — wholesale ik_llama.cpp CPU-FA engine.** Vendored
+  behind `GGML_IQK_FLASH_ATTENTION` + `--iqk-flash-attn on|off` (default
+  off) + one dispatch hook (`opencoti-hook: f5-opt-cpufa`). SHIPPED
+  2026-05-30. **Verification:** flag-OFF byte-identical (neo-pipeline C4
+  PASS); flag-ON RULER vt+niah @ 4k/32k (Gemma-4 A4B + M2 + full stack)
+  100% all 8 cells, **ON-vs-OFF wall-time flat**. The flat result is the
+  key finding: on prefill-bound RULER cells the CPU-FA *kernel speed* is
+  not the lever — **the CPU split existing at all while VRAM is free** is.
+  That motivated the M7 **GPU_RESIDENT-by-default / maximize-VRAM**
+  redesign (`rolling_kv.md` Decision 4): start GPU-resident, engage
+  CPU/stream relief only under VRAM pressure, release it as load drains.
+  W2 stays valuable as the CPU-spill *fallback* kernel. **#291 Q8_KV
+  DROPPED 2026-05-30**: recon found it needs a full mainline-ggml-type
+  registration (it is the pseudo-type `(ggml_type)151` inside the engine,
+  not a real type), not the "nearly-free" flip assumed — and it is a
+  redundant 8-bit option (`q8_0` covers it; **TurboQuant** TBQ3_0/4_0 in M6
+  is the aggressive-compression KV-quant driver). `0041` retired; the
+  engine's dormant Q8_KV FA path stays gated off at zero cost.
+- **W3 (`0042`, #292) — op-level fused MoE up+gate+GLU. SHIPPED 2026-05-30.**
+  New ggml op `GGML_OP_MOE_FUSED_UP_GATE` collapses a MoE FFN's two separate
+  per-expert projections (`up_exps @ cur`, `gate_exps @ cur`) + the GLU into
+  ONE decode-time op, dispatched to our **existing** fused mmvq kernel
+  (`ggml_cuda_mul_mat_vec_q` + `fusion.gate`/`glu_op`) — eliminating the
+  gate_up HBM round-trip + the standalone GLU launch per MoE layer per token.
+  **Zero ik_llama code vendored** — only the op-level-fusion *idea*
+  (ik_llama PR #229/#520, MIT). Off by default (`--fused-moe-up-gate on`).
+  **Scope discovery:** the hook only fires for **separate-up/gate** layouts
+  (`gate_exps && !gate_up_exps`, same type, SILU\|GELU, decode `n_tokens==1`)
+  = **Qwen2-MoE / Qwen3-MoE / OLMoE / Mixtral-style**. The original target
+  **Gemma-4 A4B already fuses gate+up into one `ffn_gate_up_exps` matmul**
+  (`gemma4-iswa.cpp:158` passes `gate/up = nullptr`), so W3 is **N/A to
+  Gemma-4** — it gains nothing (only a cheap elementwise GLU remains, <1%
+  ceiling, confirmed by flat OFF/ON tok/s on Gemma-4). CPU backend aborts the
+  op (CUDA-only). **Validated on OLMoE-1B-7B Q4_K_M** (nsys can't trace
+  llamafile's `dlopen`'d CUDA → engagement via `GGML_SCHED_DEBUG=2 --verbose`
+  graph dump: 34 op-nodes on CUDA0 ON / 0 OFF): greedy decode **byte-identical**
+  OFF vs ON, **+2.4% decode tok/s** (265.9→272.2). ABI `GGML_OP_COUNT` 96→97
+  (binary + `ggml-cuda.so` rebuilt & paired). Adapter `--fused-moe-up-gate`
+  after `--iqk-flash-attn` (87 tests).
+- **W4 (`0070`, #296) — M7 Rolling KV — IN FLIGHT (2026-05-31).** The
+  residency/transfer redesign the W2 data points to. Shipped on `dev` in
+  cosine-gated rungs: **Rung 0** (M7-A auto-residency — GPU_RESIDENT when KV fits
+  the VRAM budget, spill only under pressure) + **Rung 1** (M7-B runtime
+  `layer_tactic[]` table + `rolling_kv_plan`, populated/logged, inert on the
+  compute path) + **Rung 2 R2-a** (the `GGML_OP_STREAMING_FLASH_ATTN` op + the
+  `get_layer_tactic(il)==GPU_STREAM` dispatch shell — CUDA forward reuses
+  `ggml_cuda_flash_attn_ext`, byte-identical to GPU_RESIDENT/CPU_SPILL). Canonical
+  `0070` captured (47 hunks / 19 files, 5 `f5-rolling-kv` markers; bug-250
+  patch-chain blocker fixed first, #309). **The streaming kernel proper is still
+  unwritten** — R2-a is only the op-dispatch shell (`n_slots == 0`, KV still
+  device-resident, no tile loop). Next: pinned-host residency + slot pool + tile
+  loop + *inter-tile* online-softmax via `flash_attn_combine_results` (M7-C/M7-D),
+  then double-buffer overlap (R2-b) + scheduler (R2-c), then M7-F/G/H. See
+  [`rolling_kv.md`](rolling_kv.md) "Rung 2 implementation blueprint." bug-226 not
+  yet superseded.
+
+## Surgical-hook footprint
+
+- Zero hooks in the opencode source tree from F5.
+- Each milestone adds 1 (or 2) new patch files under
+  `vendors/patches/llamafile/`. The patch protocol header is
+  enforced by `vendors/patches/llamafile/README.md`.
+
+## Open questions
+
+- **ReST-KV reference choice.** Resolved at M1 start: pick the
+  cleanest retention-scoring paper-or-code combination
+  available at that time. SnapKV and GraphKV are both live
+  candidates if Make-Each-Token-Count remains code-less.
+- ~~**ScoutAttention publication date.**~~ **Resolved 2026-05-29.** Paper
+  published 2026-03-28; reference code confirmed absent at M4-start.
+  Decision: defer M4 to August 2026 post-DAC. See M4 section above.
+- **PolyKV ↔ HeadInfer conflict.** Research suggests these
+  conflict on shared-pool vs per-head residency assumptions.
+  Resolution path is M6's compose-with-glue step; if the
+  conflict is irreconcilable, document it and let users pick
+  one or the other via flags.
+
+## Risks
+
+- **Each patch starts against the pinned upstream SHA; later
+  patches modify the same source files.** Lexical apply order
+  is enforced by the `NNNN-` prefix protocol. Gaps in the
+  numbering leave room for per-technique tweaks without
+  renumbering downstream patches.
+- **M4 (ScoutAttention) has no public reference code.** Riskiest
+  milestone. Same escalation path as above.
+- **Llama.cpp portability scores ≤ 3/5 for all six techniques.**
+  Each technique is a port-from-PyTorch-or-vLLM, not a direct
+  drop-in. Estimate scope generously when picking the next
+  milestone after M0.
+- **Multi-tenant value of PolyKV doesn't land until Tier 0
+  multi-agent fan-out ships.** M6's M5 compose-step assertion
+  is the contract; the bench is only meaningful once concurrency
+  > 1.

docs/features/fused_nextn_mtp.md

@@ -0,0 +1,272 @@
+# Fused NextN draft graph (task #590 / bug-858, bug-867)
+
+**Status:** scoped — design only, not built. This document is the implementation
+blueprint; the go/no-go on building it is a separate decision.
+
+**Problem.** Qwen NextN self-speculative MTP (`--spec-type draft-mtp`) on 35B-A3B
+decodes ~16% slower than upstream llama.cpp `b9859` on the *same* GGUF/HW, with
+**acceptance and cycle count at parity** (ours 0.79–0.80 accept, 307 draft_n, 244
+accepted @ 244 t/s; upstream 0.815, 303, 247 @ 283 t/s). Base single-decode is at
+parity. The gap is therefore **per-cycle host-sync/launch overhead**, not drafter
+quality and not verify criterion — see `docs/evaluations/three-way-tps.md` §"#590
+diagnostic" and buglog bug-864/866/867. The overhead lives in the **un-fused
+per-step draft loop**: `common_speculative_impl_draft_mtp::draft`
+(`common/speculative.cpp:702-803`) does, *per drafted token*, a
+`llama_decode(ctx_dft)` + a `llama_get_embeddings_pre_norm_ith` **host readback**
+(`:751-752, :796`) — one launch + one D2H sync per step.
+
+**Fix.** Give NextN a **fused draft graph**: the NextN analogue of the
+Gemma-assistant `decode_mtp_fused` (`src/llama-context.cpp:2900`). Build the N draft
+steps as *one* graph with an on-device argmax→embed→hidden chain between steps, so
+the whole block costs **one launch + one readback** (`get_argmax()` I32[N] +
+`get_embd()`), eliminating the per-step host round-trip. Payoff scales with n_max
+(modest at the n=2 cap, expected to fix the n=3 collapse).
+
+`decode_mtp_fused` is **not reusable as-is** — it is hard-wired to Gemma
+(`model.mtp_assistant` at `:2908`; `dynamic_cast<llama_kv_cache_iswa>` at `:2916`).
+The NextN drafter is a *separate* draft context (`ctx_dft`) with its own model, its
+own **unified** (non-iSWA) KV cache, and its own NextN block. So this is a
+from-scratch build that *reuses the scheduler/readback machinery* but adds a new
+Qwen fused builder + a new driver.
+
+---
+
+## The two MTP lanes (why reuse is partial)
+
+`src/llama-graph.h:37-39` graph types:
+
+| type | drafter | context | KV cache | fused today? |
+|---|---|---|---|---|
+| `LLM_GRAPH_TYPE_MTP` | Gemma-4 **assistant** (sub-model in target ctx) | target | iSWA (shared) | **yes** (`decode_mtp_fused`) |
+| `LLM_GRAPH_TYPE_DECODER_MTP` | Qwen **NextN** (own block in draft model) | separate `ctx_dft` | unified | **no** (per-step loop) |
+
+The Gemma assistant shares the *target's* iSWA KV via cross-attention
+(`build_attn_mtp`, `src/llama-graph.cpp:2977`); Qwen NextN runs a full draft context
+with its own unified KV (`src/llama-model.cpp:1979-1995`) and reuses the trunk
+layer's weights for its single NextN block (`src/models/qwen35.cpp:510-656`,
+`qwen35moe.cpp:575`).
+
+---
+
+## Reuse as-is (model-agnostic machinery)
+
+From `src/llama-context.cpp`, all shared once the iSWA/assistant specifics are
+generalized out:
+
+- `process_ubatch_mtp` (`:1538-1588`) — dedicated `sched_mtp` + reusable
+  `gf_res_prev_mtp`, reuse gate + `set_inputs`.
+- `graph_compute_mtp` (`:1591`).
+- `ensure_sched_mtp` N×-nodes reserve pattern (`:1419-1445`:
+  `max_nodes = graph_max_nodes * n_steps`).
+- The readback contract: `res->get_argmax()` → I32[n_steps] +
+  `res->get_embd()` (`:3000-3013`); accessors `src/llama-graph.h:679-719`
+  (`t_argmax` is set by **direct field assignment**, no `set_argmax` method —
+  reset at `src/llama-graph.cpp:898`).
+
+The **only** things inside these that must be generalized: the
+`dynamic_cast<llama_kv_cache_iswa>` guards (`:1451`, `:2916`) and the
+`mtp_assistant`-derived arch/width (`:2681-2685`, `:2930`).
+
+---
+
+## The 7 deltas (what must be built)
+
+1. **N-step chaining loop in the NextN builder.** `qwen35.cpp:graph_mtp`
+   (`:510-656`) is strictly single-block/single-pass. Add a `for k in n_steps`
+   unroll modeled on `gemma4-assistant.cpp:341-357`, re-invoking
+   eh_proj→attn→ffn→head N times in one graph. Factor the current body into a
+   `graph_mtp_build_one_step` primitive (mirror
+   `gemma4_mtp_build_one_step`, `gemma4-assistant.cpp:45-258`).
+
+2. **`n_mtp_steps` plumbing to the Qwen path.** `n_mtp_steps`
+   (`llama-graph.h:595-599`) is today only read by the Gemma builder and only set
+   via `mtp_fused_steps`→`graph_params_mtp` (`llama-context.cpp:2696`). The Qwen
+   draft ctx builds through the *normal* `process_ubatch`/`build_graph`, so there
+   is **no wire** carrying a step count into `graph_mtp`. Add a Qwen equivalent of
+   `graph_params_mtp` (arch = draft model's own arch, `gtype =
+   LLM_GRAPH_TYPE_DECODER_MTP`, `n_mtp_steps = N`) used by the new driver.
+
+3. **In-graph argmax on the NextN head.** `graph_mtp` sets only `t_logits`
+   (`:654`) + `t_h_pre_norm` (`:637`), never `t_argmax`. Add per-step
+   `ggml_argmax` on the LM-head output + concat into I32[N] (mirror
+   `gemma4-assistant.cpp:252, 362-368`), publish `res->t_argmax` (direct assign).
+   **Consequence:** the fused path is **greedy-only** and drops the host `p_min`
+   early-stop confidence gate (`speculative.cpp:767`) — see Risks.
+
+4. **On-device token→embed + hidden chaining.** `graph_mtp` already does
+   `ggml_get_rows(embed_tokens, tokens)` (`:540-543`) but on a *host-supplied*
+   `tokens`. Fused: step k+1's token = step k's on-device argmax feeding
+   `ggml_get_rows` (mirror `:355` + `:72`); step k+1's `h_input` = step k's
+   `t_h_pre_norm` (mirror `:356`) — entirely on-device.
+
+5. **NextN per-step position input — RESOLVED: reuse `llm_graph_input_mtp`.**
+   Its setter (`llama-graph.cpp:116-131`) is **width-agnostic** — `inp_h_prev` is
+   written `inp_h_prev->ne[0]` floats — so Qwen reuses it verbatim by creating
+   `inp_h_prev` at `n_embd` width (not Gemma's `n_embd_out_impl`). The only change
+   needed: extend `llm_graph_input_mtp::can_reuse` (`:134-145`) to also accept
+   `gtype==LLM_GRAPH_TYPE_DECODER_MTP` (today it hard-returns false unless
+   `LLM_GRAPH_TYPE_MTP` at `:135`), else the fused Qwen graph rebuilds every draft
+   and the perf win is lost (correct but slow).
+
+6. **KV for the N fused steps — RESOLVED (Option A, 2026-07-02): read-only
+   cross-attend the frozen prefix; NO new cache surgery.** Reading `build_attn_mtp`
+   (`llama-graph.cpp:2977`, esp. `:3011` `get_k`/`:3012` `get_v`) showed the
+   proven Gemma fused path is a **read-only cross-attention into the frozen prefix
+   KV** — it never writes draft KV; the recurrence is carried entirely by the
+   hidden-state chain (`h_post`→next `h`). That is why one slot suffices:
+   `llama_kv_cache::mtp_slot_info` (`llama-kv-cache.cpp:1985`) returns a **single**
+   idx (the `pmax` cell), and it already exists on the **unified** cache (iSWA's
+   `init_mtp` just calls it on `kv_base`/`kv_swa`). So the unified path needs **no**
+   new `init_mtp`/N-cell/causal-mask work — build one `llama_kv_cache_context` from
+   `mtp_slot_info(seq_id)` (mirror `llama-kv-cache-iswa.cpp:238`).
+
+   **Fork decided: Option A "Gemma-mirror" (prototype-and-measure).** Qwen's
+   per-step loop uses *self-attention that writes each draft token's KV* (so step
+   k sees drafts 0..k-1); Option A cross-attends only the frozen prefix (step k
+   sees the prefix, not 0..k-1) → drafts differ (`real_frac≠0`) and acceptance may
+   dip, but correctness is guaranteed by verify. Chosen because it's the proven
+   pattern with no KV-cache surgery. Measure acceptance in S5; only build the
+   self-attn N-cell exact-match variant (Option B) if A regresses materially vs
+   the 0.80 baseline. **Impl:** the fused steps need a Qwen read-only cross-attn
+   into the unified cache (an analog of `build_attn_mtp` — get_k/get_v from the
+   unified `llama_kv_cache_context` + the `build_attn_inp_kv` mask exposing the
+   prefix + `build_attn_mha`, no KV write), branched inside `build_one_step` on a
+   `fused` flag so the single-step path stays byte-identical.
+
+7. **New driver = `decode_mtp_fused` twin without the Gemma guards.** Drop:
+   `model.mtp_assistant` checks (`:2908, 1428`) → replace with "draft ctx is
+   `LLAMA_CONTEXT_TYPE_MTP`"; the iSWA cast (`:2916-2920, 1451-1457`) → unified
+   cache + its `init_mtp`; seed width `n_bb = n_embd_out_impl` (`:2930`) → plain
+   `n_embd` (Qwen's `t_h_pre_norm` is `[n_embd, n_outputs]`, read via
+   `get_embeddings_pre_norm_ith`, `:1061`); `graph_params_mtp`'s
+   `arch/gtype` (`:2681-2685`) → draft model's own arch + `DECODER_MTP`.
+
+Then **swap the driver into the loop**: `common_speculative_impl_draft_mtp::draft`
+(`speculative.cpp:702-803`) calls the new fused driver once instead of the per-step
+`llama_decode`+readback loop, and reads back I32[N] drafts + last hidden.
+
+---
+
+## Progress (2026-07-02)
+
+- **S0 — DONE.** vendor-backup `20260702-143936-pre-fused-nextn-590` (9338
+  entries, 37 patches, verified).
+- **S1 — DONE + gated byte-identical.** `build_one_step` lambda extracted in
+  `qwen35.cpp` + `qwen35moe.cpp` (`this`-capturing generic lambda, minimal-diff,
+  no re-indent). Incremental host build OK (2 TUs, host-only — no CUDA DSO
+  touched). bs2 35B NextN n2 chash `f80af37c9a` == pre-refactor baseline; decode
+  243–245, accept 0.79–0.80. **Build cadence confirmed cheap:** edit → incremental
+  `bun run build:llamafile:make` (2 TUs + relink) → rsync binary to bs2 (reuses
+  cached sm120 DSO, correct since host-only) → gate ~5 min.
+- **S2 — DONE + gated byte-identical.** `build_one_step` now takes the token index
+  (embeds internally via `ggml_get_rows`) and returns `{h_pre_norm, logits, arg}`
+  where `arg = ggml_argmax(logits)`. Added the **dormant** fused N-step branch
+  (gated `n_mtp_steps>1`, inert at default 1) in both files: reuses
+  `llm_graph_input_mtp` (N per-step I32[1] positions), chains `tok_k=arg_k` /
+  `h_k=h_pre_norm_k` on-device, concats argmaxes → `res->t_argmax`. Build OK; bs2
+  35B NextN n2 chash `f80af37c9a` (single-step path byte-identical, argmax node
+  pruned when unreferenced). **All graph-builder work is now in place, dormant.**
+
+## Staged plan (de-risk ordering)
+
+- **S0 — vendor-backup WHOLE tree** (`scripts/vendor-backup.sh backup
+  pre-fused-nextn`). Mandatory before source edits + build cycles. ✅
+- **S1 — refactor, no behaviour change.** Extract `build_one_step` from
+  `qwen35.cpp:graph_mtp` (and the MoE twin `qwen35moe.cpp:575`); the single-step
+  path calls it once. Build + prove **byte-identical** decode vs current
+  (chash) — pure refactor gate. ✅ (chash `f80af37c9a`).
+- **S2 — fused builder (greedy N-step) + input + argmax.** Deltas 1,3,4,5. Publish
+  `t_argmax`/`t_embd`. Still driven single-step (N=1) first to prove the argmax
+  path matches host `common_sampler_sample` (real_frac=0 vs the per-step loop).
+- **S3 — KV slots + driver.** Deltas 2,6,7. New `decode_mtp_fused_nextn` +
+  unified-cache `init_mtp`. Wire into `speculative.cpp` behind an env gate
+  (`OPENCOTI_MTP_FUSED_NEXTN`, default **off** until proven, then flip on like the
+  Gemma default per the standing "if it wins, default it" rule).
+- **S4 — build + restamp.** Host `rm -rf o` + make on solidPC (consistent tree),
+  CUDA DSO if any device code touched (argmax/get_rows are existing ops → likely
+  **no** new kernel, host-only), restamp both DSO paths, `nm -D` verify.
+- **S5 — gates.** (a) **Correctness:** logit-equiv / `real_frac=0` between fused
+  and per-step drafts on the same prompt (drafts must be identical token IDs since
+  both are greedy-argmax over the same NextN head); niah retrieval unchanged;
+  byte-identical final decode with fused **off**. (b) **Perf:** ours-fused vs
+  upstream b9859 on 35B NextN n=2 **and** n=3 — target: close the +16% at n2 and
+  kill the n3 collapse. Deploy binary to bs2 (GPU0), reuse `m592-accept.sh`
+  harness shape.
+- **S6 — ship.** Additive patch(es) into `vendors/patches/llamafile/` (backup +
+  snapshot-diff, byte-identical re-apply), `opencoti-hook:` markers +
+  UPSTREAM_SYNC registry, `docs/evaluations/mtp.md` + this doc's status, README
+  row, `.wolf` (anatomy/cerebrum/memory), pgvector. Commit on dev only when asked.
+
+---
+
+## Risks / open questions
+
+- **Loses `p_min` adaptive early-stop — negligible at the n=2 cap, and
+  recoverable.** The fused path is greedy-argmax with a fixed N (like Gemma), so
+  the host confidence gate (`speculative.cpp:767`, `cur_p->data[0].p < p_min` →
+  stop drafting; llama.cpp default ~0.75) is gone and it always drafts exactly N.
+  - **Correctness impact: none.** The **target verify pass rejects bad drafts**
+    (`llama-context.cpp:2896-2899`); output is exactly the target's regardless of
+    draft depth or greedy-vs-sampled. Fused greedy draft was measured
+    **byte-identical** to the sequential path in #454.
+  - **Throughput impact: near-neutral at n=2.** `p_min` exists to avoid spending a
+    *sequential* per-step `llama_decode`+readback on a token that will likely be
+    rejected. The fused path computes all N steps in **one** launch, so that
+    per-step cost the gate protected against no longer exists. Always-drafting-2
+    costs only the fused graph's marginal step-2 compute + one extra position in
+    the verify batch (3 vs 2 — negligible), and can *gain* acceptance when a
+    sub-threshold second token is actually correct. At large n_max the waste
+    grows, but that's where fused's cheap launch wins most, and we cap n anyway.
+  - **Recovery if measurement ever shows it matters (no per-step sync):** emit a
+    second small in-graph tensor — the per-step softmax-max (confidence) alongside
+    the argmax — read it back in the *same* single D2H, and trim the proposed
+    draft block host-side by `p_min` before submitting to verify. That restores
+    adaptive depth for the price of one extra tiny readback, not N syncs. Keep as a
+    follow-up lever; do not build unless S5's acceptance-delta check regresses.
+- **Unified-cache `init_mtp` is new surface.** iSWA has `init_mtp`; the unified
+  cache does not. This is the highest-uncertainty delta (S3) — the N draft
+  positions must be placeable and each step's KV visible to the next step's
+  attention. De-risk by reading `llama_kv_cache_iswa::init_mtp` and mirroring on
+  the unified type.
+- **Likely host-only (no CUDA rebuild).** The chain uses existing ops
+  (`ggml_argmax`, `ggml_get_rows`, attention). If true, S4 skips the CUDA DSO
+  rebuild — confirm no new device kernel is introduced before assuming so.
+- **Effort:** multi-session. S1 (refactor) + S2 (builder) are the bulk; S3
+  (KV/driver) is the risk. Payoff is ~16% on the 35B-class NextN spec path only —
+  base and the KV/DCA/turbo moat are unaffected either way.
+
+---
+
+## Measured — Qwen3.6-35B-A3B NextN, ours vs b9859 (bs2, 2026-07-14, #654 short4)
+
+Coherent prompt, `n_ctx=4096`, greedy, `cache_prompt=false`, `n_predict=300`, `--spec-type
+draft-mtp --spec-draft-n-max {1,2,3}`. ours = `llamafile.dualctx 7abd684039a3`; b9859 =
+upstream `llama-server`.
+
+| config | ours decode t/s | ours accept | b9859 decode t/s | b9859 accept |
+|--------|-----------------|-------------|------------------|--------------|
+| base (no spec) | 208.3 | — | 212.0 | — |
+| n1 | 226.4 | 0.892 | 258.3 | 0.869 |
+| n2 | 230.4 | 0.739 | 269.9 | 0.770 |
+| n3 | 240.1 | 0.638 | 277.3 | 0.735 |
+
+**Read:** base is at **parity** (208 vs 212). ours' NextN decode still trails b9859 by **~12–15 %**
+(n1 −12 %, n2 −15 %, n3 −13 %) — the unchanged **bug-858 Qwen verify/CUDA-graph gap** (accept is at
+or above upstream at n1; the tps deficit is spec-orchestration/verify host overhead, not draft
+quality). This supersedes the pre-#614 "~16 % slower (244 vs 283)" note with current-binary numbers.
+The earlier #654 pass BOOT-FAILED on ours (harness omitted `--server`; llamafile came up in CLI chat
+mode — buglog bug-2179); re-run with the conditional `--server` fix produced the table above.
+
+---
+
+## Anchors (quick index)
+
+- Un-fused loop: `common/speculative.cpp:702-803` (readback `:752`).
+- Qwen NextN builder: `src/models/qwen35.cpp:510-656` (dispatch `:133-138`);
+  MoE `src/models/qwen35moe.cpp:156-158, 575`.
+- Gemma fused model: `src/models/gemma4-assistant.cpp:45-258` (one-step),
+  `:260-374` (N-step chain), argmax concat `:362-368`.
+- Gemma fused driver: `src/llama-context.cpp:2900-3017`;
+  runner `:1538-1588`; params `:2673-2698`; reserve `:1419-1470`.
+- Graph types: `src/llama-graph.h:37-39`; `n_mtp_steps` `:595-599`;
+  `llm_graph_input_mtp` `:128-145`; result accessors `:679-719`.

docs/features/gemma4_dca.md

@@ -0,0 +1,485 @@
+# Feature — Gemma-4 A4B Dual Chunk Attention (DCA), training-free 1M context
+
+> **Status (2026-06-10): DESIGN LANDED, patch `0078` not yet captured.** Feature home for the
+> forthcoming `vendors/patches/llamafile/0078-gemma4-dca.patch`. The patch capture + build + RULER
+> gate run in an **out-of-tree clean-room** (a separate llamafile checkout) — see "Capture without
+> reset" — because the live `vendors/sources/llamafile` working tree holds the only applied source of
+> patches `0001–0077` and MUST NOT be `reset`/`clean`ed (buglog 2026-06-04: `build:llamafile:reset`
+> = `git reset --hard` + `git clean -fdx` wipes uncommitted patch source). T87.pD.
+
+## Goal
+Extend Gemma-4 26B-A4B (and 31B dense) to ~1M context **training-free**, matching Qwen2.5-14B-1M's
+RULER class (92.5 @ 1M), by porting **Dual Chunk Attention** (DCA, the Qwen 1M mechanism) into the
+llamafile graph. Runs on consumer GPUs (RTX 3090) — the path our users actually have — instead of
+the parked vLLM FA4-sm120 route (which needs an unfinished CuTe Blackwell kernel; see backup_models
+task #592). DCA composes with YaRN mscale for the final 2×–4× past the chunk grid.
+
+## Why llamafile is sufficient (feasibility verified 2026-06-10, read-only)
+Gemma-4 26B-A4B attention is **mixed-head** (config + tensor-shape confirmed):
+- **25 sliding layers @ head_dim=256** (window 1024, θ=1e4) — bounded by their window, need NO
+  long-range handling.
+- **5 global/full layers @ head_dim=512** (idx **5, 11, 17, 23, 29**; `global_head_dim=512`,
+  `num_global_key_value_heads=2`, `attention_k_eq_v=true` → no `v_proj`, K=V; `partial_rotary_factor
+  0.25` → n_rot=128 of 512; θ=1e6, `rope_type=proportional`). **These carry all long-range
+  information → DCA touches ONLY these 5 layers.**
+- No `attn_logit_softcapping` (Gemma 3/4 dropped it; only `final_logit_softcapping=30` on the LM
+  head) → the chunk merge is the plain no-softcap online-softmax.
+
+Every primitive DCA needs already exists in `vendors/sources/llamafile/llama.cpp` (RULER-validated):
+- **Proportional rope** — `src/models/gemma4-iswa.cpp:56-59` feeds `model.layers[il].rope_freqs` as
+  `freq_factors` to `ggml_rope_ext` for `!is_swa` layers; `rope_freqs` loads at
+  `src/llama-model.cpp:2955`. Per-layer `freq_base_l/freq_scale_l/n_rot_l` (`gemma4-iswa.cpp:47-49`).
+- **YaRN** — `ggml_rope_ext(..., ext_factor, attn_factor, beta_fast, beta_slow)` (`gemma4-iswa.cpp:73,97`)
+  + hparams `n_ctx_orig_yarn`/`yarn_*` + `rope_scaling_type ∈ {none,linear,yarn,longrope}`.
+- **hd=512 flash-attention** — patch `0075-d512-turbo-vec` (D-generic FA-VEC for the global layers).
+- **LSE-emitting FA + exact online-softmax merge** — patch `0070-rolling-kv`: `launch_fattn` per-row
+  `dst_lse = max + logf(rowsum)` via the `opencoti_fattn_dst_lse` consume-once channel +
+  `GGML_OP_STREAMING_FLASH_ATTN` + `streaming_combine_kernel`. **This is DCA's chunk-partial merge**;
+  generalizes from 2 regions (window⊕tail) to N chunk-groups.
+- **Training-free position-remap precedent** — `grp_attn_n/grp_attn_w` (LongLM Self-Extend),
+  `tools/completion/completion.cpp:511-517`. Same CLASS as DCA. **Caveat: completion-tool-only**, so
+  DCA hooks at the **graph** `build_attn` seam (serves the server opencoti drives).
+- Clean per-layer seam: `build_attn(inp_attn, wo, …, Qcur, Kcur, Vcur, …, f_attention_scale, il)`
+  (`gemma4-iswa.cpp:102`).
+
+**No blocking limitation.** Nuances handled by design: extension config (YaRN factor / `n_ctx_orig`)
+via GGUF metadata or CLI; Self-Extend is completion-only so we hook the graph; our A4B GGUFs already
+carry `rope_freqs` (they serve at 256k today).
+
+## DCA algorithm (global layers only)
+Split positions into chunks of size `c` (`c` < trained length; e.g. 65536). For each global layer,
+attention is computed in position regimes merged by LSE:
+- **intra-chunk** — queries attend keys in the SAME chunk at LOCAL positions `[0,c)` (preserves the
+  trained short-range rope behavior).
+- **inter-chunk** — queries attend EARLIER chunks at a clamped inter-chunk (chunk-index) position so
+  relative positions stay inside the trained range.
+- **successive-chunk** — the immediately-preceding chunk uses a continuous bridging position so the
+  intra/inter seam doesn't jump.
+On Gemma-4 only the **rotary 128 dims** (partial_rotary 0.25) are remapped; the other 384 pass
+through. Each regime = a separate rope'd-Q × KV attention emitting LSE; the partials merge via the
+existing online-softmax combine. YaRN mscale (attn_factor) optionally rides on top for the final
+2×→4×.
+
+## Patch `0078-gemma4-dca` structure
+Base for capture: live tree = `0001-0042 + 0070..0077` applied. Number `0078` (applies AFTER `0077`).
+
+### A. Flags (additive, default-OFF → byte-identical when off)
+`common/arg.cpp` + `common/common.h` + `src/llama-cparams.h`: `--dca {off,on}` (default off),
+`--dca-chunk-size N` (default = n_ctx_train ⇒ inert: `c ≥ seqlen` collapses to a single intra
+regime = plain attention), optional `--dca-yarn-factor F`. `LLAMA_ARG_DCA*`, `LLAMA_EXAMPLE_SERVER`.
+New `cparams`: `bool dca_enabled; uint32_t dca_chunk; float dca_yarn_factor;`. OFF path grep-clean
+byte-identical to pre-0078.
+
+### B. Graph hook (one surgical marker) — `src/models/gemma4-iswa.cpp`
+For `!hparams.is_swa(il) && cparams.dca_enabled` ONLY, route attention through the new additive
+builder instead of the single `build_attn(...)`. Marker `// opencoti F5 gemma4-dca`. Sliding layers
+and `dca_enabled==false` keep the unchanged `build_attn` → OFF byte-identical.
+
+### C. Additive builder — NEW `src/models/gemma4-dca.cpp` (+ decl in `src/models/models.h`)
+`build_attn_dca_gemma4(...)`: builds intra/successive/inter rope'd-Q variants (3× `ggml_rope_ext`
+on `inp_pos`-derived remapped position tensors), runs 3 `build_attn`-style FA calls over the global
+KV emitting `dst_lse` (reuse `0070`'s channel) with chunk masks, merges the 3 partials with a
+`streaming_combine`-family reduce (extend the 2-region form to N-region). Keeps the in-tree hook to a
+single marker (NEW file per UPSTREAM_SYNC §Additive).
+
+### D. Position inputs — additive `llm_graph_input_dca`
+Mirrors `llm_graph_input_mtp` (0074) / position-window inputs (0070): holds intra/inter/succ position
+vectors derived from `inp_pos` + `dca_chunk`. Registered in graph-input assembly; no upstream-source
+edit beyond registration.
+
+### E. Acceptance gate — NEW `.opencoti/dca-ruler-gate.sh`
+- **OFF byte-identity**: `--dca off` greedy decode byte-identical to the pre-0078 binary.
+- **Below-one-chunk no-op**: at ctx ≤ 256k with `--dca-chunk-size ≥ ctx`, `--dca on` RULER vt/niah
+  == `--dca off` (DCA must be a no-op when `c ≥ seqlen`). Uses the RULER native runner.
+- **Extension win**: RULER vt/niah at 512k & 1M with `--dca on` vs YaRN-only degradation — target
+  the T87 1M class (≥ Qwen2.5-14B-1M's 92.5@1M behavior on the global layers). Uses the
+  merge→serve→RULER-ladder driver.
+- Acceptance = logit-distribution / RULER-score equivalence, NOT greedy byte-equality past one chunk
+  (online-softmax is fp-non-associative — same standard as `0070`).
+
+### F. README row + UPSTREAM_SYNC
+Add the `0078-gemma4-dca` row to `vendors/patches/llamafile/README.md` (## Current series) and
+register the single `// opencoti F5 gemma4-dca` marker in `docs/protocols/UPSTREAM_SYNC.md` (same
+mechanism as `0076`/`0077`). Provenance: opencoti-original; algorithm from the DCA paper
+(Qwen2.5-1M) + the deleted v0 vLLM `dual_chunk_flash_attn.py` reference (read, not transcribed).
+
+## Capture without reset (the live tree holds the only patch source)
+The standard `bun run build:llamafile:{check,apply,build}` flow assumes a pristine submodule
+(`git apply --check`/end-to-end apply), which the live tree is NOT (0001-0077 already applied) and we
+must NOT `reset` it. Therefore:
+1. **Clean-room build/gate**: in a SEPARATE out-of-tree llamafile checkout (not the held submodule),
+   `reset → bootstrap → apply 0001-0077 → apply 0078 edits → build (cosmocc + ggml-cuda DSO) → run
+   gate E`. The held `vendors/sources/llamafile` is never touched.
+2. **Capture the diff** against an out-of-tree base **copy** of the touched files (NOT `git apply`
+   from inside an in-tree baseline — that is a silent no-op, buglog bug-431; use `diff -u base live`
+   to produce the patch and `patch -p1` to re-apply into a copy when verifying).
+3. **Verify round-trip**: `0078-gemma4-dca.patch` reverse-applies clean vs the post-0078 tree and
+   forward-reproduces it byte-for-byte (0 diffs), both directions.
+4. **Land** only the artifacts in the opencoti repo: `0078-gemma4-dca.patch`, the README row, the
+   UPSTREAM_SYNC entry, `.opencoti/dca-ruler-gate.sh`, this doc. The held submodule working tree
+   stays exactly as the other session left it.
+
+## Cheaper baseline to A/B first
+`grp_attn` (Self-Extend) already exists; A/B its RULER@512k for A4B as the bar DCA must beat. Blocker:
+completion-tool-only → needs a tiny server-path `grp_attn` hook OR running the completion tool for
+the probe. Document the gap vs full DCA before committing to the 0078 build.
+
+## Section-C merge: implementation decision (grounded 2026-06-10, read-only)
+
+The graph seam (`gemma4-iswa.cpp` global block, `!is_swa(il)`): `rope_ext(Qcur/Kcur, inp_pos, freq_factors=rope_freqs, n_rot_l, freq_base_l=1e6, ...)` → `build_attn(inp_attn, wo, …, Qcur, Kcur, Vcur, …, f_attention_scale, il)`. Hook here for `dca_enabled && !is_swa`.
+
+0070 primitives available host-side: `ggml_streaming_flash_attn(q,k,v,mask,scale,max_bias,softcap)` (emits O+LSE), `ggml_streaming_flash_attn_window(q,k_win,v_win,k_tail,v_tail,mask,…)` (2-region FUSED FA+combine), `ggml_flash_attn_ext_tail_partial(q,k,v,…)` → un-normalized partial `[2+DV, n_head, n_q, n_b]` = (m, s, VKQ). **There is NO standalone host-exposed N-way (O,lse) combine** — the online-softmax merge is fused inside `GGML_OP_STREAMING_FLASH_ATTN` (`fattn.cu streaming_combine_kernel`).
+
+DCA needs to merge **3** regime partials (intra/successive/inter). Two paths:
+- **C1 — graph-math merge (host-only, PREFERRED, no DSO rebuild):** per regime, emit the un-normalized partial (m_i, s_i, O_i·s_i); combine N partials with a small subgraph of EXISTING ggml ops: `m=max_i(m_i)`, `w_i=exp(m_i−m)`, `S=Σ w_i s_i`, `O=Σ w_i O_i s_i / S`. All elementwise/reduce ops already in ggml → compiles into the host binary, executes on the existing fat `ggml-cuda.so 0.10.1` (sm_86+sm_120). Build = cosmocc `make` only. **Risk:** must get each regime's FA to emit the GPU partial to a graph tensor (verify `ggml_streaming_flash_attn` partial-output mode is graph-addressable, not just the thread-local channel; the `_tail_partial` op is CPU-only — too slow at 512k, so the GPU streaming op's partial path is required).
+- **C2 — new N-way combine kernel (DSO rebuild):** add `ggml_streaming_combine_n(partials[])` + a `streaming_combine_n_kernel` generalizing the 2-region kernel → rebuild `ggml-cuda.so` via `cuda.sh`. Cleaner/faster but heavier build + a CUDA-source edit (per UPSTREAM_SYNC, a NEW kernel file to keep the in-tree hook minimal).
+
+**Decision: implement C1 first** (host-only build, fastest to a gated result); fall back to C2 only if the GPU partial isn't graph-addressable or numerics demand a fused kernel. Either way the 3 regime Qs come from 3 `ggml_rope_ext` on remapped `inp_pos` views (intra=local [0,c); inter=chunk-index clamp; succ=continuous bridge), remapping only the rotary 128 dims (partial_rotary 0.25); the other 384 pass through.
+
+**Build path is cosmocc, NOT cmake** (confirmed: patch sources `ggml-neo-pipeline/iqk/turbo/d512-vec` are NOT in any `CMakeLists.txt`; native `cmake --build` link-fails on those symbols). Clean-room = out-of-tree copy of the held `vendors/sources/llamafile` tree (already has 0001-0077 applied to source); edit C1 there; `bun run build:llamafile:make` (host binary, reuses fat DSO); gate E with `--dca on` + `--override-kv gemma4.context_length` serve; capture 0078 by `diff` vs a frozen copy of the held tree. Held submodule never reset.
+
+### Section-C VERDICT (2026-06-10, op contract verified read-only) — C1 FALSIFIED → C2 required
+
+Read the actual 0070 op definitions (`vendors/patches/llamafile/0070-rolling-kv.patch`). The "implement
+C1 first" decision above is **falsified by the op contract**:
+
+1. The GPU op `ggml_streaming_flash_attn` returns **normalized O only** — `ne[4]={DV, n_head, n_q,
+   n_batch}`, F32 (patch L539). Its per-tile LSE lives in **internal pool allocs** `O_tiles`/`lse_tiles`
+   (L1327-8) consumed by a **`static __global__ streaming_combine_kernel`** *inside* fattn.cu (L1113,
+   invoked L1859). **LSE is never a graph tensor.** → cannot call it 3× and merge (O,LSE) host-side.
+2. The only op that emits the addressable un-normalized partial `[2+DV,n_head,n_q,n_b]`=(M,S,VKQ),
+   `ggml_flash_attn_ext_tail_partial`, is **CPU-only** (dispatched solely in ggml-cpu/ops.cpp L694/811;
+   **no CUDA case** — CUDA dispatch L1895/1912 is `STREAMING_FLASH_ATTN` only) **and decode-only `n_q==1`**
+   (its own header, L618-620). Useless for the 512k-token **prefill** where DCA matters; CPU scalar at
+   512k is intractable regardless.
+3. Eager/materialized DCA (3 masked QK^T blocks combined by pure graph math, single softmax — which
+   *would* be genuine C1, no DSO rebuild) needs the full O(n²) score matrix: ≈1 TB/head @512k. Infeasible.
+
+**Therefore: the performant DCA path requires a CUDA-source edit → DSO rebuild (C2).** There is no
+host-only route to per-regime (O,LSE) at prefill scale. Build flips from cheap `build:llamafile:make`
+(reuse fat DSO) to **`cuda.sh` ggml-cuda.so rebuild** (the multi-hour cosmocc CUDA path), done in the
+out-of-tree clean-room (held submodule still never touched).
+
+**C2, minimal-footprint shape (reuses machinery that already exists internally):** expose the existing
+per-tile partial + combine as graph-addressable —
+- (a) a GPU op variant that runs FA over one regime and writes the **un-normalized partial** (M,S,VKQ)
+  to a graph tensor (the streaming op already computes exactly this per tile before its internal
+  combine — emit it instead of/in addition to normalized O), and
+- (b) a host-exposed N-way `streaming_combine_n(partials[3])` generalizing `streaming_combine_kernel`
+  (which already combines N tile-partials) so the 3 regime partials merge on-device.
+Per UPSTREAM_SYNC, new CUDA code lands in a NEW file; the in-tree hook stays a single marker. The 3
+regime Qs still come from 3 `ggml_rope_ext` on remapped `inp_pos` views (intra=local [0,c);
+inter=chunk-index clamp; succ=continuous bridge), rotary-128 dims only (partial_rotary 0.25); other
+384 pass through. Numerics validated against the existing op's internal combine (same LSE math).
+
+## MULTI-ARCH generalization (2026-06-10, user directive: "don't limit DCA to Gemma4 — same 1M extension on Qwen")
+
+DCA is now an **architecture-general** training-free 1M extension, not a Gemma4 one-off. Patch renamed
+`0078-dca` (was `0078-gemma4-dca`). Verified the Qwen seam is identical to Gemma4's:
+`qwen2.cpp`: `ggml_rope_ext(Q/K, inp_pos, freq_factors=nullptr, n_rot=n_embd_head /*full rotary*/,
+rope_type, n_ctx_orig, freq_base, freq_scale, …)` → `build_attn(inp_attn, wo, …, Q,K,V, …, scale, il)`
+— same shape, differing only in `freq_factors` (Gemma4 global = `rope_freqs`; Qwen = null) and
+`n_rot` (Gemma4 partial-128 of 512; Qwen full head_dim). So **one generic helper serves all**:
+
+- **Section C (core, arch-agnostic)** — new `dca.{cpp,h}`: `llm_graph_input_dca` (the 3-regime remapped
+  position vectors, built once like `inp_pos`) + `llm_graph_context::build_attn_dca(inp_attn, inp_dca,
+  wo, wo_b, Qcur_PREROPE, Kcur_PREROPE, Vcur, …, dca_params, rope_params, kq_scale, il)`. It applies the
+  3 regime ropes (intra/successive/inter) via `ggml_rope_ext` on remapped positions, runs the
+  partial-emit FA per regime, merges with `streaming_combine_n`, then the `wo` projection (mirrors
+  `build_attn`'s tail). `rope_params` carries exactly the args each builder already passes to
+  `ggml_rope_ext` (n_rot, rope_type, n_ctx_orig, freq_base/scale, ext/attn_factor, beta_fast/slow,
+  freq_factors) → handles Gemma4 partial-rotary AND Qwen full-rotary with no special-casing.
+- **Section B (per-arch hooks, small)** — in each builder, gate the full-attention layers through
+  `build_attn_dca` instead of `rope_ext+build_attn`: Gemma4 `is_dca_layer = !is_swa(il)` (5 global only);
+  Qwen `is_dca_layer = true` (all layers, no SWA). First-landing hooks: `gemma4-iswa.cpp`, `qwen2.cpp`
+  (Qwen2.5-1M, the canonical DCA target), `qwen3.cpp`, `qwen3moe.cpp`; `qwen35/qwen35moe/qwen2moe` are
+  one-marker each, added as needed. Marker `// opencoti F5 dca`.
+- **Section D (CUDA, arch-agnostic)** — partial-emitter op + `streaming_combine_n`; operates on tensors,
+  no arch knowledge. This is the C2 DSO-rebuild payload.
+- **Gate E** — RULER-VT @512k on BOTH A4B Q4_K_M AND a Qwen 1M model, vs the naive-ext bar (VT 0.36).
+
+### Clean-room standup (confirmed mechanics)
+Build pipeline: `bun --cwd packages/opencoti-llamafile script/build-pipeline.ts {reset,apply,make,cuda}`.
+`apply` = `reset`(`git clean -fdx`+checkout pin `6490e16`) → `bootstrap`(download cosmocc + Mozilla-Ocho
+`make setup`) → `git apply` 0001-0077. The held submodule has 8 uncommitted WIP files (`cuda.sh`,
+`llamafile.c/h`, build mk, nested `llama.cpp`) — `reset` would WIPE them (buglog bug-431). So the
+clean-room is a **separate `git clone /shared/dev/opencoti <dca-clone>`** (committed state only; held WIP
+excluded by design), where `apply`→edit 0078→`cuda` runs freely. Capture 0078 by diffing the clone's
+post-0078 source vs its post-0077 (pre-edit) source. Held tree never reset, never touched.
+
+## Section-C ENGINEERING DESIGN (2026-06-10, grounded in clone applied source) — the cache-rope problem
+
+Reading `llm_graph_context::build_attn` (kv variant) exposed the crux: **K is stored to the KV cache
+already rope'd** (model builder ropes Kcur → `build_attn` → `mctx_cur->cpy_k(Kcur)`), and attention
+reads it back rope'd. DCA needs the SAME keys re-rope'd with DIFFERENT positions per regime
+(intra=native pos, inter=chunk-clamped, succ=bridge) — a single cached rope can't provide that.
+
+**Decision: DCA layers cache K UN-ROPE'd and apply all rope at attention time, per regime, inside
+`build_attn_dca`.** Consequences:
+- The per-arch hook (Section B) passes **pre-rope Q AND pre-rope K** to `build_attn_dca` (not just
+  re-routes the call). On Gemma4 global, pre-rope K = `attn_k_norm(K_proj)` (the tensor right before
+  `ggml_rope_ext`); V = `rms_norm(K_proj)` is already un-rope'd (k_eq_v) and unchanged. On Qwen,
+  pre-rope K = the reshaped projection before its `ggml_rope_ext`.
+- `build_attn_dca` stores pre-rope K + V to cache (`cpy_k`/`cpy_v`), then per regime r∈{intra,succ,
+  inter}: `Q_r=rope(Qcur_pre, pos_q_r)`, `K_r=rope(get_k_raw, pos_k_r)`, `partial_r =
+  streaming_flash_attn_PARTIAL(Q_r, K_r, V, mask_r, kq_scale)`; then `O =
+  streaming_combine_n([partial_intra, partial_succ, partial_inter])`; then `wo` projection (mirrors
+  build_attn tail: `build_lora_mm(wo,·)` + optional `wo_b`).
+- Rope-on-cached-raw-K is an ESTABLISHED pattern here: `llama_kv_cache::build_rope_shift`
+  (llama-kv-cache.cpp:3893, used by context-shift) re-ropes cached K via a position/`k_rot` tensor —
+  the structural template (DCA ropes a READ COPY with absolute regime positions, does NOT mutate the
+  cache).
+- **`llm_graph_input_dca`** (built once like `inp_pos`, host-filled from batch positions + KV-cache
+  cell positions) supplies, per regime: `pos_q_r` (len n_tokens), `pos_k_r` (len n_kv), and `mask_r`
+  ([n_kv × n_tokens] additive). The pos remaps + masks encode the DCA chunk math
+  (chunk_size c = `--dca-chunk-size`, default ← n_ctx_orig). intra = same-chunk causal, native pos;
+  inter = q-chunk > k-chunk, k pos within-chunk + q pos clamped ≤ pretrain window; succ = adjacent
+  chunk bridge. rope uses `rope_params` (n_rot, rope_type, freq_base/scale, ext/attn_factor,
+  beta_fast/slow, freq_factors) — Gemma4 partial-128 / Qwen full-rotary handled by the n_rot arg.
+
+**Toggle is launch-time only** (cache contents differ: raw-K for DCA vs rope'd-K for non-DCA) — fine,
+it's a serve flag. Wiring caveat: Gemma4 0076 shared-KV means some global layers `reuse KV cache of
+earlier layers` (`has_kv(il)` false) — DCA must rope-on-read consistently for the layer-group that
+owns the cache; verify against the 0076 shared-kv mechanics when wiring Section B.
+
+## Implementation progress (2026-06-10) + Section-D simplification
+
+Patch number **0078 confirmed** (user, keep 0078). Build host = solidpc RTX 3090 (sm_86, CUDA 13.2).
+
+**Section-D shrinks to ONE new op + graph-math combine** (no new combine kernel). Insight: if the new
+FA op emits NORMALIZED O packed with per-row LSE as `[DV+1]` (reusing 0070's existing `dst_lse`
+channel — the `lse = m + logf(sum exp)` math already lives in `launch_fattn`), the 3-regime merge is
+pure existing-ggml graph math: `M = max_r(lse_r)` (binary max via `a + relu(b−a)`), `w_r =
+exp(lse_r−M)`, `O = Σ_r w_r·O_r / Σ_r w_r` (`ggml_exp/mul/add/div`, broadcasting w over DV). So the
+DSO rebuild only adds a CUDA *forward* for the new op (FA + write lse), not a bespoke combine kernel.
+
+**DONE this session (clean-room `/shared/dev/opencoti-dca`, baseline nested `43ccc8a56`):**
+- **Section A (flags)** — `--dca`/`--dca-chunk-size`/`--dca-yarn-factor` through all 6 opencoti sites
+  (arg.cpp, common.h, common.cpp, llama-context.cpp ×2, llama-cparams.h, llama.h). Verified.
+  Script: `backup_models/scripts/dca/apply_section_a_flags.py`.
+- **Section D-host** — `GGML_OP_FLASH_ATTN_EXT_LSE` + `ggml_flash_attn_ext_lse(q,k,v,mask,scale,
+  max_bias,logit_softcap)` → dst `[DV+1, n_head, n_q, n_batch]` ([0,DV)=O, [DV]=lse). Registered in
+  ggml.h enum+decl, ggml.c name/symbol arrays, both static_asserts (100→101), constructor.
+  Script: `backup_models/scripts/dca/apply_section_d_host.py`.
+
+**REMAINING:** D-cuda (CUDA forward for the new op: `ggml_cuda_flash_attn_ext` path + dst_lse→dst[DV]
+slot; supports_op + scheduler-pin like the bug-259 STREAMING_FLASH_ATTN hook) → Section C
+(dca.cpp/h: `llm_graph_input_dca` + `build_attn_dca`) → Section B (per-arch hooks) → cuda.sh DSO
+rebuild → RULER-VT gate (A4B + Qwen) → capture `0078-dca.patch`.
+
+## Section D-cuda DONE (2026-06-10)
+
+`ggml_cuda_flash_attn_ext_lse` authored + wired (8 sites, `apply_section_d_cuda.py`):
+- **fattn.cu** — the forward: O via an O-typed view of the packed dst → existing
+  `ggml_cuda_flash_attn_ext` (proven streaming-op `ggml_tensor o=*dst; o.op=FLASH_ATTN_EXT` trick);
+  per-row lse via the existing static `streaming_lse_kernel` into the trailing `n_rows` (softcap-aware,
+  works for any n_q incl. prefill — avoids the `decode_lse`/bug-263 finalize fragility). Asserts Q f32 / K f16.
+- **fattn.cuh** decl; **ggml-cuda.cu** compute_forward dispatch + supports_op (validates the O FA at
+  ne[0]=DV, not the packed DV+1); **ggml-backend.cpp** bug-259 scheduler CUDA-pin; **ggml-cpu** ×3
+  (compute_forward abort, n_tasks single-task, supports_op=false) — CUDA-only mirror of STREAMING_FLASH_ATTN.
+
+Op fully complete (host + CUDA). Compile-verified later in the single cuda.sh DSO build (after C+B).
+**REMAINING:** Section C (dca.cpp/h: `llm_graph_input_dca` + `build_attn_dca` with DCA position-remap +
+graph-math combine — O view ne=[DV,H,nq,nb] @offset0, lse view ne=[1,H,nq,nb] @offset DV*n_rows) →
+Section B (per-arch hooks) → cuda.sh DSO rebuild → RULER-VT gate → capture 0078-dca.patch.
+
+## Section C design (2026-06-10) — grounded against the clean-room seams
+
+Read of the clean-room (`43ccc8a56`) settles the algorithm and the integration points.
+
+**Seams (verified):**
+- kv-variant `build_attn` (llama-graph.cpp:2571) and **iswa**-variant `build_attn`
+  (llama-graph.cpp:2841). Both: `cpy_k/cpy_v(k_cur,…)` store **whatever K they're handed** →
+  `get_k/get_v(ctx0,il)` read-back → `build_attn_mha` → `build_lora_mm(wo)` + `wo_b`. Gemma-4
+  globals ride the **iswa** path (`build_attn_inp_kv_iswa`, `is_swa(il)==false` → base mctx);
+  Qwen rides the **plain-kv** path. ⇒ `build_attn_dca` needs **both** input-type overloads.
+- Host position source: `set_input_kq_mask` (llama-kv-cache.cpp:3773) / `set_input_pos_bucket`
+  (:3812) read `v_cells[strm].pos_get(j)` / `cells.is_empty(j)` per cached cell. A
+  `llm_graph_input_i::set_input` only gets the ubatch → the per-cell fill **must** be a new
+  `llama_kv_cache[_context]::set_input_dca` (mirrors `set_input_kq_mask`), delegated to from
+  `llm_graph_input_dca::set_input`.
+- Rope contract (gemma4-iswa.cpp:73/97): `ggml_rope_ext(x, inp_pos, freq_factors, n_rot_l,
+  rope_type, n_ctx_orig, freq_base_l, freq_scale_l, ext, attn, beta_fast, beta_slow)` — Gemma-4
+  globals use `n_rot_l=hparams.n_rot(il)` (partial-128 on the hd512 global), `freq_base_l=1e6`.
+  Qwen full-rotary uses `n_rot = n_embd_head`, `freq_base=1e6`/yarn. `build_attn_dca` reproduces
+  this call three (Q) + one (K) times with regime position vectors instead of `inp_pos`.
+
+**Algorithm — rope-K-ONCE, rope-Q-THRICE (ChunkLlama scheme).** All three regimes share one key
+assignment `pos_k[j] = cellpos(j) mod c`; only the query position differs:
+
+| regime | `pos_q[i]` | keep-mask (causal `j≤i`, chunk `=⌊pos/c⌋`) |
+|---|---|---|
+| **intra** | `i mod c` | `chunk(j) == chunk(i)` |
+| **succ**  | `(i mod c) + c` | `chunk(j) == chunk(i) − 1` |
+| **inter** | `c` (const) | `chunk(j) <  chunk(i) − 1` |
+
+The three keep-masks **partition the causal lower-triangle exactly** → the LSE-merge of the three
+softmaxes equals full attention under DCA-remapped positions (no double-count, no gap). No q-k
+distance exceeds the pretrain window: intra `<c`, succ `∈(0,2c)`, inter `=c−ok∈[1,c]`. `c` =
+`dca_chunk_size` or (auto) `n_ctx_orig_yarn`. YaRN mscale composes via the existing rope `ext/attn`
+factors (`dca_yarn_factor` feeds `cparams.yarn_*`; the rope kernel itself is unchanged).
+
+**`build_attn_dca(inp, wo, wo_b, Qpre, Kpre, Vcur, …, kq_scale, il)` graph body:**
+1. `cpy_k(Kpre)` / `cpy_v(Vcur)` to cache (Section B hands **pre-rope** K + un-roped V).
+2. `Kraw = get_k(ctx0,il)`; `V = get_v(ctx0,il)`; `Kdca = ggml_rope_ext(Kraw, inp_dca->pos_k, …)`
+   — **one** rope of the large cached K (raw cells, regime-invariant key positions).
+3. For `r ∈ {intra,succ,inter}`: `Qr = ggml_rope_ext(Qpre, inp_dca->pos_q[r], …)`;
+   `Pr = ggml_flash_attn_ext_lse(Qr, Kdca, V, inp_dca->mask[r], kq_scale, max_bias, softcap)`
+   → packed `[DV+1, H, nq, nb]`. O view `ggml_view_4d(Pr, DV,H,nq,nb,…,0)`; lse view
+   `ggml_view_4d(Pr, 1,H,nq,nb,…, DV*nb0)`.
+4. Combine (pure ggml): `M = a+relu(b−a)` over the 3 lse; `wr = ggml_exp(lse_r − M)` (broadcast over
+   DV); `O = (Σ wr·Or) / (Σ wr)`; reshape `[n_embd, n_tokens]`; `build_lora_mm(wo)` + `wo_b`.
+
+**`llm_graph_input_dca`** (new, dca.h): `pos_k` (I32 [n_kv]), `pos_q[3]` (I32 [n_tokens]), `mask[3]`
+(F32→F16 cnv, `[n_kv, GGML_PAD(n_tps,GGML_KQ_MASK_PAD), 1, n_stream]` like `build_attn_inp_kq_mask`),
+`const llama_kv_cache_context * mctx`, `uint32_t c`. `set_input(ubatch)` →
+`mctx->set_input_dca(pos_k, pos_q, mask, ubatch, c)`.
+
+**Files touched by Section C:** new `src/dca.cpp` + `src/dca.h` (the two builders) **and**
+`llama-kv-cache.{cpp,h}` (the `set_input_dca` host-fill — folded into C) **and** `llama-graph.h`
+(the `build_attn_dca` decls + `llm_graph_input_dca`). dca.cpp compiles against the already-present
+host `ggml_flash_attn_ext_lse` (D-host); the CUDA forward (D-cuda) need not exist until the DSO
+build. **Sub-blocks:** C1 = dca.h + `llm_graph_input_dca` + `set_input_dca` host-fill (this block);
+C2 = `build_attn_dca` graph bodies (both overloads).
+
+## Section C2a (host-fill) DONE (2026-06-10)
+
+`dca.h` (C1) + the host-fill half of C2 landed in the clean-room. Builds independently of the
+`build_attn_dca` graph bodies (next block) and of D-cuda (host `ggml_flash_attn_ext_lse` already
+present from D-host).
+- **`src/dca.cpp`** (new) — `dca_resolve_chunk_size` (cparams.dca_chunk_size or auto =
+  n_ctx_orig_yarn / n_ctx_train), `llm_graph_input_dca::set_input` (delegates to
+  `mctx->set_input_dca`), `::can_reuse` (false — pos/masks are position-specific), and
+  `llm_graph_context::build_attn_inp_dca(mctx_kv)` (allocates pos_k I32[n_kv], 3× pos_q I32[n_tokens],
+  3× mask F32[n_kv,n_tokens,1,1] + F16 cnv; single-stream, mirrors `build_attn_inp_kq_mask`).
+- **`src/llama-kv-cache.cpp`** — `llama_kv_cache::set_input_dca` (reads `v_cells[0]` per-cell
+  positions → `pos_k=cellpos%c`, 3× pos_q, and the 3 partition masks; `n_stream==1` assert like
+  `set_input_pos_bucket`; causal + same-seq + nonempty band test) + the `llama_kv_cache_context`
+  per-batch wrapper.
+- **`src/llama-kv-cache.h`** — `set_input_dca` decl in both classes (×2).
+- **`src/llama-graph.h`** — `build_attn_inp_dca` method decl + `class llm_graph_input_dca;` fwd-decl.
+
+Apply script `backup_models/scripts/dca/apply_section_c_hostfill.py` (5 anchored edits, all OK).
+Verified: kv-cache.cpp ×2, kv-cache.h ×2, graph.h ×2, dca.cpp def ×1, braces balanced.
+
+**⚠ Build-wiring TODO for the cuda.sh/make block (#598):** confirm the cosmocc Makefile globs
+`src/*.cpp` (so the new `src/dca.cpp` compiles) — else add it to the file list explicitly, or
+`build_attn_inp_dca` link-errors.
+
+**REMAINING:** Section C2b (`build_attn_dca` two overloads — rope cached raw-K once with pos_k +
+rope Q thrice + 3× `ggml_flash_attn_ext_lse` under the partition masks + LSE graph-math combine +
+wo/wo_b; declared in llama-graph.h alongside this block's decls). **Before authoring C2b, read
+`get_k`'s returned cache-K layout (llama-kv-cache.cpp:2204) + `build_rope_shift`'s caller
+(~:4000-4030)** to nail the rope-on-cached-K axis (positions vary on the cell axis). → Section B
+(per-arch hooks) → cuda.sh DSO rebuild → RULER-VT gate → capture 0078-dca.patch.
+
+## Section C2b (build_attn_dca bodies) DONE (2026-06-10) — Section C COMPLETE
+
+The rope-on-cached-K axis is settled by `get_k` (llama-kv-cache.cpp:2305): it returns
+`[n_embd_head, n_head_kv, n_kv, 1]` — **cells on ne[2]** — so `ggml_rope_ext(get_k, pos_k, …)` with
+`pos_k` length `n_kv` indexes the cell axis exactly as `inp_pos` indexes live tokens. K is roped
+ONCE; Q thrice; both pre-permute (matching the model rope), then permuted into the flash-attn layout.
+
+- **`src/dca.cpp`** — `build_attn_dca_core` (rope cached raw-K once; read+permute V once; per regime
+  rope Q + `ggml_flash_attn_ext_lse` under `mask_cnv[r]`; extract O-view `[DV,H,NQ,1]@0` + lse-view
+  `[1,H,NQ,1]@DV·nb0`, both `ggml_cont`; combine `M=a+relu(b−a)`, `w=ggml_exp(lse−M)`, `O=Σw·O/Σw`;
+  `reshape_2d`→`build_lora_mm(wo)`+`wo_b`) + the two public overloads (iswa: store to
+  `inp->mctx->get_base()`, asserts `!is_swa`; plain-kv: `inp->mctx`; both guard `k_cur`/`v_cur` for
+  Gemma-4 shared-KV layers).
+- **`src/dca.h`** — `dca_rope` struct (per-layer freq_base/freq_scale/n_rot/freq_factors).
+- **`src/llama-graph.h`** — 2 `build_attn_dca` overloads + `build_attn_dca_core` decls + `dca_rope`
+  fwd-decl. Apply script `apply_section_c2b_decls.py`.
+
+**Empty-regime safety (verified, no code needed):** a chunk-0 query's succ/inter regimes have an
+all-`-inf` mask row → FA O=0 and `streaming_lse_kernel` (fattn.cu:693 `lse = (l>0)?…:-INFINITY`)
+emits `-inf` (the `l<=0`/NaN guard already there) → `w=exp(-inf−M)=0`, dropping the regime cleanly.
+Intra is never empty (diagonal), so `M` is always finite.
+
+**Two runtime caveats for the gate (#598):**
+1. ~~**f16 KV required**~~ — **LIFTED by #444 (DCA all-KV C1, 2026-06-18).** `build_attn_dca_core` no
+   longer asserts f16; `dca_lift_to_f16` dequant-on-lifts a quantized/bf16/turbo cache to a transient
+   f16 for the f16-only fused kernel (the stored cache stays quantized — the long-ctx memory win). Scalar
+   quants (q8_0/q4_0/q4_1/q5_0/q5_1) hop `q8→f32→f16` because CUDA CPY has no direct scalar-quant→f16
+   kernel (that pair CPU-spills); f16 no-op, bf16/turbo cast direct. Host-only (no `.cu` change). Gated
+   `real_frac=0` vs f16-DCA on Qwen3-8B-Q8 at multi-chunk ctx (`.opencoti/m444-dca-allkv-gate.sh`). **#445
+   (C2/C3) extends the gate to the full matrix — q8_0, q5_1, q5_0, q4_0, bf16 + asymmetric q8_0-K/q4_0-V
+   ALL PASS** (`.opencoti/m445-dca-scalar-matrix-gate.sh`): per type `real_frac(<t>-DCA vs f16-DCA) ==
+   real_frac(<t>-noDCA) == 0.0`, so DCA adds zero argmax error beyond the type's own quant loss. (Prefill
+   note: noDCA-quant FA prefill is type-specific — q4_0/bf16 have a fast batched tile-FA kernel ~3.5k
+   tok/s, q8_0/q5_0/q5_1 fall to the decode-VEC path ~12–42 tok/s; DCA's f16-lift prefills uniformly
+   ~1.5k tok/s — a win for the slow types.) q8-DCA
+   decode ≈0.47× f16-DCA, and that 2× is characterized (not a regression to chase away): it is
+   GPU-resident (CPU-spill would be <5 tps) and NOT cuda-graph loss (`GGML_CUDA_DISABLE_GRAPHS=1`
+   changes neither path — the fused DCA op uses no graph). The cost is the dequant-on-lift itself,
+   DOMINATED by the `quant→f32` CPY kernel's `<<<ne, 1>>>` one-thread-per-CUDA-block launch geometry
+   (cpy.cu, ~1/32 warp utilization vs the 256-thread scalar cpy), with the f32 two-pass hop secondary;
+   raw cast bandwidth alone (~3 ms/tok) does not explain it. The single-pass direct `quant→f16` CPY
+   kernel (proper thread geometry, no f32 intermediate) recovers it — C-series perf follow-on at #448.
+   So `-ctk q8_0 -ctv q8_0` is now valid with `--dca on`. The 0078 patch re-capture is batched at #448
+   (umbrella C6); the live source is banked `.opencoti/dca/m444-dca.cpp.banked`.
+2. **FA precision defaults** — unlike `build_attn_mha` (which forces `GGML_PREC_F32` via
+   `ggml_flash_attn_ext_set_prec`, an op-specific call that would assert on `_LSE`), the lse op runs
+   at default prec. If RULER shows precision drift at 1M, force F32 on the O-view in D-cuda's
+   `ggml_cuda_flash_attn_ext_lse` (set the prec slot before dispatching the inner FA).
+
+Verified: dca.cpp 3 defs + braces 26/26; graph.h 3 decls + `dca_rope` fwd; ruff clean.
+
+**Section C COMPLETE** (C1 header + C2a host-fill + C2b bodies). Compiles against host
+`ggml_flash_attn_ext_lse` (D-host); CUDA forward (D-cuda) needed only at the DSO build.
+**REMAINING:** Section B (per-arch hooks: gemma4-iswa global layers + qwen2/3/3moe all layers — gate
+on `cparams.dca_enabled`, pass PRE-rope Q+K, build `build_attn_inp_dca(base)` once, fill `dca_rope`)
+→ cuda.sh DSO rebuild (confirm `src/dca.cpp` globbed) → RULER-VT gate → capture 0078-dca.patch.
+
+## Section B (gemma4) DONE (2026-06-10)
+
+`src/models/gemma4-iswa.cpp` now routes the 5 GLOBAL (`!is_swa`) layers through `build_attn_dca`
+when `cparams.dca_enabled`; SWA layers + the dca-disabled path are byte-unchanged. 7 anchored edits
+(`apply_section_b_gemma.py`):
+- `#include "dca.h"`.
+- before the layer loop: `llm_graph_input_dca * inp_dca = cparams.dca_enabled ?
+  build_attn_inp_dca(inp_attn->mctx->get_base()) : nullptr;` (built once on the base/global context).
+- per layer: `const bool use_dca = cparams.dca_enabled && !hparams.is_swa(il);`.
+- Q-rope + K-rope wrapped in `if (!use_dca) {…}` — DCA layers cache **PRE-rope** Q+K (rope happens
+  per regime inside build_attn_dca). Vcur stays the rms-normed un-roped projection.
+- both `build_attn(…)` call sites (has_kv + shared-KV reuse-KV branch) gain an `if (use_dca)`
+  arm calling `build_attn_dca(inp_attn, inp_dca, wo, nullptr, Qcur, [Kcur|null], [Vcur|null],
+  {freq_base_l, freq_scale_l, n_rot_l, freq_factors}, f_attention_scale, il)`.
+
+DCA reduces to identity for n_ctx ≤ c (all positions in chunk 0 → intra-only, succ/inter empty,
+distances < c → native rope), so enabling `--dca` never regresses short-context serving. Verified:
+2 build_attn_dca calls, 1 inp builder, use_dca 5×, braces 27/27, ruff clean.
+
+**REMAINING:** Section B (qwen2/qwen3/qwen3moe — all layers full-attention, `use_dca =
+cparams.dca_enabled`, same PRE-rope Q+K + `build_attn_inp_dca(static_cast<…kv_cache_context*>(mctx))`
+on the plain-kv overload) → cuda.sh DSO rebuild (confirm `src/dca.cpp` globbed) → RULER-VT gate
+(Gemma-4 A4B first, f16 KV) → capture 0078-dca.patch.
+
+## Section B (qwen) DONE (2026-06-10) — Section B COMPLETE → ALL SOURCE AUTHORED
+
+`qwen2.cpp` / `qwen3.cpp` / `qwen3moe.cpp` route **all** layers through `build_attn_dca` when
+`cparams.dca_enabled` (Qwen = the canonical training-free 1M arch; every layer is full attention).
+The Q/K-rope + `build_attn` blocks are byte-identical across the three files, so one shared anchor
+set (`apply_section_b_qwen.py`, 5 edits ×3) applies to each: `#include "dca.h"`; `use_dca =
+cparams.dca_enabled` + `inp_dca = build_attn_inp_dca(inp_attn->mctx)` (plain-kv context, no
+get_base); Q/K rope wrapped in `if (!use_dca)`; the `build_attn` site gains an `if (use_dca)` arm
+calling `build_attn_dca(..., model.layers[il].wo, model.layers[il].bo, Qcur, Kcur, Vcur,
+{freq_base, freq_scale, n_rot, nullptr}, 1/√head, il)` (wo_b = bo carried; qwen3/3moe `wo_s`
+post-scale untouched). Verified: each file 1 dca-call, 1 inp builder, use_dca 5×, braces balanced,
+ruff clean.
+
+**ALL 0078-dca SOURCE AUTHORED** — A (flags) + D (LSE op host+CUDA) + C (input + build_attn_dca) +
+B (gemma4 globals + qwen2/3/3moe all layers). Nothing compiled yet — the single `cuda.sh` DSO build
+(#598) compiles A+C+D+B together. New files: `src/dca.{h,cpp}`. Apply scripts (8) in
+`backup_models/scripts/dca/`.
+
+**REMAINING (#598, the build+gate+land block):**
+1. **Build wiring** — confirm the cosmocc Makefile globs `src/*.cpp` (so `src/dca.cpp` compiles);
+   add it explicitly if not. Then `bun --cwd packages/opencoti-llamafile script/build-pipeline.ts
+   make` + `… cuda` (the multi-hour `ggml-cuda.so` DSO rebuild, sm_86). Fix any compile errors
+   surfaced (first real compile of A/C/D/B).
+2. **RULER-VT gate** — serve A4B Q4_K_M (or bf16) with `--dca on --override-kv
+   gemma4.context_length=int:1048576`, **f16 KV** (NOT `-ctk q8_0` — DCA core asserts f16), default
+   prec. RULER-VT @ ≥256k vs the naive-extension bar (VT=0.36 from T87.ext). Then a Qwen 1M model.
+3. **Capture** — `git diff 43ccc8a56` in the clean-room → `0078-dca.patch`; write patch + README row
+   + UPSTREAM_SYNC entry + `.opencoti/dca-ruler-gate.sh` into the HELD opencoti repo (top-level
+   tracked only; submodule never touched); round-trip `apply` verify.

docs/features/introspection.md

@@ -0,0 +1,245 @@
+# opencoti runtime introspection & control surface (#676/#677)
+
+Complete reference for **observing and controlling the opencoti agentic-serving
+features** (PolyKV, SharedKVPool, rolling-KV residency, DCA, MTP/spec-decode,
+session KV-reuse, retention eviction, sparse-attn, RYS) — what is configurable
+at boot, what a client can set per request, and what the server reports back at
+runtime.
+
+Status: implemented 2026-07-15 (tasks #676 `/props` echo + core KV-info API,
+#677 `/slots` enrichment). Vendored-source additive edits; host-only (no CUDA).
+
+---
+
+## 1. The three planes
+
+| Plane | Mechanism | When |
+|---|---|---|
+| **Boot knobs** | CLI flags (`common/arg.cpp`) ← typed TS adapter fields ← `OPENCOTI_LLAMAFILE_*` env | server launch |
+| **Per-request control** | 3 JSON body fields (`session_id`, `shared_pool_slot`, `shared_prefix_n_tokens`) | every completion request |
+| **Runtime introspection** | `/props → "opencoti"` (boot-state echo + effective KV), `/slots → per-slot "opencoti"` (live state), completion `timings` fields | any time |
+
+---
+
+## 2. Boot knobs — by feature
+
+Typed fields live in `packages/opencoti-llamafile/src/config.ts` (each maps to
+env `OPENCOTI_LLAMAFILE_<SNAKE_CASE>`); CLI flags are registered in
+`vendors/sources/llamafile/llama.cpp/common/arg.cpp`. Flags marked *(extraArgs
+only)* have no typed adapter field yet.
+
+### PolyKV — KV-quant tiers (M6)
+- `ctypeK` / `ctypeV` → `-ctk` / `-ctv` — window KV types; scalar
+  (`f16,bf16,q8_0,q6_0,q5_1,q5_0,q4_0`) + turbo (`turbo2/3/4/8`) + TCQ
+  (`turbo2_tcq,turbo3_tcq`).
+- `ctypeKTail` / `ctypeVTail` → `-ctkt` / `-ctvt` — **spilled-tail** types
+  (#622 P2 mixed-KV); unset = same as window.
+- `kvAutoTier` → env **`OPENCOTI_KV_AUTO_TIER=1`** — boot policy picks the
+  least-compressing scalar pair that stays fully resident (#621). Explicit
+  `-ctv` disables auto entirely; explicit `-ctk` holds K, walks V only. Dense
+  full-attention models only; iSWA keeps f16.
+- env **`OPENCOTI_KV_AUTO_TIER_TAIL=1`** — when auto-tier still spills, also
+  auto-pick a q4_0 tail (separate opt-in).
+
+### SharedKVPool (M6-S1) — per-request only, see §3. No boot flag.
+
+### Rolling-KV residency / spill (M7, #582)
+- `vramTargetMiB` → `--vram-target` — VRAM budget cap (MiB); 0 = free-VRAM auto.
+- `kvResidencyMode` → `--kv-residency-mode` — residency tactic knob (0 = auto).
+- `headinferGpuHeadsFrac` → `--headinfer-gpu-heads-frac` (default `auto`).
+
+### Session KV-reuse (M0) + host-state guard
+- `nParallel` → `--parallel`, `slotPromptSimilarity` → `-sps`,
+  `cacheReuse` → `--cache-reuse`, `ctxSize` → `-c`.
+- `cacheRamMiB` → `--cache-ram`, `ctxCheckpoints` → `--ctx-checkpoints`
+  (bounded host-RAM defaults, bug-1325).
+- `--kv-unified` — required for assistant-MTP with `--parallel > 1` (bug-2110 guard).
+
+### Retention eviction (M1)
+- `restKvEviction` → `--rest-kv-eviction`, `restKvRecent` → `--rest-kv-recent`,
+  `restKvLayer` → `--rest-kv-layer`. Boot-global (no per-request override).
+
+### DCA — Dual Chunk Attention (0078)
+- `dcaEnabled` → `--dca on|off`, `dcaChunkSize` → `--dca-chunk-size`,
+  `dcaYarnFactor` → `--dca-yarn-factor`.
+
+### Spec-decode / MTP (M6-S4, bug-858)
+- `mtpHead` → `--mtp-head` (alias of `-md`; gemma4-assistant GGUF),
+  `specType` → `--spec-type` (`draft-assistant` | `draft-mtp` | …),
+  `draftNGpuLayers` → `-ngld`, `--spec-draft-n-max` / `--spec-draft-n-min`,
+  `--draft-block-size`, draft KV types `-ctkd` / `-ctvd`.
+
+### Sparse attention (#551) *(extraArgs only)*
+- `--sparse-attn`, `--sparse-attn-block-size/-topk/-recent/-sink/-refresh/-mode`.
+
+### RYS layer duplication (#656) 
+- `repeatLayers` → `--repeat-layers`; probe: `--rys-probe*` *(extraArgs only)*.
+
+### Lazy slot context (F4 M3) *(server: extraArgs only; typed on embedder)*
+- `--slot-initial-ctx`, `--slot-shrink-idle-ms`.
+
+---
+
+## 3. Per-request control (JSON body fields)
+
+Parsed in `tools/server/server-task.cpp` (~line 275):
+
+| Field | Default | Feature | Effect |
+|---|---|---|---|
+| `session_id` | `""` | M0 session KV-reuse | Slot affinity: the same session routes back to the slot that holds its KV (prevents cross-session eviction at `--parallel > 1`). opencode gets this injected automatically by `withKvReuseFetch` (which also forces `cache_prompt: true`). |
+| `shared_pool_slot` | `-1` | SharedKVPool (M6-S1) | Bit-share this request's prefix KV with pool slot N (read-only). |
+| `shared_prefix_n_tokens` | `0` | SharedKVPool | Length of the shared prefix in tokens. |
+
+Everything else (tier, residency, DCA, retention) is **boot-global by design** —
+changing it requires a relaunch.
+
+---
+
+## 4. Runtime introspection
+
+### 4.1 `GET /props` → `"opencoti"` object (#676)
+
+Boot-state echo, computed once at model load (sleep-safe — never touches the
+live context at request time). Shape:
+
+```jsonc
+"opencoti": {
+  "kv": {
+    "cache_type_k": "q8_0",          // configured (-ctk)
+    "cache_type_v": "q4_0",          // configured (-ctv)
+    "cache_type_k_tail": null,       // configured (-ctkt); null = same as window
+    "cache_type_v_tail": null,
+    "auto_tier": false,              // OPENCOTI_KV_AUTO_TIER set
+    "auto_tier_tail": false,         // OPENCOTI_KV_AUTO_TIER_TAIL set
+    "effective": {                   // READ BACK from the live KV cache —
+      "type_k": "q8_0",              // reflects auto-tier + residency sizing,
+      "type_v": "q4_0",              // not just what was asked for
+      "type_k_tail": "q8_0",         // == window type when no distinct tail
+      "type_v_tail": "q4_0",
+      "n_cells": 524288,             // allocated KV cells per stream
+      "n_cells_resident": 524288,    // device-resident window (min over spilling layers)
+      "n_layers_spilling": 0,        // layers with an engaged rolling-KV window
+      "fully_resident": true,
+      "is_iswa": true                // reported from the iSWA base (full-attn) cache
+    }
+  },
+  "residency":  { "kv_residency_mode": 0, "vram_target_mib": 0, "headinfer_gpu_heads_frac": 1.0 },
+  "dca":        { "enabled": true, "chunk_size": 0, "yarn_factor": 1.0 },
+  "sparse_attn":{ "enabled": false, "block_size": 128, "topk": 0, "recent": 0, "sink": 1, "refresh": 8, "mode": 0 },
+  "speculative":{ "types": ["none", "draft-assistant"],   // "none" is always slot 0 of the type list
+                  "draft_model": "…assistant-Q8_0.gguf",
+                  "n_max": 3, "n_min": 0, "draft_block_size": 0, "n_gpu_layers": 99,
+                  "cache_type_k": "f16", "cache_type_v": "f16" },
+  "kv_reuse":   { "n_parallel": 4, "kv_unified": true,    // kv_unified auto-forced for assistant-MTP (bug-2110)
+                  "cache_reuse": 0, "slot_prompt_similarity": 0.1,
+                  "ctx_checkpoints": 32, "cache_ram_mib": 8192 },
+  "rest_kv":    { "eviction": false, "recent": 256, "layer": -1 },
+  "repeat_layers": null
+}
+```
+
+(Verified live 2026-07-15 against the v7-coder server on :8231 — values above are the
+actual output for `-ctk q8_0 -ctv q4_0 -c 524288 --dca on --spec-type draft-assistant`.)
+
+The `kv.effective` block is the **only authoritative source for the auto-tier
+decision** — the decision happens inside the KV-cache constructor and is read
+back through the new core API (§5). `configured != effective` is expected
+whenever auto-tier engaged. Drafter note: on dual-ctx MTP the target context is
+reported; the drafter's own KV layer is f16 by design.
+
+### 4.2 `GET /slots` → per-slot `"opencoti"` object (#677)
+
+Live state per slot (requires `--slots`, i.e. `params.endpoint_slots`):
+
+```jsonc
+{
+  "id": 3, "n_ctx": 131072, "speculative": true, "is_processing": false,
+  "opencoti": {
+    "draft_n_total": 114,            // LIFETIME draft tokens generated (this slot) —
+    "draft_n_accepted": 62,          //   survives task end; resets only on server restart
+    "draft_acceptance": 0.544,       // lifetime ratio; null before any draft ran
+    "session_id": "ses_09f4…",       // current/last task's session binding (M0)
+    "shared_pool_slot": -1,          // current/last task's SharedKVPool binding
+    "shared_prefix_n_tokens": 0
+  }
+}
+```
+
+The draft counters are **lifetime accumulators** (deliberately not cleared by the
+slot's per-task reset — the per-task counters feed `timings.draft_n` instead). To
+watch a single long turn, poll twice and diff. The session/pool fields appear only
+once the slot has served at least one task (they come from the current-or-last
+task's params); a freshly booted idle slot shows just the three draft fields.
+
+Practical use: **draft_acceptance ≳ 0.95 sustained at turn end is the
+rumination/loop tell** (healthy agentic decode sits ~0.4–0.9); this is now
+pollable instead of log-scraping.
+
+### 4.3 Per-completion `timings` (pre-existing, unchanged)
+
+When timings are requested, each response carries: `cache_n` (KV prefix-reuse
+hits — the M0/PolyKV effect), and `draft_n` / `draft_n_accepted` for that
+response's spec rounds.
+
+### 4.4 Still log-only (known gaps)
+
+PolyKV pool share/reject lines (`poly-kv-pool: shared …`), retention-eviction
+discards, rolling-KV tactic selection detail, auto-tier WARN line. Engine
+events do **not** feed opencoti-server `/v1/tier-events` (that store records
+tier-*routing* only).
+
+---
+
+## 5. Core C API (for embedders / tools)
+
+Added in `include/llama.h`:
+
+```c
+struct llama_opencoti_kv_info {
+    enum ggml_type type_k, type_v;            // effective window types (post auto-tier)
+    enum ggml_type type_k_tail, type_v_tail;  // spilled-tail types (== window when none)
+    uint32_t n_cells;                         // allocated KV cells per stream
+    uint32_t n_cells_resident;                // min resident window across spilling layers
+    uint32_t n_layers_spilling;               // 0 = fully resident
+    bool     is_iswa;                         // reported from iSWA base cache
+};
+LLAMA_API bool llama_memory_opencoti_kv_info(llama_memory_t mem,
+                                             struct llama_opencoti_kv_info * info);
+```
+
+Dispatch: virtual on `llama_memory_i` (default false → pure-recurrent memories
+report nothing); `llama_kv_cache` reads its live tensors; `llama_kv_cache_iswa`
+forwards to its non-SWA base; `llama_memory_hybrid` forwards to its attention
+child. Boot-static — safe to read once after context creation and cache (the
+server does exactly that in `get_meta()`).
+
+Touched files: `include/llama.h`, `src/llama-memory.h`,
+`src/llama-kv-cache.{h,cpp}`, `src/llama-kv-cache-iswa.{h,cpp}`,
+`src/llama-memory-hybrid.{h,cpp}`, `src/llama-context.cpp`,
+`tools/server/server-context.{h,cpp}`, plus `llamafile/BUILD.mk` (bug-2186: the
+llamafile-mode `server.cpp.o` rule had no header deps — a stale object linked an
+old inline `update_meta` with a mismatched `server_context_meta` layout, silently
+yielding `"opencoti": null`; the rule now depends on `tools/server/*.h`). All
+additive; off-path behavior byte-identical (pure read-side additions to two JSON
+endpoints).
+
+## 6. Quick recipes
+
+```sh
+# What is this server actually running?
+curl -s :8231/props | jq .opencoti
+
+# Did auto-tier change my KV types?
+curl -s :8231/props | jq '.opencoti.kv | {configured: {k:.cache_type_k, v:.cache_type_v}, effective}'
+
+# Is anything spilling?
+curl -s :8231/props | jq .opencoti.kv.effective.fully_resident
+
+# Live loop-tell: per-slot draft acceptance
+curl -s :8231/slots | jq '.[] | {id, processing:.is_processing, acc:.opencoti.draft_acceptance}'
+```
+
+Related docs: [advanced_kv.md](advanced_kv.md) (M0-M3), [poly_kv.md](poly_kv.md)
+(M6/SharedKVPool), [rolling_kv.md](rolling_kv.md) (M7 residency),
+[gemma4_dca.md](gemma4_dca.md), [mtp.md](../evaluations/mtp.md),
+[tstar_auto_spill.md](tstar_auto_spill.md) (auto-tier/auto-spill policy).

docs/features/llamafile_build.md

@@ -0,0 +1,444 @@
+# F4 M6 — Production llamafile build pipeline
+
+Owns: producing the `.llamafile` artifacts the embedder package
+consumes at runtime, from source under `vendors/sources/llamafile/`
+plus the opencoti patch series at `vendors/patches/llamafile/`.
+
+The build orchestrator is
+`packages/opencoti-llamafile/script/build-pipeline.ts`. Helpers live
+at `packages/opencoti-llamafile/src/build-pipeline.ts`.
+
+## Why this exists
+
+Through F4 M0-M4 the embedder ran a prebuilt
+`mann1x/claude-hooks v1.4.0 qwen3-embedding-0.6b-16k.llamafile`
+asset pinned at `vendors/pin/embedder.txt`. That binary embeds
+`ggml-cuda.so` and a Qwen3 GGUF, and predates every opencoti
+patch (lazy slot context, shrink-on-idle, et al.). For F4
+production we need:
+
+- **All backend DSOs built from source** (CUDA / ROCm / Vulkan),
+  not a runtime side-load nobody controls.
+- **The opencoti patch series actually compiled in** — the prebuilt
+  rejects `--slot-initial-ctx` and `--slot-shrink-idle-ms` because
+  those flags don't exist in upstream 0.10.1.
+- **No cross-repo dependency** on `mann1x/claude-hooks` for an
+  opencoti artifact.
+
+The result is one or two artifacts under `dist/llamafile/`:
+
+- `opencoti-llamafile-<ver>-<arch>.llamafile` — thin binary +
+  every staged DSO. The "GPU runtime", no model bundled.
+- `opencoti-embedder-<ver>-<arch>.llamafile` — the above + an
+  embedded GGUF + `.args`. The embedder package's launcher target.
+
+Each artifact has a sibling `MANIFEST.json` recording SHA256 of
+the artifact, the embedded patches, and per-DSO digests.
+
+## Toolchains per backend
+
+| Backend | Build script                                    | Requirement              | Probe env                    | Default path           |
+|---------|-------------------------------------------------|--------------------------|------------------------------|------------------------|
+| CUDA    | `vendors/sources/llamafile/llamafile/cuda.sh`   | NVIDIA CUDA 12.x or 13.x | `OPENCOTI_CUDA_PATH`         | `/usr/local/cuda-12.6` |
+| ROCm    | `vendors/sources/llamafile/llamafile/rocm.sh`   | AMD ROCm 5+ (hipcc)      | `OPENCOTI_ROCM_PATH`         | `/opt/rocm`            |
+| Vulkan  | `vendors/sources/llamafile/llamafile/vulkan.sh` | Vulkan SDK 1.3+ (glslc)  | `OPENCOTI_VULKAN_SDK`        | system `glslc` on PATH |
+
+Each backend probe runs read-only — missing toolchain produces a
+WARN and the backend is skipped, not a hard build failure. Only
+the `package` subcommand hard-fails if zero backends produced a
+DSO (and `--cpu-only` wasn't passed explicitly).
+
+### CUDA version per GPU generation
+
+Mozilla-Ocho's `cuda.sh` enables different `-gencode` arch lists
+depending on the detected CUDA toolkit version:
+
+| Target GPU                         | Compute cap   | Minimum CUDA | Notes                                                                    |
+|------------------------------------|---------------|--------------|--------------------------------------------------------------------------|
+| Turing (RTX 20xx, T4)              | sm_75         | 12.x         | Baseline arch — included in every build                                  |
+| Ampere (A100, RTX 30xx)            | sm_80 / sm_86 | 12.x         | Baseline arch                                                            |
+| Ada Lovelace (RTX 40xx)            | sm_89         | 12.x         | Baseline arch                                                            |
+| Hopper (H100)                      | sm_90         | 12.x         | Baseline arch                                                            |
+| **Blackwell consumer (RTX 50xx)**  | **sm_120**    | **13.x**     | Requires CUDA 13.x branch — adds `sm_120f` + `--compress-mode=size` (4x DSO size reduction) |
+| Blackwell Jetson Thor (aarch64)    | sm_110        | 13.x         | aarch64 only                                                             |
+| Blackwell DGX Spark GB10 (aarch64) | sm_121        | 13.x         | aarch64 only                                                             |
+
+**For RTX 50xx targets, set `OPENCOTI_CUDA_PATH=/usr/local/cuda-13.x`
+when building** — the 12.x branch will emit a DSO with no sm_120
+kernels and the driver cannot JIT-fallback (the `code=sm_X` form
+embeds SASS only, not PTX). Symptom on RTX 50xx with a 12.x-built
+DSO: `ggml-cuda.cu:97: CUDA error` during first kernel launch.
+
+## End-to-end pipeline
+
+```bash
+# 1. Reset vendored sources, init nested submodules, apply
+#    Mozilla-Ocho's own patches, then apply opencoti's series.
+bun run build:llamafile:apply
+
+# 2. Build the thin patched llamafile binary (no DSOs embedded yet).
+bun run build:llamafile        # = apply + make
+# Output: vendors/sources/llamafile/o/llamafile/llamafile
+
+# 3. Build every available backend DSO. Soft-fails per backend on
+#    missing toolchain; hard-fails only if zero built.
+bun run build:llamafile:all-backends
+# Output (per backend that built):
+#   vendors/dist/llamafile/<backend>/<ver>/ggml-<backend>.so
+#   ~/.llamafile/v/<ver>/ggml-<backend>.so   (runtime side-load mirror)
+
+# 4. Assemble production artifacts.
+bun run build:llamafile:package
+# Always emits:
+#   dist/llamafile/opencoti-llamafile-<ver>-<tag>-<arch>.llamafile
+#   dist/llamafile/opencoti-llamafile-<ver>-<tag>-<arch>.llamafile.MANIFEST.json
+
+# 5. Or, for the composite-embedder asset (binary + DSOs + Qwen3 GGUF + .args):
+bun run build:llamafile:package -- --with-model ~/.opencoti/qwen3-embedding-0.6b.gguf
+# Additionally emits:
+#   dist/llamafile/opencoti-embedder-<ver>-<arch>.llamafile
+#   dist/llamafile/opencoti-embedder-<ver>-<arch>.llamafile.MANIFEST.json
+```
+
+Iteration on a single backend uses its dedicated subcommand
+(`build:llamafile:cuda`, `:rocm`, `:vulkan`). The `:make` subcommand
+re-runs cosmocc-make alone — keeps in-tree edits in place while
+you're polishing a patch you haven't yet captured under
+`vendors/patches/llamafile/`.
+
+## `package` subcommand details
+
+```
+build:llamafile:package [--with-model PATH] [--with-args PATH] [--arch ID] [--cpu-only]
+```
+
+- `--with-model PATH` — GGUF to embed. Filename is preserved; the
+  embedded `.args` points at `/zip/<basename>`.
+- `--with-args PATH` — explicit `.args` to embed instead of the
+  generated one. The generated default matches `buildEmbedderArgs()`
+  in `packages/opencoti-embedder/src/manager.ts` (--server,
+  --embedding, --pooling last, --ctx-size 16384, --parallel 2,
+  --slot-initial-ctx 4096, --slot-shrink-idle-ms 30000).
+- `--arch ID` — architecture string in the output filename.
+  Detected from `os.arch()` by default (`x86_64`/`aarch64`).
+- `--cpu-only` — skip embedding any DSO. Produces a binary that
+  works on any host but always runs on CPU. Useful for CI smoke
+  on machines with no GPU.
+
+Embedding uses the vendored zipalign at
+`vendors/sources/llamafile/o/third_party/zipalign/zipalign`. For DSOs
+the `-j0` flag strips directory prefixes so cosmo_dlopen resolves
+the entry as the bare filename it expects.
+
+## Embedder package: local-asset modes (dev escape hatch)
+
+The embedder downloader at
+`packages/opencoti-embedder/src/download.ts` understands two
+local-asset modes added in M6 to avoid a GitHub Release round trip
+during dev cycles:
+
+### `file:` pin
+
+Change the first field of `vendors/pin/embedder.txt` from
+`<owner>/<repo>` to `file:<path>`. Path is anything `path.resolve()`
+can interpret (absolute or relative to caller cwd). The SHA256 is
+still verified.
+
+```
+file:./dist/llamafile/opencoti-embedder-0.10.1-x86_64.llamafile  v0  opencoti-embedder.llamafile  <sha256-from-manifest>
+```
+
+### `OPENCOTI_EMBEDDER_LOCAL` env
+
+Bypasses the pin's path entirely while still verifying SHA. Set
+`OPENCOTI_EMBEDDER_LOCAL=$(pwd)/dist/llamafile/opencoti-embedder-…llamafile`
+and run the embedder smoke as usual — the downloader copies bytes
+from disk instead of fetching from GitHub.
+
+Both modes still write to `destDir/<asset>` so the rest of the
+launcher pipeline works unchanged.
+
+## Verifying a built / deployed binary carries the full patch chain
+
+A dev `o/` build (`:make` + `:cuda`) emits **no** `MANIFEST.json` — only
+`package`/`release-cut` do. So when you have a binary of uncertain vintage
+(an `o/` artifact after a source reset, or a copy deployed to another host)
+and need to answer *"is this the full all-patches build, or vanilla/partial?"*
+**without** a 45–86 min rebuild, verify by symbols + arg names. The vendored
+`llama.cpp` tree is normally **clean** at rest (patches apply at build time and
+the tree is reset for upstream-syncability), so a clean tree + a feature-rich
+binary is the *expected* steady state — never infer "missing patches" from a
+clean source tree; inspect the artifact instead.
+
+**CUDA DSO** (`~/.llamafile/v/<ver>/ggml-cuda.so` and
+`vendors/dist/llamafile/cuda/<ver>/ggml-cuda.so` — must be byte-identical, G4):
+
+```sh
+nm -D --defined-only ~/.llamafile/v/<ver>/ggml-cuda.so | c++filt | \
+  grep -ciE 'dca_fused_case|ggml_flash_attn_ext_dca_fused|ggml_cuda_flash_attn_ext_lse'  # DCA  (0078+0081)
+# expect: DCA family ~150 syms incl ~144 dca_fused_case<…> (the 0082/#531 shard),
+#         turbo ~24 (ggml_turbo_wht, turbo_cpu_fwht_inverse, ggml_cpy_turbo_f16_cuda),
+#         tcq ~12, q6_0 present. Zero DCA/turbo syms ⇒ vanilla or pre-0078 DSO.
+```
+
+**Host binary** (cosmo APE — `nm` won't read it; use `strings`):
+
+```sh
+strings -n4 vendors/sources/llamafile/o/llamafile/llamafile | \
+  grep -E '^(q6_0|turbo[2348]|turbo[23]_tcq|dca-chunk-size|QK_TURBO3_TCQ)$'
+# all present ⇒ arg.cpp kv_cache_types + DCA flag patches (0073/0078/0081) are in.
+```
+
+**Host vs DSO mtime split is benign.** `:make` (host) and `:cuda` (DSO) build
+independently; when the most recent patches are CUDA/build-only (e.g. 0082
+build-parallelism, 0083 tcq-warp, 0084 cuda-graphs), only the DSO is rebuilt, so
+the host binary legitimately predates the DSO. Verify by *symbols*, not mtime.
+To make a build self-describing, run `package` (emits the sibling
+`MANIFEST.json` with per-artifact SHA256); the in-tree `o/` build has none.
+
+## What's NOT in M6
+
+These are tracked separately and intentionally out of M6 scope:
+
+- **GitHub Releases publish pipeline.** The artifacts produced by
+  `package` are local; uploading them as a `mann1x/opencoti`
+  release asset (and flipping the pin away from claude-hooks) is
+  the M7 ticket.
+- **macOS / Metal backend.** `metal.c` is macOS-only; the M6
+  pipeline is Linux-first. Adding macOS needs a Darwin CI runner.
+- **Cross-arch builds.** cosmocc APE is dual-arch by design, but
+  ggml-*.so are arch-specific. The `<arch>` token in the output
+  filename is anticipatory; M6 ships x86_64 only.
+- **`build:llamafile:cuda --cublas`.** Mozilla-Ocho's cuda.sh
+  optionally links against vendor cuBLAS instead of TinyBLAS. F4
+  defaults to TinyBLAS (works on every NVIDIA GPU without runtime
+  CUDA install); cuBLAS is a follow-up if profile-guided choices
+  ever need it.
+
+## Verification checklist
+
+After a successful `package` run, the manifest tells you exactly
+what was embedded. To cross-check from the shell:
+
+```bash
+# Confirm DSOs are inside the bare artifact.
+unzip -l dist/llamafile/opencoti-llamafile-0.10.1-x86_64.llamafile | grep "ggml-"
+
+# Confirm the manifest matches the artifact's actual SHA.
+sha256sum dist/llamafile/opencoti-llamafile-0.10.1-x86_64.llamafile
+jq -r .sha256 dist/llamafile/opencoti-llamafile-0.10.1-x86_64.llamafile.MANIFEST.json
+```
+
+The embedder smoke at
+`packages/opencoti-embedder/script/smoke.ts` runs the full launcher
+against whatever asset the downloader resolves. Set
+`OPENCOTI_EMBEDDER_LOCAL` to point at a freshly-built composite to
+exercise the new patches without a release upload.
+
+## Incremental rebuilds, header dependencies, and ccache
+
+Iterating the vendored patch series (F4/F5) means editing C++ inside
+`vendors/sources/llamafile` and rebuilding with
+`bun run build:llamafile:make` (the incremental path — it does **not**
+reset the tree, so in-flight edits survive). Two build-infra facts make
+that loop work; both bit hard before being fixed (see buglog `bug-170`).
+
+### Header dependency tracking (`0007-build-header-deps.patch`)
+
+The upstream build resolves header deps via cosmopolitan's `mkdeps`
+(`build/deps.mk` → `o//depend`). `mkdeps` only follows cosmopolitan's
+root-relative include style; llama.cpp sources use `#include "foo.h"`
+resolved through `-iquote` dirs, which `mkdeps` cannot follow — so it
+emitted **zero** dependency edges for `llama.cpp/src` headers. Editing a
+header therefore never marked its dependent objects stale, and `make`
+reported them "up to date" → the only way to push a header change through
+was to delete objects by hand and rebuild from scratch.
+
+Patch `0007` adds `-MMD -MF $@.d` to the `COMPILE.c`/`COMPILE.cc`
+commands so cosmocc emits an accurate per-translation-unit `<obj>.o.d`
+(project headers only), and `-include`s those files in `build/deps.mk`.
+clang resolves the real `-iquote` includes, so the header set is exact. A
+header edit now recompiles **exactly** its dependents (both fat-object
+arches, via the existing rule) and nothing else.
+
+The explicit `-MF $@.d` is **load-bearing, not cosmetic** — and it's
+what made ccache start working at all (see below). cosmocc defaults the
+dep file to `<obj>.d` by *appending* (`foo.cpp.o` → `foo.cpp.o.d`), but
+ccache derives the expected name from `-o foo.cpp.o` by *stripping* `.o`
+(→ `foo.cpp.d`). With bare `-MMD` the two disagree: ccache `stat`s a file
+cosmocc never wrote, logs *"failed to stat …foo.cpp.d"*, and aborts every
+compile with *"ccache internal error"* → **nothing is ever cached** (the
+cache sat at 0 files). Pinning `-MF` to cosmocc's actual name makes them
+agree, and ccache caches/restores normally.
+
+> The **first** build after this patch lands is a one-time full compile
+> that populates the `.o.d` set. Every build after that is correctly
+> incremental. Verify with: rebuild a TU, `touch` a header it includes →
+> only its dependents recompile; `touch` a header it does *not* include →
+> nothing recompiles.
+
+### ccache + cosmocc fat objects
+
+ccache (enabled by default; cache at `.opencoti/ccache/`, disable with
+`OPENCOTI_NO_CCACHE=1`) wraps cosmocc's **fat** driver: one
+`cosmoc++ -c -o foo.o` writes **both** `foo.o` (x86-64) and
+`.aarch64/foo.o`, but ccache only knows the x86 `-o` output. So a cache
+**hit restores the x86 object only — never its `.aarch64` twin**. A hit
+is therefore safe only when the twin already exists on disk.
+
+Consequences, and how they're handled:
+
+- **Never delete a single-arch object** to force a rebuild. If you must
+  force one, `touch` its source (or delete *both* `foo.o` **and**
+  `.aarch64/foo.o`). Deleting only the x86 object and rebuilding makes
+  ccache restore it while leaving no aarch64 twin → the APE link fails
+  with *"concomitant .aarch64 file missing"*. With header tracking fixed,
+  manual deletion is no longer needed at all.
+- **The full `build` resets the tree** (`git clean -fdx` wipes `o/`). A
+  warm cache restoring x86-only objects into an empty tree would orphan
+  every aarch64 twin. So `runBuild` sets `CCACHE_RECACHE=1`: cosmocc
+  re-runs and writes both arches while still refreshing the cache for the
+  next incremental `:make`. The incremental path runs against a populated
+  `o/`, where hits are safe, so it uses the cache normally.
+
+## Troubleshooting — DSO load & symbol resolution
+
+These two cost ~1h during M6-S2 (turbo3 KV). Both masquerade as logic
+bugs. **Do the diagnostic-first steps before touching any source.**
+
+### 1. A CUDA change has "no effect" → confirm which `.so` is actually loaded
+
+The dev binary probes for `ggml-cuda.so` at **two** paths and loads the
+first that resolves:
+
+1. `~/.llamafile/v/<version>/ggml-cuda.so` — the cache. `build:llamafile:cuda`
+   mirrors the freshly-built DSO **here only**.
+2. `$HOME/ggml-cuda.so` — `cuda.sh`'s default `OUTPUT=${HOME}/ggml-cuda.so`.
+
+A stale `$HOME/ggml-cuda.so` (left from an old manual `cuda.sh` run)
+silently shadows every rebuild that only updated the cache; the boot log
+gives no hint. Before theorizing about source/object/enum, confirm the
+loaded inode:
+
+```sh
+pid=$(pgrep -f 'llamafile.*--server' | head -1)
+grep ggml-cuda.so /proc/$pid/maps        # which path is mapped
+stat -c '%i %n' /root/ggml-cuda.so vendors/dist/llamafile/cuda/*/*/ggml-cuda.so
+```
+
+If the loaded inode isn't the file you just built, sync it (`cp` the fresh
+DSO over `$HOME/ggml-cuda.so`, or `rm` it to force the cache path).
+
+### 2. "no usable GPU found" with a healthy GPU == DSO `dlopen` failure
+
+The dlopen error is swallowed at default verbosity. **Re-run with
+`--verbose`** to get the real line, e.g.
+`cuda: INFO: failed to load library .../ggml-cuda.so: undefined symbol: dequantize_row_turbo2_0`.
+
+### Adding a new `ggml/src/*.c` core source — it must land in THREE places
+
+The DSO is **self-contained**: `llamafile/build-functions.sh`
+`ggml_core_sources` compiles `ggml.c` + `ggml-quants.c` into the `.so` as
+`ggml-core-*.o`. `ggml.c`'s `type_traits` table holds **data-relocation
+function pointers** to every type's CPU `to_float`/`from_float` impl —
+resolved **eagerly at dlopen**, so a missing impl is an undefined-symbol
+*load* failure even with `RTLD_LAZY`. A new core `.c` (e.g.
+`ggml-turbo-quant.c`) must therefore be added to:
+
+1. `llamafile/build-functions.sh` → `ggml_core_sources` (DSO self-compiles it).
+2. `llama.cpp.patches/llamafile-files/BUILD.mk` → `GGML_SRCS_C` (host links it).
+3. **Force the host relink** — `make` does *not* propagate a `GGML_SRCS_C`
+   addition into the final binary. `rm o/llamafile/llamafile o/llama.cpp/llama.cpp.a`,
+   rebuild, and verify the binary mtime advanced.
+
+If a previously-working file goes missing from these lists, suspect a
+`build:llamafile:reset` having wiped outer-vendor edits
+([DESTRUCTIVE_OPS.md](../protocols/DESTRUCTIVE_OPS.md)).
+
+## #613 — the packaged engine as a zero-dependency single-file product
+
+Shipped 2026-07-05. The `package` subcommand's bare artifact is now the
+PRIMARY deliverable of the llamafile lane: a self-contained APE that runs the
+full opencoti inference engine (llama-server OpenAI-compatible HTTP API + CLI)
+with **zero external dependencies** — no opencoti framework, no side files.
+
+### Versioning
+
+`opencoti-<llamafileVersion>-<opencotiTag>`, e.g. `opencoti-0.10.3-c2`. The
+tag lives in the committed `vendors/llamafile/OPENCOTI_TAG` (comment-tolerant,
+same format as the pin file; bump on every packaged cut whose binary or DSO
+bytes change). C-series lineage: `c1` = the un-packaged split deploy on bs2
+(thin binary + external side-loaded DSO at `/srv/ml/opencoti-c1/`), `c2` =
+first packaged single-file cut. The MANIFEST.json records `versionString`,
+`opencotiTag`, `gitCommit`, the full patch list, and the sha256 of the
+artifact AND of every embedded DSO — the binary⇄DSO pairing is self-verifying
+(motivated by bug-937: a deleted external DSO silently fell back to CPU).
+
+### Zero-dependency guarantee (what's compiled in)
+
+- **DSO**: zipalign-embedded; on first boot the APE extracts it to
+  `~/.llamafile/v/<ver>/` and side-loads from there. A fresh `$HOME` is
+  sufficient — proven by the c2 smoke on bs2 (scratch `HOME`, extracted DSO
+  sha == manifest sha `eec7436f…`, Blackwell GPU decode at 272.8 tps, dual-ctx
+  MTP engaged at accept 0.728).
+- **TCQ codebooks**: decode AND encode use compiled-in `__constant__`
+  codebooks in the DSO (`fattn-tcq.cuh` / `turbo-tcq-cuda.cuh`). The
+  `TURBO_TCQ_CB` env is an opt-in override for codebook experiments only; if
+  set but unreadable it warns on stderr and keeps the compiled-in default on
+  BOTH encode and decode (consistent, no KV split). Exotic `turbo*_tcq` KV
+  tiers therefore need no side file. Same story for `TURBO_INNERQ_*` — all
+  opt-in calibration hooks, inert unless set.
+- **All opencoti features** are flag/env-activated on the standalone binary
+  (`--spec-type`, `-ctk/-ctv` turbo tiers, `--dca-*`, `--vram-target`,
+  `--kv-residency-mode`, …); nothing requires the TS/Bun or Go framework.
+
+### Launch caveat (cross-platform)
+
+The APE is cosmocc cross-platform (Linux/Mac/Windows) but GPU support is
+CUDA-only today (Metal is a known gap; `runRocm`/`runVulkan` exist in the
+pipeline but aren't part of the c2 cut). On Linux without binfmt_misc,
+`exec` of the APE fails ENOEXEC (bug-156) — launch as `sh ./opencoti-llamafile-….llamafile`.
+Mac/Windows have not been smoke-tested for c2 (Linux-only exercised so far).
+
+### Deploys
+
+The c2 artifact is deployed next to the split c1 binaries at
+`bs2:/srv/ml/opencoti-c1/opencoti-llamafile-0.10.3-c2-x86_64.llamafile`.
+Eval scripts still drive the split `llamafile.dualctx` + external DSO (fine
+for dev inner loop — a raw source edit only needs make, not packaging); the
+packaged artifact is the distribution/embedding vehicle. For packaged
+deploys the bug-224 DSO-cache-clear ritual is unnecessary: each cut extracts
+its own DSO keyed by version dir, and clearing `~/.llamafile/v` is harmless.
+
+### Secondary native-ELF lane
+
+A plain-gcc/CMake build of the vendored `llama.cpp` tree (standard rpath
+`libggml-cuda.so`, clean gdb/nsys) for Linux/container embedders. Only one
+patch in the chain is cosmo-specific (`0071-state-io-cosmocc`), and the tree
+carries upstream CMakeLists (models/*.cpp is globbed), so the source is
+~toolchain-agnostic. Note the CMake build uses cuBLAS where the cosmocc DSO
+uses TinyBLAS — a different perf profile (bug-2103/2104 class), so gate perf
+claims per-toolchain. **SHIPPED 2026-07-05 as patch `0097-native-elf-cmake`**
+(#619): five CMakeLists surgical hooks (opencoti TUs into the upstream source
+lists + `-Dmain=llama_server` for the overlay server.cpp) plus a
+CMake-lane-only stub TU (`ggml-cpu/opencoti-elf-stubs.cpp`) for the
+cosmocc-bound `llamafile/sgemm.cpp` CPU-fastpath dispatchers — return-false =
+fall back to upstream generic paths (Mozilla's own `fa_helpers_unsupported`
+contract), so the ELF build loses only CPU fastpaths, never correctness.
+
+Build (bs2-proven):
+
+```sh
+cmake -S llama.cpp -B build-elf -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON \
+      -DCMAKE_CUDA_ARCHITECTURES=120 -DCMAKE_BUILD_TYPE=Release \
+      -DLLAMA_CURL=OFF -DGGML_NATIVE=OFF
+cmake --build build-elf -j24 --target llama-server
+```
+
+`-DGGML_CUDA_FA_ALL_QUANTS=ON` is REQUIRED: upstream's CMake only globs
+`fattn-vec*.cu` (all turbo/TCQ FA-VEC instances + codebook setters) under
+that flag; the cosmo build always sets it. Gate (bs2, RTX 6000 Blackwell):
+A4B `-ngl 99` boots, dual-ctx assistant-MTP accept 0.764, **310.2 tps**
+decode (vs 274 on the APE — native-arch cuBLAS build), coherent chat output,
+`ldd` shows rpath `libggml-cuda.so.0`. The cosmocc lane is untouched
+(BUILD.mk lists sources explicitly and never reads CMakeLists).

docs/features/local_llamafile.md

@@ -0,0 +1,323 @@
+# F2 — Local llamafile Model Serving
+
+> Parent: [../MASTER_PLAN.md](../MASTER_PLAN.md)
+> Status: **planning**
+> Owner: TBD
+
+## Problem
+
+The GPU realization of Tier 0 in the tiered inference engine
+([F1](tiered_inference.md)) wants a local backend that works **out of
+the box** — no separate Ollama install required, no separate model
+server. We want a single binary opencoti can launch on demand.
+
+Beyond out-of-box, the GPU path is where opencoti gets to expose
+"advanced features" the no-GPU path (Ollama Cloud) can't: KV-cache
+reuse across agentic turns, big-context throughput, and direct mount
+of an existing Ollama blob without duplicating the download. The
+mainline llamafile / llama.cpp do parts of this — opencoti wants a
+patched build that does all of it.
+
+## Goals
+
+- **G1.** Ship a patched llamafile binary inside opencoti release
+  artifacts.
+- **G2.** Provide a small TypeScript launcher that starts / stops the
+  llamafile, exposes an OpenAI-compatible endpoint to the tier
+  engine, and pins it to loopback by default.
+- **G3.** Maintain patches against upstream llamafile / llama.cpp in
+  `vendors/patches/llamafile/`, applied at build time.
+- **G4.** Patches focus on three areas:
+  - **Big-context throughput** — keep tokens/sec usable at the
+    context sizes agents actually use (64K+, up to the model's 256K cap).
+  - **KV-cache reuse across agentic turns** — when the new turn
+    shares a prefix with the previous turn (system prompt + tool
+    log), skip reprocessing.
+  - **Ollama-blob mount** — load a GGUF directly from
+    `~/.ollama/models/blobs/sha256-…` without copying. Single
+    download, reusable by both Ollama and the patched llamafile.
+
+## Non-goals
+
+- Replacing Ollama for users who prefer it as Tier 0. The no-GPU
+  path of F1's Tier 0 explicitly targets Ollama Cloud as an
+  OpenAI-compatible provider — that doesn't go through this package.
+- Bundling models. Models are downloaded on first use into the
+  opencoti data dir (or imported from an existing Ollama install);
+  we don't ship weights inside the binary.
+
+## Default model
+
+**Primary GPU target:**
+[`ManniX-ITA/gemma-4-A4B-98e-v5-coder-it`](https://huggingface.co/ManniX-ITA/gemma-4-A4B-98e-v5-coder-it-GGUF).
+
+- Architecture: Gemma 4 (MoE), 98-expert prune of Gemma 4 26B-A4B
+  (30 experts dropped per layer, `protect_top=16`, no shared FFN
+  scaling).
+- Total ~20.8 B params, ~4 B active per token.
+- Context: **256 K max** (recipe defaults to a smaller window for
+  speed but the model supports the full 256 K).
+- Pitch: HumanEval 98.17 %, HumanEval+ 92.68 %, LCB-medium 85.45 %.
+  Top of the 14–22 B coder field at this writing.
+
+**Quant table** (the GGUFs we surface as choices during first-run
+setup, see F1 M5.7):
+
+| Quant       | Size     | HE+    | When to pick                          |
+|-------------|----------|--------|----------------------------------------|
+| IQ2_S       | 7.83 GB  | 85.37% | tight VRAM, 8 GB cards                 |
+| IQ3_M       | 9.82 GB  | 91.46% | **default for ≥10 GB VRAM**            |
+| CD-IQ4_K_M  | 10.29 GB | 92.07% | per-layer "Canary W" variant           |
+| IQ4_XS      | 11.01 GB | 93.29% | 12 GB cards                            |
+| Q4_K_S      | 12.21 GB | 93.29% | 16 GB cards, good headroom             |
+| CD-Q4_K_M_L | 13.00 GB | 93.29% | per-layer mid-quant variant            |
+| Q5_K_L      | 15.25 GB | 93.29% | 24 GB cards, room for big context      |
+| Q8_0        | 21.16 GB | 93.90% | 24 GB cards, maximum quality           |
+| F16         | 39.8 GB  | —      | reference only                         |
+
+The CD-* variants are "ContribDynamic" per-layer quants from the
+model card. Both plain and CD- families are presented; the setup
+flow recommends one based on detected VRAM, the user can override.
+
+**Secondary target (deferred):** smaller Gemma 4 variants (E2B, E4B,
+or a non-pruned 26B-A4B). Out of scope until the primary target is
+working end-to-end. Stock Gemma 3 / Gemma 3 1B are not on the
+roadmap.
+
+## Design sketch
+
+### Layout
+
+```
+vendors/
+├── sources/
+│   └── llamafile/            # upstream llamafile tree (vendored, unmodified)
+└── patches/
+    └── llamafile/
+        ├── 0001-kv-reuse.patch
+        ├── 0002-big-context-throughput.patch
+        └── README.md         # what each patch does, against which upstream
+packages/
+└── opencoti-llamafile/       # launcher + adapter for the tier engine
+    ├── src/
+    │   ├── launch.ts
+    │   ├── adapter.ts        # OpenAI-compatible HTTP client wired to tier engine
+    │   └── kv-cache.ts       # session-aware cache key derivation
+    └── package.json
+```
+
+Pattern: same as Ollama's vendoring of `llama.cpp`.
+
+### Build-time patch application
+
+A small build script under `vendors/llamafile/build.{sh,ts}` does:
+
+1. `git -C vendors/sources/llamafile clean -fdx && git checkout <pinned-sha>`
+2. Apply every patch under `vendors/patches/llamafile/*.patch` in
+   lexical order. Bail on conflict.
+3. Build (mode depending on target — release CI builds binaries,
+   local dev can use system llamafile if present).
+4. Emit the resulting binary to a known location the launcher checks.
+
+The pinned upstream SHA is recorded in
+`vendors/sources/llamafile/UPSTREAM.txt`. Updating the SHA is a
+normal PR — patches that no longer apply must be rebased in the same
+PR.
+
+### Launcher
+
+`packages/opencoti-llamafile/src/launch.ts` is a thin process
+manager: starts the binary, waits for the OpenAI-compatible endpoint
+to respond healthy, exposes `port` + `baseURL`. Tears down on
+opencoti shutdown.
+
+### Adapter
+
+`packages/opencoti-llamafile/src/adapter.ts` implements whatever the
+tier engine consumes (provider interface — to be defined in F1 M1).
+It is purely an HTTP client to the local llamafile + cache-key
+plumbing.
+
+### KV-cache reuse — sketch
+
+The agentic loop has a strong invariant: turn N+1 reuses turn N's
+context prefix verbatim, then appends the model output + new tool
+result. A KV-reuse patch:
+
+- Adds a session id to the request schema.
+- On the server side, persists the KV cache state keyed by
+  `(session_id, hash(prefix))`.
+- On the next request with matching `(session_id, hash(prefix))`,
+  skips re-prefilling the prefix tokens.
+
+This is conceptually like prompt caching in cloud providers — applied
+to a local llamafile.
+
+### Ollama-blob mount — sketch
+
+If the user already has Ollama installed and has pulled the target
+model, opencoti should not re-download the GGUF. Ollama stores blobs
+content-addressed at `~/.ollama/models/blobs/sha256-<hash>`, with a
+manifest at
+`~/.ollama/models/manifests/registry.ollama.ai/library/<model>/<tag>`
+mapping the model+tag to a digest list (the GGUF blob is one of the
+layers).
+
+The detection helper (F1 M5.7) parses the manifest and returns the
+blob path. The patched llamafile accepts that path directly via
+`-m`. Single download, no duplication — the same physical file is
+used by both Ollama and llamafile.
+
+The patch in llamafile itself is small: it already accepts `-m
+<path>`, but blob filenames are hash-named (no `.gguf` extension), so
+any code path that gates on file-extension needs adjusting. That's
+the patch.
+
+## Milestones
+
+### M1 — Llamafile vendor present, launcher boots it *(done — 2026-05-21)*
+
+- [x] Vendor llamafile under `vendors/sources/llamafile/` (git
+  submodule pinned to `0.10.1` / `6490e16`).
+- [x] `vendors/patches/llamafile/` exists with `README.md`
+  documenting the patch protocol (numeric prefixes, lexical apply
+  order, upstreaming policy). No patches yet.
+- [x] `packages/opencoti-llamafile/` (`@opencoti/llamafile`) ships
+  `config.ts` (binary resolution: env → vendored build path → PATH),
+  `launch.ts` (`Bun.spawn` + `waitForReady` polling `/v1/models`),
+  `adapter.ts` (M2 stub), and `script/smoke.ts`. Typecheck is clean.
+- Smoke script exits 2 with a clear diagnostic when no binary is
+  resolvable, and 0 when one is. Live `--server` launch + `/v1/models`
+  ping is gated on `OPENCOTI_LLAMAFILE_MODEL` being set.
+
+### M2 — Adapter wired to tier engine *(done — 2026-05-21)*
+
+- [x] `packages/opencoti-llamafile/src/tier-provider.ts` exports
+  `tierProvider()` returning a `TierProvider` (the interface
+  defined by F1 M3 in `@opencoti/tiers/provider`).
+- [x] Implementation uses `@ai-sdk/openai-compatible` to build a
+  `LanguageModel` against the running llamafile's `/v1` endpoint;
+  the previous M1 stub `adapter.ts` is preserved alongside (it's
+  used by the smoke script).
+- [x] Singleton llamafile process per host (`runningPromise` module
+  cache); cleanup is process-exit (good enough for M3, refined
+  later).
+- [x] `@opencoti/llamafile/package.json` declares deps on
+  `@ai-sdk/openai-compatible` and `@opencoti/tiers` (for the
+  interface).
+- [x] F1 M3 (Tier 0 = llamafile end-to-end) is achieved jointly
+  with this milestone.
+
+### M3 — Build pipeline applies patches *(done — 2026-05-22)*
+
+Pure-helpers + CLI orchestrator split (same testability pattern as
+M5.7's setup-flow):
+
+- `packages/opencoti-llamafile/src/build-pipeline.ts` (new) —
+  pure helpers: `discoverPatches`, `validatePatchName`,
+  `summarizePatchSet`, `readPinnedSha`. Re-exported from the
+  package and from a new `./build-pipeline` subpath.
+- `packages/opencoti-llamafile/script/build-pipeline.ts` (new) —
+  CLI orchestrator with subcommands `check`, `reset`, `apply`,
+  `build`, plus `help`. Shells to git + make via `Bun.spawn`.
+  Detects uninitialized submodule and emits a clear "run
+  `git submodule update --init`" message.
+- `vendors/pin/llamafile.txt` (new) — pinned SHA + tag,
+  machine-readable. Lives OUTSIDE the submodule so
+  `git clean -fdx` doesn't wipe it. The user-facing pin table in
+  `vendors/README.md` is kept in sync manually.
+- Root `package.json` scripts: `build:llamafile`,
+  `build:llamafile:check`, `build:llamafile:apply`,
+  `build:llamafile:reset`.
+
+Patch protocol enforced:
+
+- `NNNN-<kebab>.patch` with 4-digit prefix and lowercase kebab title.
+- Duplicate prefixes are flagged (both names appear in the
+  `invalid` list, neither in `patches`).
+- Empty patch set produces "no patches" — the build still runs to
+  completion.
+- `check` warns when the submodule HEAD has drifted from the pin
+  but doesn't fail (a developer may be intentionally bumping).
+
+Test coverage: 21 new unit tests in
+`packages/opencoti-llamafile/test/build-pipeline.test.ts` covering
+all helpers + the fixture-based discover path. Fixtures live under
+`test/fixtures/patches/` (`0001-foo.patch`, `0002-bar-baz.patch`,
+README.md to exercise the non-`.patch`-ignore path). The CLI
+orchestrator itself is NOT unit-tested (it shells out to git/make;
+integration testing belongs to F2 M6).
+
+Verification: `bun run build:llamafile:check` against the current
+empty patch set:
+```
+[build-pipeline] Pinned upstream: 6490e16
+[build-pipeline] Vendor HEAD:     6490e16f7a8fb7d8a0a0734b41490c3f8db78763
+[build-pipeline] no patches
+[build-pipeline] check OK
+```
+
+Zero new surgical hooks. Hook footprint stays at 4 markers.
+
+### M4 — First patch: KV-cache reuse v1 *(absorbed into F5 M0 — shipped 2026-05-24)*
+
+This milestone was absorbed into the F5 advanced-KV series as **F5 M0**
+(homologous to the rest of that patch series) and has **shipped**. It landed
+as `vendors/patches/llamafile/0006-kv-reuse-prefix.patch` (not `0001-`, since
+F4 M3's lazy-slot work consumed 0001–0005): an optional `session_id` request
+field + session→slot affinity, with the adapter threading the session id and
+`cache_prompt` through a `fetch` wrapper. Bench
+`perf/llamafile/turn-2-latency.bench.ts` shows turn-2 prefill collapsing from
+2061→20 tokens. See
+[docs/features/advanced_kv.md](advanced_kv.md) "M0" for the design, the
+upstream-already-reuses finding, and the deferred prefix-hash/disk extension.
+
+### M5 — Big-context throughput patches
+
+- Targeted micro-patches that improve tokens/sec at 64K+ contexts
+  (up to the 256K model cap).
+- Each patch documented in `vendors/patches/llamafile/README.md`.
+- Each patch with a benchmark in `perf/` (we can reuse the existing
+  `perf/` dir from opencode).
+
+### M6 — Release artifact bundling
+
+- CI builds llamafile-with-patches for the platforms opencoti ships
+  on. Binary is included in the opencoti release tarball under a
+  known path the launcher will find.
+
+### M7 — Ollama-blob mount patch
+
+- Patch lands in `vendors/patches/llamafile/0002-ollama-blob-mount.patch`.
+- The patched llamafile accepts hash-named (extension-less) GGUF
+  paths from `~/.ollama/models/blobs/sha256-…` directly.
+- Wired into F1 M5.7's first-run setup flow as the "Import from
+  local Ollama" option.
+- Smoke test: install Ollama, `ollama pull gemma3:1b` (or any test
+  model), point the patched llamafile at the blob, assert health.
+
+## Open questions
+
+- **Llamafile vs. raw llama.cpp.** Llamafile gives us the
+  single-binary-cross-platform property. If KV-reuse patching is
+  easier against llama.cpp upstream, we may switch vendors. Still
+  open after M1 — re-evaluate when the first patch (M4) is in flight.
+- ~~**Default model.** Qwen2.5-Coder-7B vs Qwen2.5-Coder-14B.~~
+  Resolved: `ManniX-ITA/gemma-4-A4B-98e-v5-coder-it` (see *Default
+  model* section above). Smaller Gemma 4 variants are explicitly
+  deferred secondary targets.
+- **Sandboxing.** Launching a long-running binary needs robust
+  shutdown. Use a process group + signal forwarding so an opencoti
+  crash doesn't leave a llamafile orphan.
+- **`head_dim=512` constraint.** Gemma 4 has `head_dim=512`, which
+  means FlashAttention 2 is NOT supported (per the model card).
+  Patches need to leave the `attn_implementation="eager"` path
+  intact and not enable FA2 indiscriminately for big-context work.
+
+## Risks
+
+- Patches drift when upstream moves. Mitigation: pinned SHA + the
+  rebase-on-update rule, and per-patch tests.
+- Binary size. Llamafile binaries are large. Mitigation: ship the
+  binary as a separate artifact downloaded on first run, not inside
+  the main opencoti package, if it gets prohibitive.

docs/features/memory_embedder.md

@@ -0,0 +1,668 @@
+# F4 — Memory / Embedder
+
+> Parent: [../MASTER_PLAN.md](../MASTER_PLAN.md)
+> Status: **planning** (M0+M1+M2 done — 2026-05-22)
+> Owner: TBD
+> Reference: `/shared/dev/claude-hooks` (the embedder lifecycle is
+> mirrored from claude-hooks' Python implementation)
+
+## Problem
+
+opencoti needs persistent memory on day-one — well before F3
+(`opencoti-server`, the Go companion daemon) is ready. Memory
+needs an embedder, and the embedder needs to work without
+requiring the user to install a separate runtime. The same
+constraint that drove F2 (bundle llamafile so opencoti runs out
+of the box) drives F4: ship a prebuilt embedder llamafile, manage
+it with a small TypeScript launcher, default to CPU, allow
+opt-in GPU.
+
+There's a second, sharper problem the user surfaced from
+real-world claude-hooks operation: the canonical embedder
+(`qwen3-embedding-0.6b-16k.llamafile`, ~1.5 GB on disk for a
+700 MB model) allocates **3.32 GB RSS** at idle. The bottleneck
+is *per-slot context buffer* allocation, not the autoregressive
+KV cache. The embedder server runs `--parallel 2 --ctx-size
+16384` and pre-allocates the full 16K context for both slots up
+front. Live workload shows most requests at 7–56 tokens with
+only 1 slot active most of the time — so the eager allocation
+pays full price for capacity that is almost never used.
+
+F4 ships the embedder, *and* ships a lazy-slot-context patch
+against the vendored llamafile/llama.cpp tree so the embedder's
+RSS scales with actual demand instead of `slot_count × n_ctx_slot`.
+
+## Goals
+
+- **G1.** TypeScript package `@opencoti/embedder` that owns the
+  embedder lifecycle: spawn, health-probe, idle-reap, GPU/CPU
+  mode, `embed(text)` / `embedBatch(texts)`.
+- **G2.** First-run setup downloads the prebuilt asset
+  (`qwen3-embedding-0.6b-16k.llamafile`, dim 1024, default port
+  **47092** in opencoti's reserved 47000-48000 range) into the
+  opencoti data dir. Resume + SHA256 verify.
+- **G2a.** **Daemon-launch policy** (project-wide convention).
+  Any opencoti or opencoti-server daemon that opens a listening
+  port MUST: (1) pick its default port from the **47000-48000
+  range** — opencoti must never collide with claude-hooks'
+  38000-39000 / 18790-18811 ranges so the two systems coexist
+  on one host; (2) ask the user at first-run setup where to
+  bind — `127.0.0.1` (default), all interfaces, or a specific
+  IP — and (3) propose the default port but allow the user to
+  override it before persisting the choice. The setup flow
+  validates that the chosen port is free and within range
+  before writing the config.
+- **G3.** GPU mode = auto-detect with CPU fallback; default = CPU.
+  Reuses F1 M5.7's `gpu-detect` chain (nvidia → amd → vulkan).
+- **G4.** Composite fallback chain: an `EmbedderClient`
+  interface + a `CompositeEmbedder` that tries primary then
+  fallback on `EmbedderError`, with a dim-match assertion to
+  prevent vector-space corruption.
+- **G5.** Lazy-slot-context patch (`0001-lazy-slot-context.patch`)
+  against the vendored llamafile so RSS scales with real demand:
+  defer per-slot allocation, grow on demand, **shrink on idle**.
+
+## Non-goals
+
+- Replacing claude-hooks for users who already run it on the
+  same host. opencoti's embedder is independent and uses its own
+  port + data dir.
+- Bundling weights. The model GGUF is inside the .llamafile
+  asset; we don't ship a separate GGUF download.
+- A full vector store + recall pipeline. F4 M5 wires recall +
+  store as a thin seam; the real vector-store ownership lives
+  in F3 (opencoti-server) when it ships.
+
+## Default asset
+
+- **Repo:** [`mann1x/claude-hooks`](https://github.com/mann1x/claude-hooks)
+  releases — **transitional pin, two-stage migration**:
+  - **Stage 1 (immediate)**: mirror the byte-identical asset to
+    a `mann1x/opencoti` release (proposed tag:
+    `embedder-v1.4.0`) and flip just the `repo` field in
+    `vendors/pin/embedder.txt`. Same SHA, no rebuild —
+    decouples opencoti's release stream from claude-hooks'.
+  - **Stage 2 (after F4 M3 lazy-context patch ships)**: rebuild
+    the embedder composite-llamafile to include the lazy-
+    slot-context patch. The new binary has a different SHA;
+    `tag` + `sha256` in the pin file both bump (proposed tag:
+    `embedder-v1.5.0`). F4 M6 (release bundling) owns the
+    apply-patches → make → zipalign-with-GGUF-and-args
+    pipeline. Without Stage 2, the embedder still benefits
+    from the lazy-context patch only when run against a
+    user-supplied locally-built embedder binary — not the
+    prebuilt asset.
+- **Tag:** `v1.4.0`
+- **Asset:** `qwen3-embedding-0.6b-16k.llamafile`
+- **SHA256:** `414f616689aaba44f6982b474918e8549ad764c03b2f82f6b56a5d6e582ef59b`
+- **Pinned in:** [`vendors/pin/embedder.txt`](../../vendors/pin/embedder.txt)
+- **Default port:** **47092** (opencoti's reserved 47000-48000
+  range — see G2a).
+- **Endpoint:** `POST /embedding` (llama.cpp server style, not
+  OpenAI-compat). Request: `{"content": "<text>"}` or batch
+  `{"content": ["t1", "t2"]}`. Response: `{"embedding": [...]}`
+  (single) / `[{"embedding": [...]}, ...]` (batch).
+
+## Design sketch
+
+### Layout
+
+```
+packages/opencoti-embedder/           # NEW
+├── package.json
+├── tsconfig.json
+├── src/
+│   ├── index.ts
+│   ├── client.ts                     # EmbedderClient interface + types
+│   ├── llamafile-embedder.ts         # POST /embedding HTTP client
+│   ├── composite.ts                  # primary → fallback chain
+│   ├── manager.ts                    # process spawn / health / idle reap
+│   ├── gpu-mode.ts                   # CPU | auto resolution
+│   ├── download.ts                   # asset fetch + SHA256 verify
+│   └── config.ts                     # Zod schema for embedder config
+├── script/
+│   └── smoke.ts
+└── test/
+
+vendors/
+├── pin/
+│   └── embedder.txt                  # NEW (M0)
+└── patches/
+    └── llamafile/
+        └── 0001-lazy-slot-context.patch   # NEW (M3)
+```
+
+### Reused, not rebuilt
+
+- `@opencoti/tiers/registry/gpu-detect` — nvidia/amd/vulkan probe
+  chain (F1 M5.7).
+- `@opencoti/tiers/registry/hf-download` — resume + SHA256 verify
+  download. To be generalized into a shared
+  `@opencoti/tiers/util/download` helper consumed by both HF
+  downloads and GitHub Release downloads.
+
+## Milestones
+
+### M0 — Vendoring decision + asset URL pin *(done — 2026-05-22)*
+
+- `vendors/pin/embedder.txt` records repo / tag / asset / SHA256.
+- Decision recorded: pin claude-hooks v1.4.0 directly until
+  opencoti has its own release stream. Switch the pin URL to
+  `mann1x/opencoti` when that exists.
+- This feature plan + F5 plan + MASTER_PLAN.md updated together
+  in the same commit.
+
+### M1 — Embedder package, CPU mode, smoke green *(done — 2026-05-22)*
+
+- `@opencoti/embedder` shipped with client interface,
+  llamafile-embedder HTTP impl (POST /embedding, response-shape
+  tolerant for A/B/C/D variants seen in the wild),
+  download.ts (resume + SHA verify, builds the canonical
+  github-release URL from the pin), manager.ts (spawn +
+  `/health` poll + LRU idle reap @ 5 min default + idempotent
+  stop), config.ts (Zod strict schema, defaults claude-hooks-
+  parity), gpu-mode.ts (CPU default + auto-resolution),
+  composite.ts (primary→fallback chain with dim-match guard).
+- Shared `downloadWithResume` helper extracted to
+  `@opencoti/tiers/util/download`; `hf-download.ts` refactored
+  to consume it. New subpath exports:
+  `@opencoti/tiers/util/download`,
+  `@opencoti/tiers/registry/gpu-detect`,
+  `@opencoti/tiers/registry/hf-download`.
+- `script/smoke.ts` downloads the asset to `<repo>/.opencoti/embedder/`
+  (skips on SHA match), spawns CPU-mode, embeds "hello world",
+  asserts dim=1024 + L2-norm. Live smoke run is opt-in (asset
+  is 1.5 GB).
+- 52 unit tests across 7 files. Hook footprint unchanged
+  (still 4 `opencoti-hook:` markers).
+- **Out of scope (deferred to M5)**: opencode session
+  injection, recall-into-prompt, vector store.
+
+### M2 — Baseline measurement *(done — 2026-05-22)*
+
+Ships two scripts and three fixture files:
+
+- `script/generate-corpus.ts` — deterministic 48-prompt corpus
+  generator spanning 8-3000 approx_tokens. Buckets: 10 short
+  (8-32 tok), 11 medium (33-128), 19 medium-long (129-512), 6
+  long (513-2048), 2 very-long (2049-8192). Original draft had
+  5 in the very-long bucket; 2 mega prompts (60+ LOREM repeats)
+  aborted past the 60s embedder request-timeout during the live
+  baseline run, so they were dropped from the generator.
+- `script/measure-baseline.ts` — boots the embedder via the M1
+  manager (or attaches to a running instance with `--use-running
+  <url> --pid <int>`), captures RSS from `/proc/<pid>/status`,
+  extracts allocation lines from the embedder startup log,
+  embeds the corpus, writes the fixture.
+
+Fixtures (in `test/fixtures/`):
+- `quality-prompts.json` — 48 prompts, deterministic.
+- `quality-baseline.json` — 48 vectors (dim 1024) M3 must
+  reproduce to cosine ≥ 0.999.
+- `baseline-meta.json` — RSS + allocation breakdown +
+  startup-log excerpt.
+
+**Live measurement (2026-05-22, attached to claude-hooks
+embedder, parallel=4 default, ctx=16384, CPU mode)**:
+
+| Component | Size |
+| --- | --- |
+| Model weights (CPU_Mapped) | 603.87 MiB |
+| KV cache (all slots) | **1792.00 MiB** |
+| Compute buffer | 330.24 MiB |
+| Output buffer | 2.33 MiB |
+| RSS idle (process VmRSS) | **2.52 GB** |
+| RSS peak (after 48-prompt run) | **3.81 GB** |
+| Avg latency per prompt | 5352 ms |
+| Total corpus embed time | 256.9 s |
+
+Startup-log key lines (captured in baseline-meta.json):
+`llama_context: n_ctx = 16384`,
+`llama_kv_cache: CPU KV buffer size = 1792.00 MiB`,
+`sched_reserve: CPU compute buffer size = 330.24 MiB`,
+`server_main: embeddings enabled with n_batch (2048) >
+n_ubatch (512) → setting n_batch = n_ubatch = 512 to avoid
+assertion failure`, `slot load_model: id 0/1/2/3 | new slot,
+n_ctx = 16384`. The binary defaults to parallel=4 with no
+`--parallel` override.
+
+The earlier user observation of **3.32 GB RSS** came from a
+2-slot config explicitly setting `--parallel 2 --ctx-size
+16384`. The M3 patch targets that production-shape config:
+
+- Pre-patch per-slot KV allocation = `n_ctx_slot × 2 (K+V) ×
+  layers × per-token-dim` ≈ 448 MiB / slot in CPU mode,
+  allocated eagerly for every slot at startup.
+- M3 target: per-slot allocation deferred to first prompt,
+  sized to `--slot-initial-ctx 4096` (~112 MiB), grown on
+  demand up to 16384, shrunk back to 4096 after
+  `--slot-shrink-idle-ms`. Combined with 2 slots + model +
+  compute, idle RSS target lands near 1.0 GB.
+
+**Two corpus prompts intentionally dropped**: `lorem-mega-00`
+and `lorem-mega-01` (60+ LOREM repeats) aborted past the
+embedder's 60s default request-timeout on the CPU-mode
+production instance — n_ubatch is forced to 512 in embedding
+mode, so 9000+ token requests need ~18 forward passes that
+exceed 60s. Not an M3 concern (the patch addresses
+*allocation*, not *throughput*).
+
+### M3 — Lazy-slot-context patch
+
+> **Architecture reference:** [ADR 0001 — Lazy slot-context
+> allocation](../decisions/0001-lazy-slot-context.md). The ADR
+> documents the llama.cpp KV-cache layout, the three+1 orthogonal
+> axes of "lazy," the four-phase rollout (defer / grow / shrink /
+> per-stream split), the CLI surface, and the F5 milestone
+> hand-offs that depend on the allocator hooks landed here.
+> Implementation is multi-session per user directive B2.
+>
+> **Phase status (2026-05-23):**
+> - **Phase 1 — deferred zero-fill: shipped** as
+>   `vendors/patches/llamafile/0001-lazy-slot-context-defer.patch`.
+>   Patch applies cleanly; build succeeds on x86_64 + aarch64;
+>   smoke validation on Qwen2-Math-1.5B confirms the
+>   `KV cache zero-fill deferred until first batch` startup log
+>   line and the matching `ensure_cleared: ... (deferred)`
+>   first-batch log line. Bench against M2 corpus + RSS table
+>   for the embedder workload pending (Phase 1 alone is a
+>   cold-start win; the headline warm-state RSS savings come
+>   in Phases 2-3).
+> - **Phase 2 — grow on demand + partial zero-fill: shipped**
+>   as `vendors/patches/llamafile/0002-lazy-slot-context-grow.patch`.
+>   Adds `--slot-initial-ctx N` (default 0 = Phase 1 behavior) that
+>   caps initial KV-cell allocation per sequence; find_slot honors
+>   the soft cap; prepare() grows on overflow up to kv_size_max via
+>   next-pow2 geometric ramp; ensure_cleared partial-memsets only
+>   the [n_cells_cleared, target) range via `ggml_backend_tensor_memset`.
+>   With `--slot-initial-ctx 4096` for the production embedder,
+>   warm-state RSS scales with actual workload instead of the
+>   `slot_count × n_ctx_slot` product. Bench numbers vs M2 corpus
+>   pending.
+> - **Phase 3 — shrink on idle: shipped** as
+>   `vendors/patches/llamafile/0005-lazy-slot-context-shrink.patch`.
+>   Adds `--slot-shrink-idle-ms N` (default 30000); per-stream
+>   idle timer sweeps at `init_batch` and reclaims buffers back
+>   to `--slot-initial-ctx` after the timeout. Bench delta:
+>   **391.7 MB returned to kernel** on the CPU embedder after 32 s
+>   idle (1.68 GB peak → 1.30 GB post-shrink, cosine 0.999797 vs
+>   M2 baseline).
+>   **CPU-only — the shrink is gated on
+>   `ggml_backend_buffer_is_host()` inside
+>   `opencoti_decommit_layer_range()` because
+>   `madvise(MADV_DONTNEED)` is a no-op against device-backed
+>   pages.** On GPU (`-ngl > 0`) the `shrink_if_idle` log marker
+>   still fires and the soft-cap bookkeeping rewinds, but the
+>   CUDA/ROCm/Vulkan backend buffer is not freed and VRAM stays
+>   at the peak the workload reached. The GPU-shrink path is
+>   tracked at **F5 M2 (HeadInfer)**, not Phase 4.
+> - **Phase 4 — per-stream tensor split: DEFERRED** (task #109,
+>   target F5 M2). 2026-05-23 user decision after planning-session
+>   exploration showed Phase 4 is a deep refactor (per-stream
+>   tensors + per-stream backend buffers + reworked
+>   `cpy_k`/`cpy_v` scatter ops that currently rely on
+>   `ggml_reshape_2d(k, n_embd, kv_size*n_stream)` to flatten all
+>   streams into a single `ggml_set_rows`) and only adds value in
+>   non-unified mode. The embedder ships unified by default and
+>   already gets all Phase 1-3 wins with the contiguous tensor
+>   intact. Phase 4's real beneficiary is F5 (HeadInfer's per-head
+>   GPU/CPU split + PolyKV's shared pool). Full implementation
+>   map persisted in
+>   [docs/decisions/0001-lazy-slot-context.md §Phase 4 — implementation map](../decisions/0001-lazy-slot-context.md)
+>   for the F5 owner to pick up cold. **Not the GPU-shrink fix —
+>   that's F5 M2 (HeadInfer), independently of Phase 4.**
+
+- `vendors/patches/llamafile/0001-lazy-slot-context.patch`.
+- Behavior:
+  - Defer per-slot KV/compute-buffer allocation until slot's
+    first real prompt.
+  - Initial allocation rounded up to `--slot-initial-ctx`
+    (default 4096).
+  - Grow on demand (next power-of-two), cap at `n_ctx_slot`.
+  - **Shrink on idle** (`--slot-shrink-idle-ms`, default
+    30000): free buffers and return to initial-ctx after the
+    timeout, otherwise the first 16K request permanently sticks
+    the slot at 16K and the lazy-allocation win evaporates.
+- Bench (`perf/llamafile/embedder-rss.bench.ts`): RSS at idle /
+  after 8-token request / after 16K request / after idle
+  timeout + 8-token request. Quality: cosine ≥ 0.999 vs M2's
+  baseline.
+- Header: `Milestone: F4 M3`, `Upstreaming: TBD — propose to
+  llamafile + llama.cpp upstream once bench numbers are public`.
+
+### M4 — Auto-GPU mode + composite fallback
+
+- `gpu-mode.ts` adds `"auto"`: probe nvidia → amd → vulkan; if
+  VRAM ≥ 1.6 GB, launch with `--gpu auto` + n-gpu-layers fully
+  offloaded; otherwise silent CPU fallback.
+- `CompositeEmbedder` ships: primary (local llamafile) +
+  fallback (cloud-route delegate, when Tier 1 is configured).
+  Dim-mismatch assertion lives here.
+
+### M5 — Opencode integration: recall + storage seam *(2026-05-23 — shipped, rescoped mid-milestone)*
+
+**Original scope:** one synthetic tools pair + one surgical session-end hook +
+setup-flow listen-address question. **Actual scope after the user rescope
+(verbatim): "memory store and recall is going to be another crucial point,
+we support sqlite + sqlite_vec as base memory backend, managed by
+opencoti-server. opencoti-server will manage optional sharing on the
+network. the pgvector db is an add on memory connector. both works in
+parallel. memory in opencoti can be managed per-session. user can manage
+which memories to use and list, delete, connect to the memories via the
+harness menu. shared memory for all sessions, specific memories can be
+added and or connected. for each session the user can decide which one
+to read/write/rw"**
+
+Path taken (after user decision): **in-process M5 now, refactor to
+opencoti-server later**. The opencoti-server piece is queued for F3
+once F4 closes. The data model and ACL contract land here so they
+survive that refactor unchanged.
+
+What M5 actually shipped:
+
+- **M5-A — `@opencoti/memory` package.** `SqliteVecStore` over
+  bun:sqlite + sqlite-vec 0.1.9 vec0 virtual table. Embedding dim
+  locked at DB creation (default 1024). Schema: `collections`,
+  `memories` (with `content_hash` UNIQUE for idempotent insert),
+  `memory_acl` (per-session override), `vec_memories` (vec0
+  virtual table with FLOAT[dim]). Default DB path
+  `~/.opencoti/memory/state.db`. ACL model: explicit overrides win;
+  defaults are "rw" for global collections, "rw" for the owning
+  session of a session-private collection, "none" otherwise. The
+  resolver lives in pure code (`resolveAccessMode`, `canRead`,
+  `canWrite`) and is exhaustively tested.
+- **M5-B — synthetic tools.** `__memory_recall(query, k?, collections?)`,
+  `__memory_store(content, collection?)`, `__memory_list()`. All
+  three are ACL-aware (sessionID is the actor) and best-effort —
+  embedder failure returns `{ok: false, error: "embedder_unavailable"}`
+  rather than throwing. `ensureSessionCollection` is idempotent.
+- **M5-C — runtime wire-up via dynamic-import bridge.**
+  `@opencoti/tiers/src/memory-bridge.ts` resolves
+  `@opencoti/memory` + `@opencoti/embedder` via dynamic import to
+  break the tiers→memory→embedder→tiers cycle. Wired into
+  `runtime.ts:maybeRoute` via `mergeTools` (no new surgical
+  hook — extends an existing additive seam). The default
+  embedder is `createDefaultEmbedFn` (loopback LlamafileEmbedder
+  on 47092). Workspace dep `@opencoti/memory` added on
+  `packages/opencode` to make the dynamic import resolvable;
+  registered in UPSTREAM_SYNC.md as `memory-tools-resolvability`
+  (dep).
+- **M5-D2 — `@opencoti/memory-plugin` (opencode SERVER plugin).**
+  Two hooks: `experimental.chat.system.transform` appends a
+  system-prompt suffix advertising the memory tool surface +
+  listing the session's accessible collections;
+  `event` listens for `session.idle` and, when
+  `auto_ingest: true` (default FALSE), ensures the session-private
+  collection. Defaults reflect the rescoped vision: advertise=true
+  (the point of loading the plugin), auto_ingest=false (the agent
+  already has `__memory_store`).
+- **M5-D1 — `@opencoti/tui-memory` (opencode TUI plugin).** One
+  command `opencoti.memory` opens a DialogSelect listing every
+  collection (global + session-private), with per-session
+  resolved-mode annotations when in a session route. Drill-down
+  per collection offers: ACL toggle r/w/rw/none (session-only),
+  DialogConfirm-guarded delete, Create flow
+  (DialogPrompt name → scope DialogSelect). Lazy store open,
+  closed cleanly on `api.lifecycle.onDispose`.
+
+**Default-plugin auto-wiring (2026-05-23 follow-up to user
+directive: "opencoti will need to wire them by default"):**
+The opencoti plugins remain opt-in for upstream-opencode users
+(they stay out of upstream's plugin list unless explicitly
+declared), but opencoti's distribution auto-wires them whenever a
+project has any `opencoti.*` config section. Implementation in
+`@opencoti/tiers/default-plugins`: a pure helper
+`applyDefaultPlugins(cfg)` gap-fills `@opencoti/memory-plugin`,
+`@opencoti/tui-memory`, `@opencoti/tui-tiers` into the final
+`plugin_origins` list — user-declared entries (including their
+options) are preserved untouched. Wired via two surgical hooks
+in `packages/opencode/src/config/config.ts` (import + call-site,
+registered as `opencoti-default-plugins`). Net change vs M5
+substance: TWO new opencoti-hook markers (still tagged + in
+registry). With opencoti disabled (no `cfg.opencoti` block) the
+helper is a no-op and the byte-output is identical to upstream.
+
+What M5 deliberately did NOT ship:
+
+- The session-end **surgical** hook (the original plan). Replaced by
+  the optional `event: "session.idle"` listener in `@opencoti/memory-plugin`
+  with `auto_ingest: false` default. Net new `opencoti-hook:`
+  markers from M5 substance + follow-up: TWO (the
+  `opencoti-default-plugins` import + call-site). The
+  `memory-tools-resolvability` dep is a registry-only entry (no
+  source marker — JSON has no comments).
+- The pgvector add-on connector. Deferred — the in-process backend
+  is sqlite-vec only for now. Pgvector becomes a parallel
+  `MemoryStore` implementation once opencoti-server (F3) lands.
+- The network-sharing layer. That's opencoti-server's job (F3).
+- The "Clear ACL override" UI option in the TUI panel. The
+  `MemoryStore` interface does NOT currently expose a clear-ACL
+  method — adding it would require a new schema/interface revision
+  (deferred to a future minor bump).
+- **M5-E — setup-flow embedder daemon listen address question
+  (shipped 2026-05-23).** `@opencoti/tiers/cli/setup-flow`
+  extended with an embedder listen-address prompt in
+  `finalizePatch`. Steps: (1) DialogSelect over
+  loopback / all-interfaces / specific-IP / skip; (2) if
+  specific-IP, prompt for the IP with an IPv4 validator;
+  (3) text prompt for the port, validated against the opencoti
+  47000-48000 range; (4) port-free check via injected
+  `isPortAvailable(host, port)` (live: TCP bind probe in
+  setup.ts) — busy port logs a warn and asks the user whether to
+  proceed anyway; "no" recursively re-asks the bind block. The
+  chosen `{host, port}` lands in
+  `~/.config/opencode/opencode.jsonc` under
+  `opencoti.embedder.host` + `.port` via the jsonc-parser
+  modify-then-applyEdits path (comments preserved). A new
+  `embedderConfigured` sentinel (true when BOTH host AND port
+  already present in the user's config) gates the prompt on
+  re-runs. 16 new tests across the flow + the `validatePort` and
+  `isValidIPv4` pure helpers.
+
+### M6 — Production llamafile build pipeline *(2026-05-23 — promoted ahead of M5)*
+
+Driver: user directive 2026-05-23 — the post-F4 M4 state (prebuilt
+claude-hooks asset + manual zipalign of ggml-cuda.so + cross-repo
+pin dependency) is "patchwork […] not acceptable for production".
+M6 was originally scoped to the embedder composite alone; the
+expanded M6 below covers all three backends (CUDA / ROCm / Vulkan)
+and the local-asset modes the embedder downloader needs to
+consume an opencoti-built artifact.
+
+M6 lands before M5 (opencode session-end hook) because there's no
+point storing memories on the back of a launcher binary that
+doesn't engage the M3 lazy-context patch.
+
+**What M6 ships:**
+
+1. **Multi-backend DSO builders in
+   `packages/opencoti-llamafile/script/build-pipeline.ts`:**
+   `runCuda` / `runRocm` / `runVulkan` / `runAllBackends`,
+   driven by a shared `buildBackend(spec)` helper.
+   - Each backend has a toolchain probe: `OPENCOTI_CUDA_PATH/bin/nvcc`
+     (default `/usr/local/cuda-12.6` — CUDA 13.x is known-broken
+     with `cuda.sh`), `OPENCOTI_ROCM_PATH/bin/hipcc` (default
+     `/opt/rocm`), `OPENCOTI_VULKAN_SDK` env / system `glslc`.
+   - Soft-fail per backend: a host without ROCm produces a clean
+     CUDA-only build instead of failing the pipeline.
+   - Each built DSO is staged at
+     `vendors/dist/llamafile/<backend>/<ver>/ggml-<backend>.so`
+     AND mirrored to `~/.llamafile/v/<ver>/` so devs running the
+     thin binary directly still get GPU support locally.
+
+2. **`package` subcommand in the same script:**
+   Zipaligns the thin patched binary with every staged DSO via
+   the vendored `vendors/sources/llamafile/o/third_party/zipalign/
+   zipalign -j0` invocation. Always emits
+   `dist/llamafile/opencoti-llamafile-<ver>-<arch>.llamafile`
+   plus a sibling `MANIFEST.json` (artifact SHA + per-DSO SHA +
+   patches list).
+   - With `--with-model PATH` ALSO emits
+     `dist/llamafile/opencoti-embedder-<ver>-<arch>.llamafile`
+     by copying the bare and zipalign-embedding the GGUF + a
+     `.args` file generated from the F2 M3 canonical embedder
+     args contract (`-m /zip/<basename> --server --embedding
+     --pooling last --ctx-size 16384 --parallel 2
+     --slot-initial-ctx 4096 --slot-shrink-idle-ms 30000`).
+   - `--with-args PATH` overrides the generated `.args`.
+   - `--cpu-only` produces a binary with NO DSO embedded (for CI
+     smoke on hosts without GPU).
+
+3. **Pure helpers in
+   `packages/opencoti-llamafile/src/build-pipeline.ts`** (kept
+   side-effect-free per F2 M3 convention): `Backend` type +
+   `ALL_BACKENDS`, `dsoFilename`, `stagingDir`, `artifactDir`,
+   `readBackendArtifacts`, `sha256OfFile`, `composeEmbedderArgs`,
+   `parseBuildArgv`, `ArtifactManifest` type. 19 new unit tests
+   under `test/build-pipeline.test.ts`.
+
+4. **`@opencoti/embedder` downloader local-asset modes
+   (`src/download.ts`):**
+   - `file:` prefix on the pin's repo field — bytes come from
+     disk; SHA still verified.
+   - `OPENCOTI_EMBEDDER_LOCAL=<abs-path>` env override —
+     bypasses pin path entirely; SHA still verified.
+   - Both routed through a shared `localCopyPath` helper; the
+     HTTPS+resume path is unchanged. 12 new tests in
+     `test/download.test.ts`.
+
+5. **Hook footprint unchanged.** M6 is pure build orchestration +
+   downloader extension; no surgical hooks in upstream opencode.
+
+**Deferred to F4 M7 (release bundling):**
+
+- GitHub Releases publish pipeline. Producing artifacts under
+  `dist/llamafile/` is M6; uploading them as `mann1x/opencoti`
+  release assets and flipping `vendors/pin/embedder.txt` to
+  `mann1x/opencoti` (Stage 2 of the migration the existing pin
+  file header describes) is M7. The pin migration is a one-line
+  change once the release exists.
+- Metal / macOS backend. `vendors/sources/llamafile/llamafile/
+  metal.c` is macOS-only; M6 ships Linux-only since the build
+  host is solidPC. Needs a Darwin CI runner.
+- aarch64 / multi-arch DSO orchestration. cosmocc APE is dual-
+  arch by design, but ggml-*.so are arch-specific. The `<arch>`
+  token in the output filename is anticipatory; M6 ships x86_64
+  only.
+
+### M7 — Release bundling + dev-cut workflow alignment with claude-hooks
+
+The build pipeline (M6) produces `dist/llamafile/*.llamafile` artifacts.
+M7 wraps that with the **commit-the-contract, gitignore-the-bytes**
+release scheme claude-hooks uses, plus a `release-cut` subcommand
+that prepares (but does NOT auto-publish) a `mann1x/opencoti`
+GitHub Release.
+
+**What's tracked (committed):**
+
+- `vendors/llamafile/README.md` — audit-trail doc describing the
+  pinned upstream + the contract layout. Mirrors claude-hooks's
+  `vendor/llamafile/README.md`.
+- `vendors/llamafile/LICENSE.upstream` — single-file copy of the
+  four upstream LICENSE files (Mozilla-Ocho/llamafile Apache-2.0 +
+  llama.cpp/whisper.cpp/stable-diffusion.cpp MIT × 3) at the
+  pinned commit. Survives independently of the submodule.
+- `vendors/llamafile/SHA256SUMS.composite` — the contract.
+  `sha256sum`-compatible lines listing every released composite
+  artifact. Refreshed by `build:llamafile:release-cut`. Verified
+  by the embedder downloader at install time.
+
+**What's gitignored (reproducible from source):**
+
+- `vendors/dist/llamafile/<backend>/<ver>/ggml-*.so` — per-backend
+  staging DSOs.
+- `dist/llamafile/*.llamafile` — final composite artifacts.
+- `dist/llamafile/*.MANIFEST.json` — per-artifact manifests with
+  embedded-backend metadata.
+
+**Release-cut workflow** (matches claude-hooks's
+`make -C vendor/llamafile/dist`):
+
+1. `bun run build:llamafile` — patched binary.
+2. `bun run build:llamafile:all-backends` — every available DSO.
+3. `bun run build:llamafile:package -- --with-model PATH` — bare
+   + composite artifacts.
+4. `bun run build:llamafile:release-cut` — refreshes the SHA
+   contract; prints the suggested `gh release create` invocation
+   (the actual publish is a user-confirmation step, not
+   automated).
+5. User executes `gh release create` + `gh release upload`, then
+   commits the refreshed `SHA256SUMS.composite` + the bumped
+   `vendors/pin/embedder.txt`.
+
+**Downloader behavior:** when `shaContractText` is provided to
+`downloadEmbedder`, the pin's SHA is cross-checked against the
+contract before any download or local-copy. Disagreement is a
+hard error with a "redownload or rebuild" breadcrumb (matches
+claude-hooks's `install.py`). An asset not present in the
+contract is a soft warning, not an error — supports the
+transitional state where the pin still references claude-hooks
+pre-flip.
+
+**What's NOT in M7:**
+
+- Actual first `mann1x/opencoti` GitHub Release publish — that's
+  the user-confirmed external step the scaffolding enables.
+- Pin flip from `mann1x/claude-hooks` → `mann1x/opencoti` — only
+  meaningful after the first release exists.
+- macOS / aarch64 — same constraints as M6.
+
+**Critical files (M6):**
+
+- `packages/opencoti-llamafile/script/build-pipeline.ts` — new
+  subcommands.
+- `packages/opencoti-llamafile/src/build-pipeline.ts` — pure
+  helpers + types.
+- `packages/opencoti-llamafile/test/build-pipeline.test.ts` —
+  helper tests (19 new, 46 total).
+- `packages/opencoti-embedder/src/download.ts` — local-asset
+  modes.
+- `packages/opencoti-embedder/test/download.test.ts` — local-
+  asset tests (12 new, 13 total for download).
+- `docs/features/llamafile_build.md` — toolchain + pipeline
+  documentation.
+- Root `package.json` — new `build:llamafile:{rocm,vulkan,
+  all-backends,package}` scripts.
+- `.gitignore` — `vendors/dist/`, `/dist/llamafile/`.
+
+## Surgical-hook footprint
+
+- M1–M4, M6: zero new hooks.
+- M5: exactly one new hook (session-end memory ingest).
+
+## Open questions
+
+- **Release stream ownership.** Decision (M0): pin claude-hooks
+  v1.4.0. When opencoti's own release cadence starts, switch
+  the pin URL to `mann1x/opencoti`. Same SHA + same asset on
+  disk; just a different `repo` field.
+- **Composite-llamafile build.** F4 consumes the prebuilt
+  asset. If F4 M3's lazy-context patch lands and we want it in
+  the embedder binary too, we need to build our own
+  composite-llamafile (vendored llamafile + Qwen3 GGUF +
+  `.args`). That's potentially a small new milestone (F4 M6)
+  that overlaps with F2 M6 (release artifact bundling).
+- **Idle-reap vs always-on for the manager.** 5-min idle reap
+  matches claude-hooks. For opencoti, sessions are typically
+  longer; revisit if cold-start latency turns out to be an
+  issue in production use.
+- **Sqlite-vec dep choice.** The Bun-native binding vs
+  `better-sqlite3` + the `sqlite-vec` extension. Decide at M5
+  based on which builds cleanly across Linux/macOS/Windows.
+
+## Risks
+
+- **F4 M3 patch surface area.** Slot-init touches code that F5's
+  ReST-KV (M1) and HeadInfer (M2) also modify. Ordering: F4 M3
+  lands first as patch `0001-`; F5's start at `0010-`. On a
+  later upstream pin bump, M3 needs to rebase first.
+- **Asset hosting.** While we pin claude-hooks' release, the
+  embedder's lifecycle is tied to claude-hooks' tagging
+  cadence. Mitigation: mirror to opencoti's own release once
+  it's set up.
+- **`@opencoti/tiers` ↔ `@opencoti/embedder` boundary.** The
+  embedder depends on tiers (for GPU detect + download helper),
+  and tiers dynamic-imports `@opencoti/llamafile`. No cycle
+  introduced (embedder is the new edge of the DAG), but confirm
+  with `bun --cwd packages/opencoti-embedder typecheck` after
+  the shared-download refactor.

docs/features/opencoti_server.md

@@ -0,0 +1,1010 @@
+# F3 — opencoti-server (Go companion daemon)
+
+> Parent: [../MASTER_PLAN.md](../MASTER_PLAN.md)
+> Status: **planning**
+> Owner: TBD
+> Reference: `/shared/dev/claude-hooks` (the inspiration; mirror its
+> goals, then go further)
+
+## Problem
+
+`claude-hooks` is a Python daemon that wraps Claude Code with a hook
+API and adds persistent memory, multi-session coordination, tool
+glue, and a small zoo of integrations. It works very well *for Claude
+Code*. opencoti has the same shape of needs — and a few extra
+because opencoti has its own tier engine, its own local llamafile,
+and its own session model.
+
+We want a Go daemon that is to opencoti what claude-hooks is to
+Claude Code, **but deeper** because we control opencoti's source.
+Instead of bolting onto a hook API from outside, opencoti-server can
+be a first-class peer of opencoti's runtime.
+
+## Goals
+
+- **G1.** Single Go binary, `opencoti-server`, that exposes a small,
+  well-versioned API to opencoti.
+- **G2.** Persistent memory (vector + KG) shared across opencoti
+  sessions on a host.
+- **G3.** Multi-session coordination: opencoti CLI instances on the
+  same host see each other, share context where the user wants it.
+- **G4.** Tool glue: a place to register external tools (search,
+  filesystem extensions, MCP-like) once and have them appear in all
+  opencoti sessions.
+- **G5.** Event sink for tier-engine routing decisions (so we can
+  log, audit, and later learn from them).
+- **G6.** Hook fan-out compatible with the existing claude-hooks
+  ecosystem where it makes sense — but not bound by it.
+
+## Non-goals (for now)
+
+- Replacing claude-hooks. opencoti-server is opencoti-specific.
+- Becoming a generic "MCP host". MCP integration is a feature, not
+  the architecture.
+- Multi-host federation. Single-host first.
+
+## Design sketch
+
+### Where it lives
+
+- `opencoti/server/` — top-level non-bun source root. Go module.
+  Putting it outside `packages/` keeps the bun workspace clean.
+- `packages/opencoti-server-client/` — TypeScript client used by
+  opencoti to talk to the server.
+- **Surgical hooks** in opencode session lifecycle (session start,
+  session end, turn complete, tool invocation, model call result):
+  each hook is a one-line "if a client is available, notify it".
+  Listed in `docs/protocols/UPSTREAM_SYNC.md`.
+
+### Transport
+
+Unix domain socket on Linux/macOS, named pipe on Windows. Loopback
+TCP as a fallback. HTTP/JSON wire format for simplicity (no gRPC
+build dependency for the client).
+
+**Daemon-launch policy (applies to opencoti-server and every
+opencoti daemon)**:
+
+- Listen port MUST come from opencoti's reserved
+  **47000-48000 range** — never 38000-39000 or 18790-18811
+  (claude-hooks territory). Tentative reservation table to
+  keep adjacent ports clear of accidental collision:
+  - **47092** — `@opencoti/embedder` (F4)
+  - **47190** — opencoti-server daemon (proposed default; revisit
+    when F3 M1 ships)
+  - **47191** — opencoti-server dashboard (proposed default)
+- First-run setup MUST ask the user explicitly where the
+  daemon should bind: `127.0.0.1` (default), all interfaces,
+  or a specific IP. The default port from the reservation
+  table is proposed; the user may override either field within
+  the 47000-48000 range. Setup writes the validated choice to
+  the user's config.
+
+### Persistence
+
+Two backends, picked per-deployment:
+
+- **pgvector** — same as solidPC's `claude-hooks` setup. Best when a
+  Postgres is already on the host.
+- **sqlite-vec** — single-file, no external dependency. Default for
+  fresh installs.
+
+The backend choice is config; the memory API is identical.
+
+### Surface (illustrative)
+
+```
+POST /v1/sessions                          # register a session
+POST /v1/sessions/:id/turns                # record a turn
+POST /v1/memory                            # store a memory (M2)
+POST /v1/memory/search                     # vector recall (M2 — POST because the embedding is multi-KB)
+POST /v1/memory/collections                # collections CRUD (M2)
+PUT  /v1/memory/collections/:name/acl      # per-session ACL (M2)
+GET  /v1/memory/count                      # count (M2)
+POST /v1/memory/kg/entities                # KG ops
+POST /v1/memory/kg/relations
+POST /v1/tier-events                       # log a tier-engine decision
+GET  /v1/tools                             # list registered tools
+POST /v1/tools/:name/invoke                # invoke (server-side glue)
+GET  /v1/healthz                           # M1; M2 extended with store_ok/store_path/embedding_dim/store_error
+```
+
+The server is **embedder-agnostic** at M2: clients pass vectors,
+not text. The text→vector convenience (e.g.,
+`GET /v1/memory?q=<text>` proxying to the embedder daemon) is a
+deferred follow-up — adding it requires a runtime dep on
+`@opencoti/embedder` (F4 M5) and is most naturally landed once
+the TS client (M3) demonstrates the convenience is wanted at
+the HTTP layer vs done client-side.
+
+API is **versioned** (`/v1`) and the wire schema is owned by
+`packages/opencoti-server-client/` (so opencoti can rev independently
+of the server binary, within compatible versions).
+
+### Mirroring claude-hooks goals
+
+Same goals, mapped:
+
+| claude-hooks | opencoti-server |
+| --- | --- |
+| `claude-hooks` recall hooks | implicit recall on `/v1/memory` query, surfaced by opencoti's prompt builder |
+| `mcp__pgvector__*` MCP tools | `/v1/memory` (and a thin MCP shim if external MCP clients want it) |
+| Stop hook auto-ingest | server records turn-complete events, applies same heuristics |
+| `claude-hooks` companion tools (Episodic etc.) | first-class tools registered via `/v1/tools` |
+
+### Deeper integration than claude-hooks
+
+Because we own opencoti, we can:
+
+- Receive **tier-engine** decisions directly (provider chosen,
+  escalation reasons, cost) without scraping logs.
+- Receive **diff and tool-output** events as structured payloads, not
+  parsed from a transcript.
+- Push back: server can suggest a memory recall payload that
+  opencoti **injects directly** into the prompt builder, instead of
+  appending to a transcript.
+
+## Milestones
+
+### M1 — Server skeleton + healthz *(2026-05-23 — shipped)*
+
+- Go module under `opencoti/server/` (module
+  `github.com/mann1x/opencoti/server`, Go 1.22). Single dep:
+  `golang.org/x/sys` for the Windows `svc` packages.
+- `opencoti-server serve` binds a UDS by default on Unix
+  (`$XDG_RUNTIME_DIR/opencoti/server.sock` or
+  `~/.opencoti/server.sock` fallback) or `tcp://127.0.0.1:47190`
+  on Windows. `--addr unix:///path` and `--addr tcp://host:port`
+  override.
+- One HTTP endpoint: `GET /v1/healthz` returns 200 + JSON
+  `{status, version, started_at, uptime_seconds}`. All other
+  paths return a 404 envelope with code + message so the wire
+  format is consistent for M2+. `POST /v1/healthz` returns 405
+  with the same envelope.
+- Graceful shutdown on SIGTERM/SIGINT (Unix) or SCM
+  Stop/Shutdown (Windows). UDS file unlinked on Unix shutdown.
+- **Windows service support shipped in M1 (per user directive
+  2026-05-23):**
+  - `install` subcommand wraps SCM (`golang.org/x/sys/windows/svc/mgr`):
+    `--name`, `--display-name`, `--description`,
+    `--start-type=auto|manual|disabled`, `--addr`. Registers
+    the Event Log source under the same name.
+  - `uninstall` removes the SCM entry AND the Event Log source.
+  - `start` / `stop` wrap `mgr.Service.Start` /
+    `Service.Control(svc.Stop)`. `stop` waits up to 10s for the
+    state to become Stopped.
+  - When `svc.IsWindowsService()` returns true, `serve` enters
+    `svc.Run` with a handler that translates SCM Stop /
+    Shutdown into a graceful `http.Server.Shutdown`. Log
+    records flow to the Windows Application Event Log via
+    `eventlog.Open(name)`.
+  - Single source tree, build-tagged platform splits
+    (`*_windows.go` / `*_nonwindows.go`). Same binary surface
+    on every platform — `install` / `uninstall` / `start` /
+    `stop` on Unix print "Windows-only command" and exit 2.
+- 7 cross-platform tests (parser + listener + healthz/404/405
+  + UDS roundtrip + UDS unlink-on-cleanup + double-shutdown
+  safety) + 3 Windows-tagged tests for the `BuildMgrConfig`
+  install-config builder. All green on Linux; Windows
+  cross-compile clean.
+- **Hook footprint: ZERO.** M1 is purely additive — no
+  `opencoti-hook:` markers, no UPSTREAM_SYNC.md registry rows.
+  The autostart hook lands in M3.
+- **Deferred from M1:** Windows named-pipe TRANSPORT (use TCP
+  on Windows in M1; named-pipe support is a follow-up);
+  systemd/launchd unit files on Unix; CI workflows (the repo
+  has no CI yet — separate concern); the first-run setup
+  wizard step ("install as Windows service?" — comes when the
+  setup-flow picks up F3 concerns).
+
+### M2 — Memory backend (sqlite-vec) + recall API *(2026-05-23 — shipped)*
+
+- Go SQLite stack: `github.com/mattn/go-sqlite3` (cgo) +
+  `github.com/asg017/sqlite-vec-go-bindings/cgo`, sqlite-vec
+  statically linked via `sqlite_vec.Auto()` so the daemon ships
+  as a single binary (no `vec0.so` to bundle). CGO becomes a
+  hard build requirement at M2; cross-compile to Windows from
+  Linux needs MinGW-w64 (`apt install gcc-mingw-w64-x86-64`).
+- `internal/store/store.go` defines a `Store` interface
+  mirroring the TS `MemoryStore` in
+  `packages/opencoti-memory/src/types.ts` 1:1. M7's pgvector
+  backend will implement the same interface, so the HTTP layer
+  doesn't change.
+- `internal/store/sqlitevec/` is the M2 backend. DDL mirrors
+  `packages/opencoti-memory/src/schema.ts` byte-functionally:
+  four tables (`meta`, `collections`, `memories`, `session_acl`)
+  plus the `memory_vecs` vec0 virtual table; `SCHEMA_VERSION=1`;
+  `embedding_dim` parameter-substituted into the vec0 DDL at
+  first creation and stored in `meta` for validation on later
+  opens. Content idempotency: SHA256-hex with
+  `UNIQUE(collection_name, content_hash)`. Pure `ResolveAccessMode`
+  mirrors the TS resolver (global=r, session-owner=rw,
+  others=none; explicit `session_acl` rows override). Recall
+  is vec0 MATCH + overscan
+  (`k * min(8, max(2, len(candidates)))`) + join-back +
+  ACL filter, truncated to k.
+- **DB schema is wire-compatible with the in-process
+  `@opencoti/memory` package** — a DB created by either side
+  opens cleanly under the other. This is the load-bearing
+  invariant for M3's path-B refactor.
+- HTTP surface (eight new routes, all under `/v1/memory/*`):
+  - `POST /v1/memory/collections` — create (201 / 409
+    collection_exists / 400 invalid_collection_name).
+  - `GET /v1/memory/collections?session_id=...` — list,
+    optionally filtered through the ACL resolver.
+  - `DELETE /v1/memory/collections/{name}` — cascade-deletes
+    memories + vec0 rows + session_acl entries.
+  - `PUT /v1/memory/collections/{name}/acl` — set per-session
+    ACL (`r`/`w`/`rw`/`none`).
+  - `GET /v1/memory/collections/{name}/acl?session_id=...` —
+    read the resolved access mode.
+  - `POST /v1/memory` — store. 422 dim_mismatch on length
+    mismatch; 403 write_denied on ACL deny.
+  - `POST /v1/memory/search` — vector recall (POST because
+    a 1024-element Float32 array doesn't fit a query string).
+  - `GET /v1/memory/count?collection=...` — count, optionally
+    scoped to one collection.
+- All error responses use M1's `{error: {code, message}}`
+  envelope. Embeddings on the wire are JSON `[float32, ...]`
+  arrays (verbose but trivial for the M3 TS client; base64 raw
+  bytes is a follow-up if wire size measurably matters).
+- `/v1/healthz` extended with `store_ok`, `store_path`,
+  `embedding_dim`, and `store_error` (omitempty). When the
+  store fails to open, the daemon still serves `/v1/healthz`
+  with `store_ok=false` and `/v1/memory/*` returns 503
+  `store_unavailable` — partial degradation beats refuse-to-start
+  so the daemon stays observable when something's wrong with the DB.
+- `serve` gets `--db-path` (default
+  `$HOME/.opencoti/memory/state.db` — matches the TS
+  `defaultDbPath()` exactly so both implementations point at the
+  same file by default) and `--embedding-dim` (default 1024,
+  matches `DEFAULT_EMBEDDING_DIM`). `install` bakes both into
+  the SCM service-start arguments.
+- 30 tests cover the surface: 18 unit tests on the store
+  conformance (collection lifecycle, ACL resolver, idempotent
+  store, recall ordering, count, close+reopen, dim-mismatch
+  paths) + 12 HTTP integration tests via `httptest.NewServer`
+  (every endpoint, happy + 4xx + 5xx envelopes).
+- `testdata/crosslang/` ships a Go↔Bun schema-compat probe:
+  Go writes a DB → optionally invokes a Bun script that opens
+  the same DB via `@opencoti/memory` and confirms readback.
+  Skips cleanly when bun or the workspace package isn't on
+  PATH. `make test-cross-lang` is always safe to run.
+- **Hook footprint: ZERO** (unchanged from M1). M2 is purely
+  additive — the API exists but no surgical hooks into opencode
+  yet.
+- **Deferred from M2:** text→vector convenience
+  (`GET /v1/memory?q=<text>`); streaming recall (SSE);
+  pagination on list-collections; authentication for TCP
+  transport (UDS owner-only on Unix is the M2 security
+  boundary); pgvector backend (M7); migration tools (schemas
+  are identical — the DB just opens); server-side embedder
+  cache; bulk-store / batch endpoints; base64 raw-byte
+  embedding wire format.
+
+### M3 — TS client + opencoti hook to start the server *(2026-05-23 — shipped)*
+
+- `@opencoti/opencoti-server-client` published in-workspace under
+  `packages/opencoti-server-client/`. Single class
+  `OpencotiServerClient` implements `MemoryStore` over HTTP — all
+  eight `/v1/memory/*` routes from M2 plus a `healthz()` probe.
+  Wire format is M2's verbatim: JSON `[float32, ...]` for embeddings;
+  M1/M2's `{error: {code, message}}` envelope decoded into a
+  sentinel `OpencotiServerError({status, code, message})` so callers
+  can match on `code === "store_unavailable"`, `dim_mismatch`, etc.
+  Per-request `AbortController` + 5 s default timeout. `close()` is
+  a documented no-op (no per-instance handle to close).
+- **Transport** uses Bun's native `fetch` — `tcp://host:port` is
+  rewritten to `http://...` in `toFetchBase`; `unix:///path` passes
+  through verbatim (Bun supports UDS fetch natively, no custom
+  adapter needed). `defaultAddress(platform)` mirrors the Go
+  `DefaultAddress()` exactly: `$XDG_RUNTIME_DIR/opencoti/server.sock`
+  on Linux (`$HOME/.opencoti/server.sock` fallback), and
+  `tcp://127.0.0.1:47190` on Windows.
+- **Autostart** lives in `src/autostart.ts`:
+  `startAutostart({address, binaryPath?, dbPath?, embeddingDim?, ...})`
+  probes `/v1/healthz` first (200 → `already_running`), locates the
+  binary via `Bun.which` if none was passed (→ `no_binary` when
+  missing), spawns with `stdio: "ignore"` (mirroring the
+  `@opencoti/embedder` manager pattern), and polls `/v1/healthz`
+  every 200 ms until the deadline (default 5 s; →
+  `ready_timeout` if never recovers). Returns a discriminated
+  `AutostartOutcome` so consumers can log five distinct states
+  (`already_running | spawned | no_binary | spawn_failed | ready_timeout`)
+  without exception handling. Test seams: `spawnImpl`, `fetchImpl`,
+  `whichImpl`, `sleepImpl`.
+- **Surgical hook footprint: 3 markers + 1 package.json dep.** All
+  three TS markers land in `packages/opencode/src/config/config.ts`
+  (the same file that already carries `opencoti-default-plugins`):
+  the import next to it, the `opencoti.server.*` schema struct
+  (`autostart: boolean`, `address: string`, `binary_path: string`,
+  `db_path: string`, `embedding_dim: PositiveInt`), and the
+  fire-and-forget `void maybeStartOpencotiServer(...)` call site
+  right after the existing `applyDefaultPlugins(...)` call. The
+  hook's full substance lives in
+  `@opencoti/opencoti-server-client/autostart-hook` so the
+  opencode-side surface stays minimal — three lines plus the
+  package.json dep (no anchor comment in JSON, registered in the
+  table).
+- **Plugin swap.** Both `@opencoti/memory-plugin` and
+  `@opencoti/tui-memory` accept a new optional
+  `server_address?: string`. The default `StoreFactory`:
+  if `serverAddress` is set, instantiate `OpencotiServerClient`,
+  call `healthz()`, and return the client when
+  `storeOk === true`; otherwise silently fall through to the
+  existing in-process `SqliteVecStore.open(...)`. The
+  `__setStoreFactory` test seam is unchanged — existing tests
+  continue to inject stubs that ignore `serverAddress`.
+- **Health envelope mismatch detection.** Client validates that
+  `embedding_dim` from the server matches the configured dim;
+  on mismatch, `healthz()` returns `storeOk: false` with a
+  `dim mismatch` error so the plugin falls back to the
+  in-process store rather than corrupting the wire format.
+  This is **client-side**; the server's own dim check still
+  fires on `POST /v1/memory`.
+- **Tests.** 56 new tests in the client package: 33 mocked-fetch
+  client conformance (every endpoint, happy + 4xx + 5xx envelopes,
+  dedup, dim mismatch, store_unavailable propagation), 11 pure
+  tests for `defaultAddress` / `toFetchBase` / `joinURL` /
+  `probeReady`, 6 mocked-spawn `startAutostart` cases (already
+  running, no binary, spawned-then-ready, ready timeout,
+  spawn failed, --addr passthrough). Plus 2 new cases in the
+  memory-plugin tests and 3 new in the tui-memory tests for the
+  `server_address` swap path. All 23 workspace packages
+  typecheck clean; 82 tests pass across the three affected
+  packages.
+- **Live verification.** Manual end-to-end smoke against the
+  F3 M2 binary (TCP loopback): `healthz` returns the expected
+  shape (`store_ok: true`, `embedding_dim: 64`), `createCollection`
+  + `setSessionAcl(rw)` + `store(...)` (with `deduplicated:
+  false` then `true` on idempotency) + `recall(...)` (distance
+  0 on exact match) + `count(...)` (returns 1) + `deleteCollection`
+  all return the expected shapes. UDS path tested at the
+  discover / autostart layer; full UDS smoke deferred to
+  M4 when session-event hooks bring it under day-to-day use.
+- **Path-B loop closed.** With M3 in, the F4 M5
+  in-process-then-daemon migration path described in
+  [memory_embedder.md](memory_embedder.md) is mechanically
+  complete: setting `opencoti.server.address` (or `autostart:
+  true`) in `opencode.jsonc` is the only change a user makes
+  to switch from the in-process store to the Go daemon. No
+  code paths change in `@opencoti/memory` itself.
+- **Deferred from M3:** detached / setsid daemon lifecycle
+  (M3's spawn dies with opencode — cross-session sharing
+  requires running the daemon externally, e.g. the M1 Windows
+  service installer or a future systemd unit); the first-run
+  "install as a service?" wizard step (a follow-up once the
+  setup-flow picks up F3 concerns); a text-`q=` convenience
+  endpoint (`GET /v1/memory?q=<text>`) — every M3 caller has
+  the embedder in-process and can pre-embed; retry / backoff
+  on transient HTTP errors (single attempt, the plugin's
+  silent fallback handles the failure case); bulk-store /
+  batch endpoints; TLS / shared-secret authentication for TCP
+  transport (UDS owner-only remains the M3 security boundary).
+
+### M4 — Session + turn events flowing *(2026-05-23 — shipped)*
+
+**Design pivot vs the original spec.** M4 was originally specced
+as *surgical hooks at session start/end/turn complete*. Phase-1
+exploration confirmed opencode's `@opencode-ai/plugin` API
+already exposes those events fully typed via `Hooks.event`
+(session.created/updated/deleted/error/idle plus
+session.status). Following the precedent set by M5-D2
+(`@opencoti/memory-plugin`), M4 ships as a plugin with **zero
+opencoti-hook footprint** — strictly richer than a surgical hook
+for this use case (typed payloads, zero upstream-source touch,
+no UPSTREAM_SYNC.md row to maintain on every sync). Hook count
+stays at 9 source markers + 4 JSON deps (unchanged from F3 M3).
+
+- **Per-feature schema versioning.** `meta.schema_version` stays
+  at 1 (memory tables — kept stable so M2/M3 TypeScript
+  `@opencoti/memory` clients still open M4 DBs cleanly). New
+  `meta.sessions_schema_version = 1` is written by M4+ daemons
+  and reported on `/v1/healthz`. M3 clients without the key
+  treat the daemon as pre-M4; M4 plugins disable forwarding
+  silently when `sessionsSchemaVersion < 1`.
+- **Three new tables** in the existing `state.db` (no new
+  `--flag`, no migration tool):
+  - `sessions` — soft-delete via `deleted_at`. Indices on
+    `parent_id` (fork-tree queries) and `deleted_at`
+    (cheap `WHERE deleted_at IS NULL`). Upsert is idempotent
+    on `id`; `INSERT ... ON CONFLICT(id) DO UPDATE SET ...`
+    overwrites all columns from the latest snapshot.
+  - `session_events` — append-only audit log. `FOREIGN KEY ...
+    ON DELETE CASCADE` would normally wipe history on
+    `DELETE FROM sessions`; we use soft-delete instead so the
+    audit trail outlives the session row.
+  - `session_messages` — UPSERT on opencode `MessageID`. The
+    plugin re-sends the full latest snapshot on every turn
+    boundary; SQLite UPSERT handles dedup.
+- **SessionStore interface** (`internal/store/store.go`): 8
+  methods + sentinel errors (`ErrSessionNotFound`,
+  `ErrInvalidSessionID`, `ErrSessionPayloadTooLarge`). The
+  same `*sqlitevec.Store` implements both `Store` (memory) and
+  `SessionStore`; server.go type-asserts at each entry point.
+  M7 pgvector will implement the same interface.
+- **HTTP surface** under `/v1/sessions/*`:
+  - `POST /v1/sessions` — upsert (201 on insert, 200 on update,
+    400 invalid_session_id, 413 payload_too_large).
+  - `GET /v1/sessions[?parent_id=&include_deleted=&limit=&offset=]`
+    — list, default newest-first by `updated_at`, omitting
+    soft-deleted rows.
+  - `GET /v1/sessions/{id}` / `DELETE /v1/sessions/{id}` —
+    fetch / soft-delete.
+  - `POST /v1/sessions/{id}/events` /
+    `GET /v1/sessions/{id}/events[?type=&limit=&offset=]` —
+    append + list audit entries.
+  - `POST /v1/sessions/{id}/messages` /
+    `GET /v1/sessions/{id}/messages[?limit=&offset=&since=]` —
+    bulk-upsert (idempotent on message id) + list.
+  - `GET /v1/sessions/{id}/turns` — coarse server-side
+    projection grouping messages by user-message boundaries.
+    Authoritative turn-boundary logic stays in opencode's
+    `compaction.ts`; this view is for UI / debugging.
+- **healthz extension**: `sessions_schema_version` field
+  added; M3 clients without the field treat the daemon as
+  pre-M4.
+- **Payload size guards**: 8 MiB hard cap on
+  `session_events.payload` and the bulk-upsert messages body
+  (413 `payload_too_large`); the plugin chunks at a softer
+  1 MiB default before posting.
+- **TS client extension** (`@opencoti/opencoti-server-client`):
+  8 new `OpencotiServerClient` methods (`upsertSession`,
+  `listSessions`, `getSession`, `deleteSession`,
+  `appendSessionEvent`, `listSessionEvents`,
+  `upsertSessionMessages`, `listSessionMessages`,
+  `listSessionTurns`). Wire format snake_case → camelCase via
+  private mappers (same pattern as M2's `mapHit` /
+  `mapCollectionInfo`). `healthz()` return type +
+  `ProbeResult` gain `sessionsSchemaVersion?: number`.
+- **New plugin `@opencoti/session-events-plugin`**:
+  subscribes to `session.created/updated/deleted/error/idle`
+  and `session.status` (idle transition) via opencode's
+  typed `Hooks.event`; does **not** subscribe to
+  `message.updated` (fires thousands of times per turn —
+  text deltas, tool calls, reasoning chunks — and forwarding
+  every delta would melt the daemon). On idle, the plugin
+  fetches the session's latest messages via
+  `input.client.session.messages.list` and bulk-upserts in
+  byte-bounded chunks. Maintains a per-session merge cache so
+  opencode's partial `session.updated` payloads are reconciled
+  into a full snapshot before reaching the daemon.
+  Best-effort, never-throws: a forwarding plugin must not
+  propagate daemon failures into the opencode session.
+- **session.error before sessionID exists** lands on a
+  synthetic `__pre_session_errors__` session (lazily upserted
+  on first such error) so the audit log captures pre-creation
+  failures without losing context.
+- **Auto-wire**: one-line addition to
+  `@opencoti/tiers/default-plugins` `OPENCOTI_DEFAULT_PLUGINS`.
+  Any user with opencoti config in `opencode.jsonc` gets the
+  plugin automatically; without an `opencoti.server.address`
+  the plugin's healthz probe fails, the internal client stays
+  `undefined`, and every event handler early-returns — same
+  no-op-when-unconfigured ergonomic the memory-plugin uses.
+- **Tests**: 26 new Go tests (16 store unit + 10 HTTP
+  integration) — total Go test count climbs from 30 → 63.
+  14 new TS client tests (33 → 47). 16 new plugin tests in
+  `@opencoti/session-events-plugin`. All 24 workspace
+  packages typecheck clean.
+- **Live verification**: end-to-end smoke against a real M4
+  daemon (TCP loopback) confirmed healthz reports
+  `sessions_schema_version: 1`, full upsert → update → get →
+  list → events → messages → turns → soft-delete cycle
+  works, soft-deleted session's events + messages remain
+  queryable through `include_deleted=true`.
+- **Deferred from M4**:
+  - **`message.updated` real-time forwarding** — coalescing on
+    idle is the right granularity; per-delta forwarding would
+    mean O(text-deltas) HTTP roundtrips per turn.
+  - **Per-tool-invocation events** as first-class rows —
+    captured today inside the message's `parts` JSON; can be
+    promoted to a `session_tool_calls` table later if needed.
+  - **WebSocket / SSE push from daemon → client** —
+    request-response only in M4.
+  - **TLS / authentication for `/v1/sessions/*` TCP transport**
+    — UDS owner-only on Unix remains the M4 security boundary.
+
+### M5 — Tier-engine event sink (shipped 2026-05-23)
+
+- **Daemon side: `/v1/tier-events`.**
+  - `POST /v1/tier-events` appends a row (201; 400 invalid_tier_event;
+    413 payload_too_large at the same 8 MiB cap as `/v1/sessions/*`).
+    Body: `{event_type, ts?, session_id?, payload}` where `payload`
+    is opaque JSON — the daemon does not introspect it.
+  - `GET /v1/tier-events?session_id=&event_type=&since=&limit=&offset=`
+    returns the audit log newest-first.
+  - One new `tier_events` table: `(id PK AUTO, session_id TEXT,
+    event_type TEXT, payload TEXT, ts INTEGER)` + two indices
+    `(session_id, ts)` and `(event_type, ts)`. **`session_id` is a
+    soft reference — no FK, no cascade.** Tier events fire intra-turn
+    and can land before `session.created` reaches the daemon; the
+    soft reference guarantees no event is dropped.
+  - Per-feature `meta.tier_events_schema_version = 1`. The
+    cross-language sqlite_vec contract continues to live on
+    `meta.schema_version = 1` (still M2-compatible with the TS
+    `@opencoti/memory` reader); the new key is independent.
+
+- **TS plumbing.** `@opencoti/opencoti-server-client` gains
+  `appendTierEvent` + `listTierEvents` (camelCase→snake_case wire
+  mapping mirroring `mapSessionMessage`). `healthz()` surfaces
+  `tierEventsSchemaVersion?: number`; plugins use it as their
+  feature gate.
+
+- **Design pivot continues: plugin path > surgical hook.** The
+  `Telemetry` interface in `@opencoti/tiers` (designed for this
+  milestone — its docstring at `telemetry.ts:1-4` said so) is
+  already threaded end-to-end through `runtime.ts:99 →
+  executor.ts → escalator.ts → fanout.ts`. M5 plugs a real sink
+  into the existing seam via a new module-level
+  `registered-telemetry.ts` slot (mirrors `active-config.ts`).
+  Precedence chain in `runtime.ts:99` becomes:
+
+  ```ts
+  const telemetry =
+    hook.telemetry ??
+    execOpts.telemetry ??
+    getRegisteredTelemetry() ??
+    noopTelemetry
+  ```
+
+  The new `@opencoti/tier-events-plugin` calls
+  `setRegisteredTelemetry(impl)` during plugin boot when the
+  daemon's healthz reports `tierEventsSchemaVersion >= 1`. **Zero
+  new opencoti-hook source markers** — count unchanged from F3 M4
+  (9 source + 4 JSON deps + 3 prose). See the F1 page for where
+  the Telemetry interface itself was added.
+
+- **Fire-and-forget posture.** The plugin's Telemetry impl maps
+  each method to a `client.appendTierEvent(...).catch(() => {})`
+  — synchronous to the caller, HTTP swallowed on failure. The
+  tier engine never blocks on the daemon. Cf. M4's
+  `@opencoti/session-events-plugin`, same design.
+
+- **Auto-wire via defaults.** `@opencoti/tier-events-plugin` is in
+  `OPENCOTI_DEFAULT_PLUGINS` alongside the M4 session plugin. Any
+  user with `opencoti: {...}` in their config gets the audit log
+  for free; if the daemon is unreachable, the healthz gate fails
+  and the runtime stays on noop.
+
+- **Cross-version compat.** An M5 plugin against an M4 daemon
+  sees `sessions_schema_version: 1` but no
+  `tier_events_schema_version` in healthz, so
+  `setRegisteredTelemetry` is never called and the runtime stays
+  on noop. An M5 daemon against M4 clients keeps shipping the
+  same `/v1/sessions/*` surface untouched.
+
+- **Explicitly deferred to later:** aggregation endpoints
+  (`/v1/tier-events/stats?...`), real-time push (WS/SSE),
+  server-side retention policy, client-side batching/coalescing,
+  FK on `session_id`. M5 ships raw events + filters; everything
+  on top of that is composable later.
+
+### M6 — Tool registry + invocation (shipped 2026-05-23)
+
+- **Daemon side: `/v1/tools`.**
+  - `GET /v1/tools` lists every registered tool sorted by name:
+    `{ tools: [{ name, description, params_schema, handler,
+    created_at, updated_at }] }`. `params_schema` is a
+    string-serialised JSON Schema; `handler` is a stable opaque
+    dispatch ID (debug-only — invocation always goes by name).
+  - `GET /v1/tools/{name}` fetches one (404 tool_not_found).
+  - `POST /v1/tools/{name}/invoke` runs it. Body
+    `{ arguments: {...}, session_id? }` → `{ result: {...} }` (200)
+    or `{ error: { code, message } }` (400 invalid_arguments / 404
+    tool_not_found / 500 handler_failed / 413 payload_too_large at
+    the same 8 MiB cap as `/v1/sessions/*` and `/v1/tier-events`).
+  - One `tools` table: `(name PK, description, params_schema,
+    handler, created_at, updated_at)`. Process-scoped, no
+    session FK. Dispatch is a daemon-internal
+    `map[string]ToolHandler` in `toolhandlers.go`; **no external
+    registration in M6** (no POST/DELETE on `/v1/tools` itself) —
+    the catalog is seeded into the binary at boot.
+  - Per-feature `meta.tools_schema_version = 1`, independent of the
+    cross-language `meta.schema_version = 1` (still M2-compatible
+    with the TS `@opencoti/memory` reader).
+
+- **Two seed tools** are `UpsertTool`'d at startup (idempotent —
+  `created_at` fixed on first insert, `updated_at` bumps on
+  re-seed):
+  - **`opencoti_episodic_search`** — lexical `LIKE` search across
+    the M4 `session_messages` table joined with `sessions`,
+    newest-first. Params `{ query, limit?, session_id_excludes? }`.
+    The exclude list lets the plugin pass the live session ID so
+    the model recalls *other* sessions, not echoes of the current
+    one. The integrated analog of claude-hooks's `episodic_server`
+    — same SQLite file the sessions API writes, no second daemon,
+    no shell-out. FTS5/vector ranking is an M7+ optimisation.
+  - **`opencoti_recall`** — vector recall over the M2 memory store,
+    delegating to the same `Store.Recall` path `/v1/memory/search`
+    uses. Params `{ embedding, collection?, k?, session_id? }`.
+    **Takes a pre-computed embedding, not a query string** — the
+    daemon bundles no embedder, so the caller (the plugin, or an
+    operator via curl) embeds the query and POSTs the float vector;
+    length must equal the daemon's `embedding_dim`. `session_id`
+    drives the M2 per-session ACL check. Coexists intentionally
+    with `@opencoti/memory`'s in-process `__memory_recall`: that
+    one survives a daemon being down; `opencoti_recall` gives the
+    same surface to plugins without direct DB access.
+
+- **TS plumbing.** `@opencoti/opencoti-server-client` gains
+  `listTools` + `getTool` + `invokeTool`; `healthz()` surfaces
+  `toolsSchemaVersion?: number` as the plugin's feature gate.
+  `ServerTool` is camelCase (`paramsSchema` ← `params_schema`),
+  `paramsSchema` left as `unknown` (interpreted in the plugin).
+
+- **Design pivot continues: plugin path > surgical hook.** A new
+  module-level `registered-server-tools.ts` slot in
+  `@opencoti/tiers` (mirrors M5's `registered-telemetry.ts`) holds
+  the daemon tools as a `Record<string, Tool>`; `runtime.ts` merges
+  it **last** into the tier tool list, next to synthetic-tier and
+  memory-bridge tools:
+
+  ```ts
+  const tools = mergeTools(hook.prepared.tools, {
+    ...syntheticTools(hook.input.sessionID, telemetry),
+    ...memoryTools,
+    ...(getRegisteredServerTools() ?? {}),
+  })
+  ```
+
+  Server tools merge last so user-explicit + memory tools win on a
+  name collision (none expected — server tools are `opencoti_*`,
+  memory `__memory_*`, synthetic-tier `__tier_*`). **Zero new
+  opencoti-hook source markers** — count unchanged from F3 M4/M5
+  (9 source + 4 JSON deps + 3 prose).
+
+- **The `@opencoti/server-tools-plugin`** probes `/v1/healthz`
+  (gated on `toolsSchemaVersion >= 1`), fetches `/v1/tools` once on
+  boot, and builds an AI SDK `dynamicTool` per entry:
+  `params_schema` round-trips through `ai`'s `jsonSchema()` (no
+  JSON-Schema-to-Zod reimplementation); each `execute` forwards to
+  `/v1/tools/{name}/invoke`. Tool names are prefixed (`opencoti_`,
+  idempotently). The session ID arrives via `experimental_context`
+  on the single `openStream` `streamText` call and is forwarded as
+  `session_id`. Invocations are awaited (not fire-and-forget — tool
+  calls are model-blocking), but errors crash the *tool call*, not
+  the session: 404→`tool_not_found`, 400→`invalid_arguments`,
+  5xx/network→`tool_handler_failed`, deadline→`timeout`
+  (plugin-owned `Promise.race`, separate `invoke_timeout_ms`).
+  Args over 1 MiB are pre-rejected client-side.
+
+- **Auto-wire via defaults.** `@opencoti/server-tools-plugin` is in
+  `OPENCOTI_DEFAULT_PLUGINS`. No reachable daemon → healthz gate
+  fails → `setRegisteredServerTools` never called → runtime tool
+  set unchanged.
+
+- **Cross-version compat.** An M6 plugin against an M5 daemon sees
+  no `tools_schema_version` in healthz, so the gate fails and the
+  tool set is unchanged — a model query expecting the tool gets
+  "I don't have that tool", no crash. An M6 daemon serves M5
+  clients the `/v1/sessions/*` and `/v1/tier-events` surfaces
+  untouched.
+
+- **Explicitly deferred:** external tool registration (POST/DELETE
+  on `/v1/tools` — needs a per-tool ACL/owner model); streaming
+  tool results; tool invocations as `tier_events` audit rows; an
+  MCP wrapper for the registry; FTS5/vector ranking in
+  `episodic_search`; an in-daemon embedder so `opencoti_recall`
+  can take a `query` string directly.
+
+### M7 — pgvector backend parity (shipped 2026-05-23)
+
+opencoti-server gains its **second storage backend**: PostgreSQL +
+the pgvector extension, via `github.com/jackc/pgx/v5` (pure Go, no new
+cgo). `*pgvector.Store` implements the **full** interface family —
+`Store` + `SessionStore` + `TierEventStore` + `ToolStore` +
+`SchemaInspector` — so the HTTP layer, `/v1/healthz`, and the TS client
+are **untouched**. sqlite-vec stays the zero-dependency default;
+pgvector is for hosts that already run Postgres.
+
+**Schema-phrasing correction.** The original M7 stub said *"/v1/memory
+works against pgvector with the same schema as claude-hooks."* That
+predated the M4–M6 buildout. opencoti's binding contract is now its
+**own** `Store` interface (collections + per-session ACL + sessions +
+tier_events + tools), which is a different data model from
+claude-hooks's flat `memories` + `kg_*` schema. So M7 implements
+**opencoti's own model on a dedicated `opencoti` Postgres database**,
+isolated from claude-hooks. "Same as claude-hooks" is reread as *same
+storage technology (Postgres + pgvector)*, not the same tables.
+
+- **`internal/store/pgvector/`** mirrors `internal/store/sqlitevec/`
+  table-for-table in PG dialect, with three dialect differences:
+  - the embedding lives **inline** on `memories` as a `vector(dim)`
+    column (no separate vec0 virtual table); an HNSW `vector_l2_ops`
+    index is created when `dim ≤ 2000` (pgvector's HNSW ceiling) —
+    correctness-neutral, perf-positive.
+  - `Recall` pushes the ACL filter into the query
+    (`WHERE collection_name = ANY($readable) ORDER BY embedding <-> $q
+    LIMIT k`) — Postgres can filter + rank in one statement where
+    sqlite-vec must overscan + join-back + post-filter. The `<->`
+    operator is **L2**, matching sqlite-vec's vec0 default, so
+    `MemoryHit.Distance` stays comparable across backends.
+  - `Store` dedups via `INSERT … ON CONFLICT (collection_name,
+    content_hash) DO NOTHING RETURNING id` in one round trip.
+- **Per-feature schema versions** (`schema_version`,
+  `sessions_/tier_events_/tools_schema_version`, all `1`) live in a
+  `meta` key/value table, validated on open exactly as the sqlite-vec
+  backend does — but they are pgvector's own contract (no cross-language
+  reader, since pgvector is daemon-only).
+- **Shared conformance harness** `internal/store/storetest/` is the
+  parity guarantee: a single `RunConformance` body (Memory, Sessions,
+  TierEvents, Tools sub-suites) that **both** backends opt into via a
+  thin `conformance_test.go`. Either backend drifting from the contract
+  fails the same assertions. sqlite-vec runs it with `t.TempDir`;
+  pgvector with a testcontainers fixture.
+- **Pure-logic lifted to `internal/store/common.go`** (package `store`):
+  the ACL resolver (`ResolveAccessMode`/`CanRead`/`CanWrite`), the
+  collection/session validators, `Sha256Hex`, and the dim bounds — one
+  cgo-free source of truth both backends share. sqlite-vec keeps its
+  exported names as thin delegating wrappers, so the HTTP layer's
+  `sqlitevec.ValidateCollectionName`/`ValidateSessionID` call sites are
+  byte-identical.
+- **Backend selection** is a serve-time flag, not a build flag:
+  `--backend sqlite|pgvector` (default `sqlite`) + `--pg-dsn` (falling
+  back to `$OPENCOTI_PG_DSN`). `--db-path` stays sqlite-only;
+  `--embedding-dim` applies to both (locks the `vector(dim)` column).
+  The interface var is assigned only on a successful open (avoids the
+  typed-nil trap), so a pgvector open failure degrades to
+  `store_ok=false` + a clear `store_error` rather than crashing. The
+  Windows `install` command bakes `--backend`/`--pg-dsn` into the
+  service args alongside the existing flags.
+- **healthz** reports a **redacted** `store_path` for pgvector
+  (`pg://user@host:port/db`, never the password).
+- **Tests**: `pgtest` spins an ephemeral `pgvector/pgvector:pg17`
+  container via testcontainers-go and skips cleanly when Docker is
+  absent (`testcontainers.SkipIfProviderIsNotHealthy`), so
+  `make test` stays green on a Docker-less host. Both backends pass the
+  full shared conformance suite; an end-to-end smoke (daemon →
+  pgvector container) confirms collection create, store, and `<->` -
+  ranked recall over HTTP.
+- **Toolchain**: the modern pgx / pgvector-go / testcontainers-go
+  releases require **Go 1.25**, so the module's `go` directive and the
+  host toolchain moved to go1.25 (latest stable). No build-posture
+  regression — pgx is pure Go; sqlite-vec's existing cgo requirement is
+  unchanged.
+- **Zero new surgical hooks.** pgvector is additive Go inside
+  `opencoti/server/`; no TS changes. The surgical-hook grep count
+  stays at 18 (see `docs/protocols/UPSTREAM_SYNC.md`).
+
+**Deferred from M7:** data migration between backends (a `migrate`
+subcommand — "Migration tools later"); a TS pgvector backend for the
+in-process `@opencoti/memory` (pgvector is daemon-only); external tool
+registration over HTTP (still security-gated, from M6); pgxpool tuning
+/ read replicas.
+
+### M8 — Multi-session coordination (shipped 2026-05-23)
+
+Goal **G3**: opencoti instances/sessions on one host see each other and
+cooperate. M8 ships the *live-coordination* half as an **in-memory,
+ephemeral** hub — deliberately store-independent (neither the `Store`
+interface nor either backend is touched), because presence and locks are
+runtime state that should not survive a daemon restart.
+
+- **`internal/coord.Hub`** — backend-agnostic, one `sync.RWMutex`:
+  - **Presence**: `RegisterPeer` / `Heartbeat` / `DeregisterPeer` /
+    `ListPeers`, with a background TTL sweep that reaps peers whose
+    heartbeat lapsed (default 45s) and emits `peer.left`. `ListPeers`
+    also filters expired peers lazily.
+  - **Broadcasts**: a pub/sub bus — `Publish` assigns a monotonic `seq`,
+    appends to a bounded replay ring, and fans out non-blockingly to
+    subscribers (a full subscriber channel is dropped + closed so the
+    client reconnects with its last seq). `Subscribe(since)` atomically
+    snapshots the replay backlog and registers for live events.
+  - **Advisory locks**: try-only `AcquireLock` (reentrant-by-holder
+    refresh; expired locks reclaimable) / `ReleaseLock` (holder-checked)
+    / `ListLocks`. Lock transitions emit `lock.acquired` / `lock.released`.
+- **HTTP** (`internal/server/coord.go`), hub injected via `Options.Hub`,
+  503 `coord_unavailable` when absent:
+
+  | Method + path | Purpose |
+  | --- | --- |
+  | `POST /v1/coord/peers` | register/upsert presence → PeerInfo |
+  | `POST /v1/coord/peers/{id}/heartbeat` | refresh TTL |
+  | `DELETE /v1/coord/peers/{id}` | deregister |
+  | `GET /v1/coord/peers` | list live peers |
+  | `POST /v1/coord/broadcast` | publish `{peer_id,topic,payload}` → `{seq}` |
+  | `GET /v1/coord/events?since=&peer_id=` | **SSE** event stream (the daemon's first) |
+  | `POST /v1/coord/locks/{name}` | acquire `{holder,ttl_ms?}`; 200 or 409 `lock_held` |
+  | `DELETE /v1/coord/locks/{name}` | release `{holder}`; 200 / 404 `lock_not_held` / 409 `lock_not_holder` |
+  | `GET /v1/coord/locks` | list held locks |
+
+  SSE is viable because the `http.Server` sets no `WriteTimeout`; the
+  handler exits on request-context cancellation so graceful shutdown
+  releases it within the grace window. `/v1/healthz` gains
+  `coord_ok` + `coord_peers`.
+- **TS client**: `registerPeer` / `heartbeatPeer` / `deregisterPeer` /
+  `listPeers` / `broadcast` / `acquireLock` / `releaseLock` /
+  `listLocks`, plus `subscribeCoordEvents` — the client's first
+  streaming method (reads `response.body`, parses `data:` frames).
+- **`@opencoti/coordination-plugin`**: registers each session as a peer,
+  heartbeats while active, deregisters on delete, subscribes to the
+  event stream (SSE) to keep a live peer view, and advertises
+  "N other active opencoti session(s)" in the system prompt. Auto-wired
+  via the `@opencoti/tiers` default-plugins list — **no surgical hook**
+  (grep count stays 18).
+
+**Deferred to F3 M9:** opt-in *shared context* (a thin convention atop
+existing M2 global collections + per-session ACL — a session opts to
+expose a collection to peers); the sqlite↔pgvector `migrate` subcommand
+(open from M7); blocking/queued lock acquire (M8 is try-only). Cross-host
+federation remains an explicit F3 non-goal.
+
+### M9 — Opt-in shared context (shipped 2026-05-23)
+
+Goal **G3**, the *persisted-context* half: a session exposes one of its
+session-scoped memory collections to peer sessions on the same host. The
+storage model from M2 already supports the grant, so M9 adds **no new
+`Store` method, no new schema, no new surgical hook** (grep count stays
+18). It is a thin convention bridging two things that already exist — the
+persisted per-session ACL (`SetSessionACL`) and M8's ephemeral coord bus.
+
+- **`internal/share.Manager`** — in-memory registry + one goroutine:
+  - `Share(collection, owner, mode)` records the share, grants read-ACL
+    to every live peer `!= owner` via `SetSessionACL`, and publishes
+    `collection.shared` on the hub.
+  - It **subscribes to the hub** and, on `peer.joined`, grants every
+    active share to the newcomer — so peers that join *after* a share
+    still get access (auto-grant via the bus, dogfooding M8's SSE).
+  - `Unshare(collection, owner)` revokes live peers (writes mode `none`,
+    the non-owner default) and publishes `collection.unshared`.
+  - **Persisted vs ephemeral:** the ACL grants persist (they survive a
+    restart); the "keep auto-granting new joiners" intent is in-memory
+    and lost on restart by design — existing grants remain, but the owner
+    must re-share to resume auto-granting. This keeps M9 storage-free.
+- **Recall "just works":** once a peer holds an `r` grant, its no-filter
+  `Recall` includes the shared collection automatically (M2's
+  readable-collections resolution), so no recall-path change is needed.
+- **HTTP** (`internal/server/share.go`), Manager constructed in
+  `server.New()` when `Store`+`Hub` are present, released on `Shutdown`;
+  503 `share_unavailable` when the hub is absent:
+
+  | Method + path | Purpose |
+  | --- | --- |
+  | `POST /v1/memory/collections/{name}/share` | owner-only share `{owner_session, mode?}` (mode default `r`) → `{granted}` |
+  | `DELETE /v1/memory/collections/{name}/share` | withdraw `{owner_session}` → 200 |
+  | `GET /v1/memory/shares` | list active shares |
+
+  **Owner-only:** the handler looks the collection up via
+  `ListCollections` (no filter) and requires `scope == session` and
+  `session_id == owner_session`; global collections (already `r`-for-all)
+  are rejected `not_shareable`, a different owner `not_owner` (403).
+- **TS client**: `shareCollection` / `unshareCollection` / `listShares`
+  + `SharedCollectionInfo`.
+- **`@opencoti/coordination-plugin`** (extended in place — no new plugin):
+  consumes `collection.shared` / `collection.unshared` to advertise
+  peer-shared collections in the system prompt, and gains an opt-in
+  `share_session_collection` flag (default **false**) that shares the
+  session's own collection (`sessionCollectionName(id)`) on
+  `session.created` and unshares it on `session.deleted` (best-effort; a
+  not-yet-created collection's 404 is swallowed).
+
+**Deferred to F3 M10:** the sqlite↔pgvector `migrate` subcommand — **not**
+a cutover: both backends are first-class and may run in parallel; migrate
+is an idempotent, re-runnable, either-direction copy (`--from`/`--to`,
+dedupe on `content_hash`, upsert PKs) that tops up a parallel target,
+never abandoning the source. Also deferred: revoke-on-peer-leave cleanup
+(M9 keeps grants on `peer.left`, since peers may return) and wildcard /
+group ACLs.
+
+### M10 — sqlite↔pgvector `migrate` (shipped 2026-05-24)
+
+A daemon-internal `migrate` subcommand that copies data between the two
+first-class backends. It is **not a cutover**: both backends stay
+first-class and may run in parallel (two daemons, or alternating
+`--backend`). `migrate` is an **idempotent, re-runnable, either-direction,
+selectable copy** that *tops up* a target (dedupe on natural keys), never
+"move then abandon source". Additive Go only — **no plugin, no HTTP
+surface, no TS client, no surgical hook** (grep count stays 18).
+
+**Use cases (maximum flexibility):**
+
+1. **Scale-up (primary).** Start on sqlite; migrate *everything*
+   (collections, memories+embeddings, ACL, sessions, session_events,
+   session_messages, tier_events, tools) into pgvector; then **switch the
+   primary backend** by changing the serve flag to `--backend pgvector`.
+   "Switch primary backend" is operational, no extra code: run the full
+   migrate, then change `--backend`.
+2. **Way back.** Same command with `--from`/`--to` swapped.
+3. **Single memory container / additional partial backend.** Copy only
+   specific collection(s) with `--collection`, so e.g. pgvector holds just
+   certain memories while sqlite keeps the rest. The two backends coexist,
+   each holding different data.
+4. **Additive top-up.** Re-running, or copying into a populated target,
+   merges idempotently (natural-key dedupe).
+
+**Selection model — two orthogonal selectors:**
+
+- `--include <csv>` of sections: `memory,sessions,tier-events,tools`.
+  Default (unset) = **all four** (full dataset). The `memory` section
+  carries collections + their memories + their ACL rows.
+- `--collection <name>` (repeatable) restricts the `memory` section to
+  those collections only. When `--collection` is given and `--include` is
+  unset, the default narrows to **memory-only** (the single-container
+  case).
+
+So: full switch = no selectors; single container = `--collection notes`;
+logs-only = `--include sessions,tier-events`.
+
+**Copy phases (FK-respecting; each gated by the selection):**
+
+| Section | Phase | Export | Import |
+| --- | --- | --- | --- |
+| memory | collections | `ListCollections` (filtered) | `PutCollection` |
+| memory | memories | `ListMemories` (per coll, keyset-paged) | `PutMemory` |
+| memory | session_acl | `ListSessionACLs` | `SetSessionACL` |
+| sessions | sessions | `ListSessions(+deleted)` | `UpsertSession` |
+| sessions | session_events | `ListSessionEvents`/sess | `PutSessionEvent` |
+| sessions | session_messages | `ListSessionMessages`/sess | `UpsertSessionMessages` |
+| tier-events | tier_events | `ListTierEvents` (batched) | `PutTierEvent` |
+| tools | tools | `ListTools` | `UpsertTool` |
+
+Both backends are the same concrete `*Store` implementing every feature
+interface, so migrate opens each side as `storepkg.Store`, type-asserts
+the optional `MigrationStore`, and copies a section only if **both** sides
+implement it. Per-phase counts (scanned / inserted / skipped) print at the
+end; `--dry-run` reads sources and reports would-copy counts, no writes.
+
+**Idempotency / dedup.** Memories dedupe on `(collection_name,
+content_hash)`. The append-only audit logs (`session_events`,
+`tier_events`) have **no natural key** (autoinc id only), so their
+idempotent import dedups on an **insert-if-no-identical-row** check (all
+business columns match) — re-running produces no duplicates. A second full
+run reports 0 inserts everywhere.
+
+**Two backend-specific wrinkles (both resolved):**
+
+- **sqlite-vec embeddings are an opaque BLOB.** vec0 `memory_vecs` stores
+  the embedding as a blob and the Go binding ships `SerializeFloat32` but
+  no deserialize. The on-disk format is plain little-endian `float32`
+  (`binary.Write(buf, LittleEndian, vector)`), so `deserializeFloat32`
+  reverses it with `binary.Read` at the locked dim — **byte-faithful, no
+  re-embedding**. pgvector exports via `pgvector.Vector.Scan` + `.Slice()`.
+- **The public `Store()` write path enforces ACL** (a global collection is
+  `r`-for-all → `ErrWriteDenied`), so migrate cannot reuse it for imports.
+  The new `MigrationStore.PutMemory` is **ACL-free and timestamp-
+  preserving**.
+
+**Fidelity contract (documented caveat).** Preserved exactly: collection
+`created_at`, memory `ts`+`content`+embedding (byte-identical), session
+`created_at`/`deleted_at`, message `created_at`/`finished_at`, event
+`occurred_at`/`ts`, all ACL modes. Rewritten to migrate-time (reused
+upserts stamp `now()`): `sessions.updated_at`, `tools.created_at`/
+`updated_at` — acceptable "last-written" fields (tools are also re-seeded
+at startup). A byte-identical `PutSession`/`PutTool` is deferred.
+
+The new `MigrationStore` optional interface (in `store.go`, mirroring the
+`SchemaInspector`/`SessionStore` convention) is exercised by the
+`storetest` conformance harness on **both** backends — the round-trip
+asserts the exported embedding is byte-identical, validating the sqlite LE
+deserialize and the pgvector scan together. Verified end-to-end across a
+`sqlite→pgvector→sqlite` hop: recall on the round-tripped file returns
+`distance 0` for the exact source vector.
+
+**Deferred to F3 M11+:** continuous-sync / daemon mode (migrate is
+one-shot CLI only), cross-host federation (explicit F3 non-goal),
+per-session selective log copy (selective granularity is per-collection
+for memory + section toggles for the rest), and the byte-identical
+`updated_at`/tool-timestamp imports noted above.
+
+## Open questions
+
+- **Do we ship the server inside the opencoti binary or alongside it?**
+  Alongside (separate binary) is cleaner; explore a single-fat-binary
+  option as an opt-in.
+- **Authentication.** Local UDS owner is the user; over loopback TCP
+  we need a shared secret. Default to UDS.
+- **claude-hooks coexistence.** If a user runs both Claude Code (with
+  claude-hooks) and opencoti (with opencoti-server) on the same host,
+  the two memories should *not* collide. They use different stores
+  and different schemas by default; a separate migration tool can
+  bridge them if the user wants.
+
+## Risks
+
+- Two persistence backends doubles the test matrix. Mitigation: a
+  storage trait + a shared conformance test.
+- Drift from claude-hooks features. Mitigation: don't try to replicate
+  feature-for-feature; replicate goals. claude-hooks remains the
+  reference for Claude Code; opencoti-server can diverge where it
+  makes opencoti better.

docs/features/poly_kv.md

@@ -0,0 +1,817 @@
+# F5 M6 — PolyKV (shared compressed KV pool)
+
+Child plan for the PolyKV milestone. Parent: [advanced_kv.md](advanced_kv.md) §M6.
+Composed **atop M7 Rolling KV** ([rolling_kv.md](rolling_kv.md)). Status: **M6 SHIPPED (2026-06-06)** —
+S0 asymmetric q8/q4 (#369, banked correct at 4k; large-ctx GPU-fast asymmetric folds into the S2
+kernel), S1 SharedKVPool (#370, gate PASS on Gemma-4 iSWA + 6.9× fan-out memory win), and S2
+TurboQuant tier family + InnerQ + Level-A materialize + Level-B fused FA-VEC (#371/#374–396, 4/4
+tiers logit-equiv PASS, decode ceiling broken) are all done, composed on the full F5 stack (S3 glue
+gate), and **captured as patches `0072`/`0073`** (byte-identical proof). The remaining milestone is
+**M6-S4** speculative MTP draft head (`gemma4_assistant`, #373) — a separate follow-on.
+
+## What PolyKV is (reference)
+
+Reference: arXiv 2604.24971, MIT — [github.com/ishan1410/PolyKV](https://github.com/ishan1410/PolyKV).
+A document is prefilled **once**; its KV is extracted, **asymmetrically compressed**, and held
+**once** in a host-resident `SharedKVPool`. N concurrent agents share that single compressed blob
+(no per-agent dense copy of the shared prefix) → **O(1) memory in agent count** for the shared
+context. Reference numbers: **2.91× compression**, 15 agents 19.8 GB → 0.45 GB, +1.59% perplexity
+@2k (→ +0.57% @4k).
+
+**Asymmetric scheme** (the core idea — keys and values quantized differently because key errors
+amplify exponentially through softmax while value errors scale linearly):
+- **K → int8 (q8_0)** — higher precision.
+- **V → 3-bit TurboQuant MSE** — FWHT (fast Walsh-Hadamard) rotation to spread outlier energy
+  across dims, then Lloyd-Max (MSE-optimal) quantization.
+
+The reference is **HuggingFace Transformers / PyTorch**, framework-agnostic w.r.t. serving stacks
+(no vLLM/SGLang dep). Crucially it does **decompress-then-attend** — each agent decompresses the
+pool into a dense `DynamicCache` before attention, so its savings are at the **storage** layer, not
+**in-attention**. No custom attention kernel.
+
+## The opencoti angle — why M7 makes this better than the reference
+
+M7 already ships **dequant-on-lift** (Stage 3d, `to_fp16_nc`): the streaming/window forward reads
+host-resident KV and dequantizes it to a device f16 slot **per tile, inside the attention read
+path**. That is exactly the "dequant-on-read" the PyTorch reference lacks. So opencoti can run
+PolyKV's compression **in-attention** — the compressed pool never has to be densified, getting the
+memory win at the attention layer, not just storage. The composition is already designed
+(rolling_kv.md): *"M7 streams the compressed tiles, decompressing in-slot before FA; the auto-adapt
+loop becomes `tile_bytes_max × compression_ratio`."*
+
+Building blocks already in place: host-resident KV (M7-A pinned pool), quant-aware dequant-on-lift
+(S3d), independent K/V cache types (`-ctk`/`-ctv`), session-keyed KV-reuse prefix (M0), the
+iSWA two-cache split (shared-prefix scaffolding).
+
+## Scope decision (from advanced_kv §M6)
+
+**Single-tenant correctness first; multi-tenant memory bench follows when Tier-0 fan-out lands.**
+PolyKV degenerates cleanly to a 1-tenant pool (= aggressive asymmetric KV-compression of one
+sequence), so single-tenant is a real, shippable first milestone. The user directive is to
+**validate shared-KV + rolling-KV functionality + performance together**.
+
+## Staging — fast-first (bootstrap with stock types, then the hard kernel)
+
+### S0 — Bootstrap: asymmetric KV compression via existing ggml types (ZERO new kernel)
+Use llama.cpp's independent `-ctk q8_0 -ctv q4_0` (both existing KV types, both FA-supported, both
+dequant-on-lift-able). This **is** the PolyKV K/V asymmetry (K precise, V aggressive) with stock
+types, routed through M7's existing host-residency + dequant-on-lift. Proves the
+asymmetric-compressed-KV + M7 composition architecture end-to-end with no new code.
+- Compression: q8_0 K (0.53×) + q4_0 V (0.28× vs f16) ≈ **~2.4× KV** overall.
+- **Feasibility: PROVEN 2026-06-04.** `-ctk q8_0 -ctv q4_0` boots, attends, and recalls the needle
+  on Gemma head_dim 512 (q4_0 V dequants to f16 in-path; coherent output). KV 186.9 MiB (q4_0 V) vs
+  244.4 (q8_0 V) — asymmetric compression confirmed. The head_dim-512 q4_0-V risk is retired.
+- **Correctness gate (RUNNING 2026-06-05, dir `baselines/20260604-2346`):** RULER vt+niah, asymmetric
+  ours (`KVTYPE_OVERRIDE=q8_0 CTV_OVERRIDE=q4_0 FLASH_ATTN=on BINARIES=ours`) vs cached controls.
+  `FLASH_ATTN=on` is **mandatory** — at head_dim 512 the auto-resolver disables FA for q4_0 V (no
+  native kernel), producing empty cells; forcing it on keeps FA + dequant-on-lift engaged.
+  - **The control matrix isolates V-quant exactly** (q8/q8 → q8/q4 is a V-only change):
+
+    | task | ctx | vanilla f16 | vanilla q8/q8 | ours q8/q8 | **ours q8/q4 (S0)** |
+    |------|-----|-----|-----|-----|-----|
+    | vt | 4096 | 100 | 100 | 100 | **80** |
+    | vt | 32768 | 100 | 80* | 96 | 0† |
+    | niah_single_1 | 4096 | 100 | 100 | 100 | **100** |
+    | niah_single_1 | 32768 | 100 | 100 | 100 | n/a† |
+
+    \*vanilla q8/q8 vt@32k=80 is a known vanilla-specific prompt fragility (ours q8/q8 there = 96).
+    †**ours q8/q4 at 32k is a CPU-FA timeout, not a quality result** (re-confirmed 2026-06-18 on the
+    current post-upstream-align binary, sha `5e1b3a53`): q4_0-V has no GPU FA kernel at head_dim 512,
+    so resident-mode prefill runs on CPU-FA (~22 tok/s, ~1440s for 32k) and exceeds the harness's
+    1200s/sample budget — vt 5/5 ReadTimeout, niah the 2 naturally-completing samples both ReadTimeout
+    @1200s (the other 3 were ConnectionError from the reap, not data). This is the **S0 WALL below**,
+    not a retrieval failure. niah@4k=100 (0/5 empty) is the decisive correctness signal and it holds.
+  - **vt@4096 = 80**: both f16 and q8/q8 controls hold 100, so the 20pp drop isolates to V f16→q4_0 —
+    **one flipped sample at NS=5** ("direction real, magnitude not robust" per the baseline's own
+    NS-sizing note). This is the textbook PolyKV signature: aggressive V-quant nicks hard multi-hop
+    (vt) while single-needle (niah) should stay robust — and the exact motivation for TBQ3_0 (S2,
+    MSE-optimal) over naive q4_0. **niah@4k = 100 (re-confirmed 2026-06-18) is the decisive correctness
+    signal — q4_0-V keeps the single needle, and the upstream-align rename did not regress plain gemma4
+    decode**; niah@32k could not be measured on naive q4_0-V (CPU-FA timeout — see † and the S0 WALL).
+  - **Perf note (resident regime):** at 4k/32k the KV is FULLY RESIDENT (no host tail, POSITION_WINDOW=0)
+    so M7's dequant-on-lift does **not** engage; q8/q4 pays stock llama.cpp's head_dim-512 dequant tax
+    (MMA kernel is f16-only → full-KV dequant before FA, every batch) → ~3–4× slower prefill than f16.
+    **The performance win lives in the OVERFLOW regime** (256k cell), where M7 streams compressed tiles
+    and amortizes dequant per-tile in-attention. **Attempted 2026-06-18** (`m6-s0-overflow-tps.sh`, current
+    binary `5e1b3a53`): the 256k composition cell is NOT obtainable as milestone evidence on the current
+    binary, for two independent reasons — (1) the harness's raw `/completion` prompt at 256k greedily emits
+    an immediate end-token (RES_q8q8: GPU-fast prefill 222k tok @1122 tok/s, but `eval = 1 token`,
+    needle=NO), so decode-tps is unmeasurable without a generation-capable 256k prompt; and (2) the q8/q4
+    *overflow* win is gated on the **deferred q4_0-V GPU overflow kernel** (S2 TBQ3_0) — the same head_dim-512
+    gap as the resident wall. The q8/q8 overflow path itself is already proven GPU-fast at 256k (M7 ship,
+    bug-226: vt=96/niah=100). So the rolling-KV + *asymmetric* milestone evidence **awaits the S2 kernel**
+    (and a generation-capable 256k harness), consistent with the finalized S0 verdict below. (The perf script
+    was hardened this session: `--max-time 900` on the completion curl + the resident-q4_0-V@256k config
+    skipped, since it is the CPU-FA multi-hour path — see [[bug-564]].)
+  - Pass = retrieval holds (≥ vanilla − 2pp on the robust niah signal; vt characterized at NS=5).
+
+#### S0 WALL (2026-06-05): stock q4_0-V has NO fast GPU FA path at head_dim 512
+The correctness gate surfaced a hard performance wall, not a quality result:
+- **vt@4096** = 80, **316s/5 = 63s/sample, 0/5 empty** — fast, valid (real q4_0-V quality at 4k).
+- **vt@32768** = 0 is a **TIMEOUT ARTIFACT, not a collapse**: `elapsed_secs=6001 = exactly 5×1200s`
+  (the ruler_runner `--http-timeout 1200`); server log shows `srv stop: cancel task` +
+  `stop processing: n_tokens=…, truncated=0` for every sample — **cancelled mid-prefill, never
+  decoded** (no error/NaN/abort). The 32k q8/q4 prefill does **not finish in 20 min**.
+- **GPU utilization ~5% during prefill** (14 GB used = weights resident) ⇒ the quant-V attention runs
+  on **CPU** (no GPU FA kernel for q4_0 V at head_dim 512). ~50× slower than f16. niah@32k (same slow
+  prefill) is equally doomed → killed to stop wasting GPU.
+- **Consequence:** at RESIDENT (4k/32k/256k all fit compressed), stock FA handles quant-V on CPU →
+  prohibitively slow at large ctx; the perf premise (smaller tail → higher tps) is swamped by the
+  CPU-FA tax, and a 256k RULER/needle cell is infeasible (hours/timeout).
+- **Salvage hypothesis (to test):** M7's dequant-on-lift (S3d, `to_fp16_nc`) dequants q4_0→f16 **on
+  GPU per-tile**, but ONLY engages when the **window** is active (overflow). Forcing
+  `--kv-residency-mode window` / a tight `--vram-target` may route q4_0-V through the **GPU**
+  dequant-on-lift — potentially *faster* than the resident CPU-FA path, which would make the
+  composition both correct AND the perf win. Diagnostic: 32k forced-window q8/q4, compare prefill
+  wall-time + GPU-util vs resident. If GPU-fast → composition viable; if still CPU/slow → naive q4_0-V
+  is a dead end on Gemma-4 head_dim 512, and S0 needs V=q8_0 (memory ~1.9× not 2.4×) or S2's TBQ3_0
+  must ship a fused GPU dequant-FA kernel (raising S2 scope/risk).
+
+#### S0 SALVAGE RESULT (2026-06-05): forced-window q4_0-V CRASHES → both paths blocked
+Ran `.opencoti/m6-s0-window-gpupath-probe.sh` (32k, `--kv-residency-mode window --vram-target 1700`,
+q8/q4). The window **engaged** correctly (POSITION_WINDOW=5, window 20480 / host tail 20480 cells)
+but then **aborted at `fattn.cu:509: GGML_ABORT("fatal error")`** — the `BEST_FATTN_KERNEL_NONE`
+branch: `ggml_cuda_get_best_fattn_kernel` returns NONE for q4_0-V at head_dim 512 (MMA is f16-only;
+VEC needs head_dim ≤256; WMMA excluded at 512; TILE didn't apply to this tile shape). **S3d's
+dequant-on-lift was validated only with q8_0** (bug-226 RULER vt@262144); q4_0-V tiles reach the
+kernel selector still-quantized → NONE → abort. So:
+- **Resident q8/q4:** stock TILE-kernel fallback, ~50× slower than f16, RULER-timeout at ≥32k.
+- **Window q8/q4 (M7 GPU path):** crashes at `fattn.cu:509` (q4_0-V not handled by the streaming
+  kernel selector / dequant-on-lift).
+**Verdict:** the "zero-new-kernel" naive-q4_0-V bootstrap is **perf-blocked on Gemma-4 head_dim 512**
+— correct at 4k (niah=100, vt=80) but cannot be made GPU-fast at scale without kernel work. Note:
+rolling-KV itself is already proven GPU-fast at 256k with **q8_0** (M7 ship, bug-226: vt=96/niah=100).
+The blocked piece is specifically the **asymmetric q4_0/3-bit V**. FORK (user decision pending):
+(A) fix the streaming kernel/dequant-on-lift for q4_0-V (DSO debug + rebuild) → proves asymmetric
+composition GPU-fast; (B) descope S0 to V=q8_0 (validated, GPU-fast; ~1.9× not 2.4×; loses asymmetry);
+(C) skip to S2 TBQ3_0 with a proper fused GPU dequant-FA kernel (the real PolyKV V-quant);
+(D) bank S0 correctness (4k proven) + proceed to **S1 SharedKVPool** (the "shared KV" core, host-side,
+orthogonal to the V-quant kernel gap; delivers an O(1)-in-agents memory-perf story independent of it).
+
+**DECISION (2026-06-05, user):** q4_0-V GPU kernel **DEFERRED** ("we'll see later about the q4_0
+kernel") — it folds into S2 TBQ3_0, which needs a custom fused dequant-FA kernel regardless.
+**Proceeding to S1 SharedKVPool now** (path D). S0 banked as: asymmetric architecture correct at 4k
+(niah=100, vt=80); large-ctx GPU-fast asymmetric compression awaits the S2 kernel. S1 uses the
+validated q8_0 KV (no kernel gap) and demonstrates the headline O(1)-in-agents sharing win.
+
+### S1 — SharedKVPool: shared read-only prefix across sequences (the multi-tenant core) ✅ DONE (2026-06-05)
+N slots/seq_ids attend ONE physical copy of a common document prefix + a per-slot private suffix →
+O(1) prefix memory in agent count. Builds on M0 (prefix reuse), M7 (host residency), iSWA
+(two-cache). Single agent degenerates to S0.
+
+**Design (minimal-additive, rides native unified-mode sharing).** llama.cpp already lets one KV
+cell belong to many `seq_id`s (`cells.seq[i]` is a `bitset<LLAMA_MAX_SEQ>`). In **unified mode**
+(`--kv-unified`, `n_stream=1`) all slots live on stream 0, so `llama_memory_seq_cp(pool, slot, 0,
+P)` takes the same-stream **zero-copy** branch (`cells.seq_add` — just OR the destination bit into
+the existing prefix cells; no buffer copy, no new allocation). Attention isolation is automatic:
+`set_input_kq_mask_impl` masks a cell for a seq unless `cells.seq_has(j, seq_id)`, so shared-prefix
+cells (both bits) are attended by both agents while private-suffix cells (one bit) stay isolated.
+S1 is therefore a tagged, **default-off** server hook (`shared_pool_slot`/`shared_prefix_n_tokens`
+request params; gated prefill branch that `seq_cp`'s the pool's `[0,P)` prefix into the sharer's seq
+and sets `n_past=P` to skip re-processing it) — **no new storage**. Counted-once is *structural*
+(one boot-fixed KV buffer, prefix cells bit-shared), proven by cell occupancy, not a MiB delta.
+
+**RUNTIME PREREQUISITE — `--no-clear-idle` (MANDATORY).** Upstream `clear_idle` defaults **on**
+(with default `cache_ram_mib=8192`), so `--kv-unified` alone auto-enables the idle-slot prompt-cache
+offload (PR #16391): when the pool slot goes idle, the next task launch serializes its KV to a host
+blob **and frees the device cells**, evicting the prefix before any sharer's `seq_cp` runs → silent
+prefix miss (suffix/isolation still work; only the shared prefix is lost). Booting `--no-clear-idle`
+keeps the pool slot device-resident. The hook also self-guards: if `seq_pos_max(pool) < P-1` it
+warns and falls through to a normal full reprocess (correct, just slower) instead of sharing nothing.
+Root cause: [[bug-367]] — it masqueraded as a cross-seq attention regression; the seq_cp/mask
+machinery was never at fault.
+
+**Gate (`.opencoti/m6-s1-shared-pool-gate.sh`) — PASS on Gemma-4 A4B iSWA, q8/q8, `--parallel 3
+--kv-unified --no-clear-idle`:** pool prefills an 874-token doc with needle `QUASAR-4417` on slot 0;
+agents on slots 1/2 POST `shared_pool_slot:0` + a distinct private officer (Alice/Bob). Results:
+both agents recall the needle (shared); each recalls only its own officer (isolation, R1 crux);
+`cache_n=874≈P` & `prompt_n=39≈suffix` per agent; Σ prompt_n `{pool,A,B}` = 953 ≈ `P+sufA+sufB`
+(950), **not** `3·P` (2622) → prefix enters KV exactly once; pool slot intact after both sharers;
+`shared_pool_slot:-1` default-off = full reprocess (`cache_n=0`) and still correct. iSWA two-cache
+`seq_cp` delegation (the R2 risk) confirmed across the 5-layer SWA + 25-layer global caches.
+Also verified on the non-iSWA path (Qwen2.5-1.5B-instruct).
+
+**Fan-out MEMORY win — measured (`.opencoti/m6-s1-fanout-mem.sh`, Qwen, N=8, fixed 8192-cell unified
+buffer).** Physical KV-cell occupancy (read from the `find_slot ... used =` debug counter) for the
+SAME 8 agents over an 874-token shared doc: **naive (sharing off) = 7183 cells** (≈ 8×P, one full
+copy per seq) vs **shared (S1) = 1041 cells** (874 prefix counted ONCE + 167 cells for the 7 private
+suffixes ≈ 24 cells/agent). That's a **6.9× reduction at N=8**, and projected onto the fixed buffer:
+the naive path fits **~9 agents** before exhaustion while S1 fits **~306** — the O(1)-prefix headline
+(prefix amortized once; per-agent cost is just the private suffix). This is the *memory* counterpart
+to the gate's *compute* counted-once, so S1 is validated on both axes.
+
+### S2 — TurboQuant tier family (TBQ8_0 / TBQ4_0 / TBQ3_0): the MSE-optimal KV-quant ladder (HIGH RISK)
+TurboQuant is **one** scheme — FWHT (fast Walsh-Hadamard) rotation to spread outlier energy across
+dims + Lloyd-Max (MSE-optimal) codebook — *parameterized by bit-width*. S2 ships the **full tier
+ladder**, not a one-off 3-bit kernel: new ggml types **TBQ8_0, TBQ4_0, TBQ3_0** that share the same
+rotation + codebook machinery, each with its own dequant-on-lift entry. This lets the asymmetric
+scheme mix tiers freely (e.g. K→TBQ8_0, V→TBQ3_0) and supplies MSE-optimal replacements for the
+naive q8_0/q4_0 KV used in S0/S1. **TBQ3_0 is the headline** (replaces V=q4_0 → 3-bit at reference
+quality, +1.59% ppl @2k); **TBQ8_0/TBQ4_0** are the precision/mid rungs *and* the lower-risk
+bring-up path — validate the shared rotation+codebook at 8-bit (near-lossless, easy cosine gate)
+first, then descend to 4-bit and 3-bit. Highest-risk milestone (new quant kernels + dequant-on-lift
+entries + ggml type registration + `GGML_TYPE_COUNT` bump + CUDA dequant).
+
+**Adoption source (2026-06-05): the AtomicBot-ai TurboQuant fork, not a from-scratch kernel.**
+[`AtomicBot-ai/atomic-llama-cpp-turboquant`](https://github.com/AtomicBot-ai/atomic-llama-cpp-turboquant)
+(MIT, fork of `TheTom/llama-cpp-turboquant`) already ships a working, multi-platform (CUDA + Metal +
+CPU) TurboQuant family: runtime KV types **`turbo2`/`turbo3`/`turbo4`** → `TURBO2_0`/`TURBO3_0`/
+`TURBO4_0` (their wording: **"WHT + PolarQuant"**; `turbo3` = 3-bit ~4.3×, the recommended default,
+with a Metal **"TurboFlash"** flash-attention decode kernel) selected via `-ctk/-ctv`; plus
+quantize-time **weight** types `TQ3_1S`/`TQ4_1S` ("WHT-rotated Lloyd-Max", `block_size=32`) chosen as
+`llama-quantize --type` with masks under `scripts/quantize-masks/`. **S2 becomes a study-and-adapt of
+that fork** rather than a clean-room build: port its WHT rotation + low-bit codebook + the
+CUDA/Metal dequant/`TurboFlash` decode path into our vendored `llama.cpp`, mapped onto our
+dequant-on-lift (S3d) machinery and ggml-type registration. Reconcile naming on the way in (our
+`TBQ3_0` ≡ their `TURBO3_0`/`turbo3`); decide whether to keep their `turbo*` spelling for CLI
+compatibility with the fork's `-ctk turbo3` (and the draft `-ctkd/-ctvd turbo3`) so the published
+Gemma-4 GGUFs run unmodified. **Caveat:** the fork's base differs from our vendored tree + we build
+under **cosmocc** (no CMake/CUDA-nvcc in the host path; CUDA lives in the DSO) — this is a *port*,
+not a cherry-pick, and pairs with S4 (both adopted from the same fork, both use `turbo3` KV).
+- **Gate:** cosine/RULER holds at **each** tier (TBQ8_0 near-lossless, TBQ4_0 ≈ q4_0-or-better,
+  TBQ3_0 at reference); TBQ3_0 V compression → ~2.9× (matches reference 2.91×); the fork's published
+  `-ctk turbo3` Gemma-4 path reproduces "no measurable quality regression". DSO rebuild.
+
+#### S2 InnerQ — per-channel equalization, PER-LAYER head_dim ✅ DONE + ACCEPTED (2026-06-05)
+turbo3 as a **K**-quant (not just V) needs the missing correctness ingredient: **InnerQ** per-channel
+K-variance equalization *before* the WHT (`x[ch] *= scale[ch]` at K-write; dequant `×scale_inv[ch]`
+after the inverse WHT). Without it turbo3-K is ~0.983 cosine — too lossy for softmax (K≫V asymmetry).
+opencoti uses **calibrate-then-activate-from-token-0**; `scale_inv` crosses the host↔DSO boundary as a
+**file artifact** (the turbo3 forward is GPU-only in the DSO; our active dequant is the CPU `to_float`
+on the lift path — both load the file at boot).
+
+**The mixed-head_dim problem (Gemma-4).** `ne00` of the K set-rows view is `n_embd_gqa = head_dim ×
+n_head_kv`, **not** head_dim — and head_dim is NOT derivable from `ne00` alone. Gemma-4 mixes head_dim
+**per layer**: SWA layers `head_dim 256` (`n_head_kv 8` → `n_embd_gqa 2048`) and global layers
+`head_dim 512` (`n_head_kv 2` → `1024`), ~5:1. A single head_dim env mis-folds the minority layer-type
+and contaminates the shared `scale_inv`. **Fix = per-WIDTH slots** keyed by `n_embd_gqa`: the host
+supplies `TURBO_INNERQ_HEADDIM_MAP="2048:256,1024:512"`; the DSO carries per-slot
+`scale/scale_inv/sq_accum[MAX_SLOTS=4][512]` + `slot_hd[]`, the dispatch maps `ne00→slot` and passes it
+as a **kernel arg** (no per-launch H2D), the kernel folds `ch=(i_grp*128+j) % head_dim[slot]` into that
+slot's accumulator (no cross-width contamination). Finalize writes a **v2 file** (`TIQ2`: `nslots`, per
+slot `{n_embd_gqa, head_dim, float[head_dim]}`); v1 (`TIQ1`) still loads as one wildcard slot. The CPU
+dequant selects its slot by `k == n_embd_gqa`. Empty map ⇒ one wildcard slot (uniform models, e.g. Qwen
+head_dim 128 → identity fold `ch==j`, byte-compatible). Files: `ggml-cuda/set-rows.cu`,
+`ggml-cuda/turbo-innerq.cuh`, `ggml-turbo-quant.c`; gate `.opencoti/s2-innerq-gate.sh` (+`HEADDIM_MAP`).
+
+**Fused-path invariant (S2-next).** InnerQ's forward-equalize (`×scale` at K-write) is **path-shared**;
+the un-equalize is the only path-specific half and lives **only** in the lift's `to_float` (`×scale_inv`).
+The fused kernel keeps K equalized and moves the correction to a **Q pre-scale** (`Q[c]*=scale_inv[c]`,
+`Q_pre·K_eq = Q·K`), reusing the **same per-width table** — so it must NOT route through `to_float`
+(no double-correction). Per-layer InnerQ is therefore a prerequisite the fused path consumes wholesale.
+
+**Acceptance (format-robust, NEVER greedy).** A greedy short-coherence probe is invalid here — the
+`...-it` coder model is thinking-tuned (`<|channel>thought`) and degenerates on raw short prompts *even
+at f16*. The valid gate is **logit-equivalence vs f16** (`.opencoti/s2-logit-equiv-gate.sh` over the
+#355 harness). **VERDICT PASS** on Gemma-4 A4B, turbo3/turbo3 + per-layer InnerQ vs f16/f16, 12 needle
+prompts: `real_frac=0.0` (≤0.15), answer-scoped `recall_mismatch=0` (12/12 both recall), all 10
+divergences `post_answer`-benign, `mean_tv=0.0102`/`p95=0.0255` (catastrophe 0.30). turbo3-K +
+per-layer InnerQ is distributionally faithful to f16. Validated non-regressive on Qwen3-4B (uniform
+head_dim 128, identity fold). (#379)
+
+#### S2 tier-family overview — quality × speed × memory, all 7 KV modes ✅ (2026-06-05)
+
+Full sweep over `{f16, q8_0, q4_0, turbo8, turbo4, turbo3, turbo2}` on one rig (Gemma-4 A4B
+Q4_K_M, RTX 3090, 6025-tok prompt, 256-tok forced decode `ignore_eos`, ctx 8192, single stream).
+Harness: `.opencoti/s2-tier-overview.sh` + `s2-overview-assemble.py`. Quality = logit-equiv vs f16
+(#355, n=12 needle); turbo verdicts from the tier-family gate, q8/q4 captured in the same pass.
+
+```
+mode    | verdict  real rec  meanTV  p95TV |  pre tps  dec tps  util% |  KV MiB    bpv   vsF16
+--------------------------------------------------------------------------------------------
+f16     | REF     0.000   0  0.0000 0.0000 |   2751     73.2     97  |   460.0  2.000  1.00x
+q8_0    | PASS    0.000   0  0.0122 0.0785 |   2610     65.7     94  |   244.4  1.062  0.53x
+q4_0    | PASS    0.000   0  0.0094 0.0007 |   2654     64.9     93  |   129.4  0.562  0.28x
+turbo8  | PASS    0.000   0  0.0098 0.0157 |    994      2.7     54  |   233.6  1.016  0.51x
+turbo4  | PASS    0.000   0  0.0152 0.0768 |   1069      3.2     58  |   122.2  0.531  0.27x
+turbo3  | PASS    0.000   0  0.0075 0.0090 |   1003      3.1     53  |    89.8  0.391  0.20x
+turbo2  | PASS    0.000   0  0.0228 0.1304 |   1129      3.3     58  |    61.1  0.266  0.13x
+```
+
+- **Quality:** all 6 quant modes PASS (real_frac 0, recall_mismatch 0). turbo2's p95 TV 0.13 is the
+  loosest but far under the 0.30 catastrophe line. Quality is **not** the differentiator — the whole
+  ladder is usable.
+- **Memory (the win):** 460→61 MiB, a 7.5× span. turbo8 (234) and turbo4 (122) **Pareto-beat** q8_0
+  (244)/q4_0 (129) at matched bit-width — turbo carries one `norm`/128 vals (1.6% overhead) vs stock's
+  scale/32 (6.25%). turbo3 (90) and turbo2 (61) reach footprints stock has no q3/q2 KV type for.
+- **Speed (the catch — RESOLVED 2026-06-06 by the perf-path below):** in this validation-path snapshot,
+  native quants decoded 65–73 tps / 93–97% util while **turbo decoded 2.7–3.3 tps / 53–58% util (~25×
+  slower).** The fingerprint: prefill dropped ~2.7× but decode collapsed ~25× — the CPU dequant-on-lift
+  reconstructed the WHOLE KV cache on the host every step (prefill amortizes over the query batch, decode
+  pays per single token). Root cause: the `ggml_cast(turboN→f16)` lift emitted a `GGML_OP_CPY` that
+  CUDA's CPY `supports_op` rejected (no turbo src), so the scheduler spilled it to the CPU backend →
+  whole compressed cache D2H + host dequant + H2D every token. **Fixed by the GPU-resident dequant-on-lift
+  (Level A, next subsection).** (#371)
+
+#### S2 perf-path Level A — GPU-resident InnerQ-correct turbo dequant-on-lift ✅ DONE + GATED (2026-06-06)
+
+> **⚠ SUPERSEDED — historical.** The ~41 tps plateau here is the whole-cache `ggml_cast(turboN→F16)`
+> materialize path; it is broken open by **Level B** (in-register fused FA-VEC, next subsection) and
+> again by the **WS1/WS2 in-register turbo-V** work (context.md §5x). The current turbo decode story
+> lives there — treat these Level-A numbers as a milestone record, not the shipping figures.
+
+The "Speed (the catch)" spill above is fixed. The lift stays a graph `ggml_cast(turboN→F16)` but now
+executes **entirely on CUDA**: a new block-cooperative kernel does the inverse WHT **once** per
+128-element group in shared memory (vs the per-output-pair recompute in `turbo-dequant.cuh`), applies
+InnerQ `×scale_inv[ch]` in true-channel space, and writes f16 in VRAM — stock flash-attention then runs
+unchanged. No host round-trip. Materialize-f16-then-stock-FA is preserved by design, so the ceiling is
+the f16 path (Level B = fused FA-VEC reading turbo in-register, a later milestone).
+
+**Edit surface (4 hook sites + 3 new files, all `// opencoti-hook: turboquant perf-lift`; 10 markers):**
+- `ggml/src/ggml-cuda/turbo-cpy.{cu,cuh}` (NEW) — `cpy_turbo{2,3,4,8}_f16` kernels + `ggml_cpy_turbo_f16_cuda`
+  launcher (WHT-once cooperative inverse + InnerQ un-equalize + strided f16 write).
+- `ggml/src/ggml-cuda/turbo-innerq-dev.cuh` (NEW) — cross-TU bridge declaring
+  `turbo_innerq_active_device_scale(ne00, &d_scale_inv, &head_dim)`.
+- `ggml-cuda/set-rows.cu` — implements that accessor (owns the InnerQ device symbols; selects slot via
+  the existing `innerq_slot_for((int)ne00)`, returns a `cudaGetSymbolAddress` pointer + `head_dim`).
+- `ggml-cuda/ggml-cuda.cu` — CPY `supports_op` now accepts `turbo{2,3,4,8}_0 → F16`.
+- `ggml-cuda/cpy.cu` — `ggml_cuda_cpy` dispatches turbo→F16 to the launcher (InnerQ pointer from the bridge,
+  or `nullptr` ⇒ identity).
+- `src/llama-graph.cpp` — collapses the old two-step `turbo→F32→F16` cast to a single `turbo→F16` (removes
+  the f32 intermediate whose `turbo→F32` leg was the CPU-spilling op).
+
+**Correctness model (R2, resolved empirically):** the GPU lift is a faithful numeric replica of the CPU
+dequant — same graph tensor, same `ne00`, same `innerq_slot_for`/`head_dim` table, same inverse-WHT order
+(signs2 → butterfly → 1/√128·signs1) — so it makes the *identical* InnerQ decision as the CPU lift it
+replaces. The gate passes iff the CPU gate passed (it does). No K/V flag needed.
+
+**Build:** `build:llamafile:cuda` (DSO, ~24min; restamp `/root/.llamafile/v/0.10.1/ggml-cuda.so` **and**
+`/root/ggml-cuda.so` byte-identical, bug-224) + `build:llamafile:make` (host, llama-graph.cpp). Scheduler
+proof: `GGML_SCHED_DEBUG=1` shows `graph splits = 2` (no CPU-spill fragmentation; the per-node dump is
+compiled out under `-DNDEBUG`).
+
+**Gates (both GREEN, RTX 3090, Gemma-4 A4B Q4_K_M, 6025-tok prompt, 256-tok forced decode `ignore_eos`):**
+
+```
+mode    | verdict real rec | dec tps  was   speedup  util%  was |  KV MiB  (unchanged)
+-------------------------------------------------------------------------------------
+f16     | REF    0.00   0  |  73.8    73.2    1.0x     95    97  |  460.0
+q8_0    | PASS   0.00   0  |  65.9    65.7    1.0x     95    94  |  244.4
+q4_0    | PASS   0.00   0  |  65.9    64.9    1.0x     96    93  |  129.4
+turbo8  | PASS   0.00   0  |  42.5     2.7   15.7x     96    54  |  233.6
+turbo4  | PASS   0.00   0  |  40.6     3.2   12.7x     98    58  |  122.2
+turbo3  | PASS   0.00   0  |  41.6     3.1   13.4x     96    53  |   89.8
+turbo2  | PASS   0.00   0  |  42.2     3.3   12.8x     95    58  |   61.1
+```
+
+- **Decode:** all turbo tiers **12.7–15.7×** faster (3→~41 tps), every tier clears the ≥10× bar. Prefill
+  also recovered (turbo8 994→2513 tps — the spill taxed prefill too); util 53–58% → 95–98%.
+- **Quality:** all four tiers stay **logit-equiv PASS** (real_frac 0, recall_mismatch 0) vs the SAME f16
+  ref — the GPU dequant reproduces the CPU dequant's InnerQ output distributionally (R1 validated).
+- **Memory + regression:** KV MiB identical to the validation-path table; f16/q8_0/q4_0 byte-unchanged.
+- **Residual (root-caused 2026-06-06, #389):** turbo plateaus at a uniform ~41 tps **regardless of tier**
+  (turbo8/4/3/2 all ~41 despite 234→61 MiB KV) — the tell-tale of a **tier-independent** bottleneck. It is
+  the Level-A lift itself: `build_attn_mha` casts the **whole** turbo KV view to a full **460 MiB f16
+  buffer every decode step** (`ggml_cast(turboN→F16)` over all n_kv, redundantly re-dequantizing unchanged
+  positions), then runs stock f16 FA on it. The f16 output is the same size for every tier ⇒ flat ~41 tps;
+  the turbo read (61–234 MiB) is not the bottleneck. This **throws away** turbo's bandwidth advantage —
+  turbo decodes *below* q8/q4 (which the FA kernel reads in-register, no materialize). **The AtomicBot fork
+  proves the real ceiling: its CUDA path is the fused FA-VEC (`fattn-vec.cuh` reads `TURBO{2,3,4}_0`
+  in-register as "unquantized" K/V + sparse-V skip; full template-instance set).** Fork benchmark (M4 Max
+  Metal, fused): turbo3 = **88–92 % of f16** (61.84 vs 70.09 tps MoE; 19.71 vs 21.34 dense), 78.5 tps
+  tg128 on M5 — i.e. ≈ q8/q4-class, **not** halved. Ratio-extrapolated to our rig (f16=74) the fused target
+  is **~63–66 tps turbo3**. Closing the gap = **Level B = port the fork's fused FA-VEC turbo path** (#390),
+  existing vendored code, the next milestone. (A cheaper "incremental materialize into a persistent f16
+  shadow" would remove the per-step redundancy but costs a full f16 shadow buffer — negates the runtime
+  memory win and is strictly dominated by the fused path; go straight to Level B.)
+- Patch capture: the 6-file Level-A edit surface is captured in **`0073-turboquant-kv`** at S3 (#372),
+  registered via the patch README row (same snapshot-diff mechanism as `0070`). (S3pA / #383–388)
+
+#### S2 perf-path Level B — fused FA-VEC turbo (in-register K/V, no f16 materialize) ✅ DONE + GATED (2026-06-06)
+
+Level A's flat ~41 tps ceiling is **broken**. The whole-cache `ggml_cast(turboN→F16)` per decode step is
+gone for the value tiers: flash-attention now reads turbo2/3/4 K/V **in-register** (centroid×norm, in the
+WHT-rotated domain), and the expensive Walsh–Hadamard rotation runs only on the n_q-sized **Q** (forward)
+and the small FA **output** (inverse) — never on the KV cache. Acceptance = logit-equivalence, never
+greedy byte-equality.
+
+**New op `GGML_OP_TURBO_WHT`** (`ggml/include/ggml.h`, `ggml/src/ggml.c`, `ggml-cpu/ggml-cpu.c` abort-stub,
+`ggml-cuda/turbo-wht.{cu,cuh}` + `turbo-wht-core.cuh`): direction 0 forward-rotates Q
+(`×scale_inv → signs₁ → WHT → 1/√G·signs₂`); direction 1 inverse-rotates the output. InnerQ folds into
+**both** WHT kernels via the device symbol `turbo_innerq_active_device_scale(ne[0]=head_dim)` (B-adapted —
+no graph-tensor scale src, unlike the fork's mctx tensor). **Parseval:** `WHT(Q⊘s)·WHT(K⊙s) = Q·K`, so the
+InnerQ scale cancels and the turbo K/V reads are InnerQ-**free** (pure centroid×norm).
+
+**Edit surface (all `// opencoti-hook: turboquant perf-lift (… Level B)`):** `fattn-turbo.cuh` (NEW —
+in-register `vec_dot_fattn_vec_KQ_turbo{2,3,4}` + `dequantize_V_turbo{2,3,4}`), `fattn-common.cuh`
+(`get_vec_dot_KQ`/`get_dequantize_V` turbo cases), `fattn-vec.cuh` (`is_turbo_K` config; turbo rides the
+**float** Q path), `fattn.cu` (turbo→VEC gating, D∈{128,256}), `ggml-cuda.cu` (TURBO_WHT dispatch +
+supports_op), `template-instances/fattn-vec-instance-turbo{2,3,4}_0` (+ generator), `src/llama-graph.cpp`
+(`build_attn_mha` fused gate). turbo8 + head_dim-512 global layers keep the Level-A materialize fallback
+(no vec instance / no fork reference).
+
+**Graph gate (`build_attn_mha`):** fuse iff `flash_attn && turbo2/3/4 && K==V && head_dim∈{128,256}` **and**
+(L4.5 prefill-hybrid) per-stream `n_q ≤ TBV_FUSE_MAX_NQ` (default 8) — DECODE fuses, large-n_q PREFILL falls
+to the Level-A materialize→MMA path (turbo prefill on the vec kernel is ~2–3× slower than f16-MMA).
+
+**Gate (GREEN, RTX 3090, Gemma-4 A4B Q4_K_M, 6025-tok prompt, 256-tok forced decode):**
+
+```
+mode    | verdict mean_tv | dec@6k  LevelA  low-ctx | prefill | KV MiB | path
+----------------------------------------------------------------------------------
+f16     | REF    —        |  72.4    73.8    77.6   |  2788   | 460.0  | native
+q8_0    | PASS   0.0122   |  65.4    65.9    —      |  2617   | 244.4  | native
+q4_0    | PASS   0.0094   |  64.6    65.9    70.2   |  2656   | 129.4  | native
+turbo8  | PASS   0.0181   |  42.6    42.5    —      |  2502   | 233.6  | materialize (fallback)
+turbo4  | PASS   0.0136   |  48.5    40.6    —      |   817*  | 122.2  | FUSED
+turbo3  | PASS   0.0067   |  51.4    41.6    64.1   |  1113*  |  89.8  | FUSED
+turbo2  | PASS   0.0179   |  56.0    42.2    —      |  1537*  |  61.1  | FUSED
+```
+`*` always-fuse prefill; the prefill-hybrid (default) restores turbo prefill to ~2474 tps (= materialize)
+while keeping 51.8 decode (A/B turbo3: T=0 2544/41.4, T=8 2474/51.8, T=huge 1099/52.2; hybrid logit-equiv
+PASS, mean_tv 0.0063).
+
+- **Decode ceiling broken + scales with compression:** turbo2 56 > turbo3 51 > turbo4 48 > turbo8-materialize
+  43 tps — the more compressed the tier, the less KV bandwidth/step, the faster decode (the in-register
+  payoff). turbo2 = **86 % of q8_0**, 1.31× the Level-A ceiling. GPU util 96–98 % (Level-A ~53 %).
+- **Quality:** 4/4 logit-equiv PASS (real_frac 0, recall_mismatch 0) vs the same f16 ref; **turbo3 mean_tv
+  0.0067 is the tightest of any tier** — a 3-bit cache beating 4-bit q4 (0.0094) on quality.
+- **Decode-tps is ctx-dependent** (turbo3 64.1 low-ctx → 51.4 @6k). Report the ctx.
+
+**Tier positioning (decided 2026-06-06, user):** at matched bits, native **q8_0/q4_0 dominate turbo8/turbo4**
+on speed AND quality at ~equal VRAM — TurboQuant's MSE centroids buy nothing at 4–8 bit (scalar quant is
+already near-lossless). **Use q8_0/q4_0 for 8/4-bit KV.** TurboQuant earns its place at **3-bit and below**,
+where native has no KV type: **turbo3** Pareto-beats q4 (90 vs 129 MiB, TV 0.0067 vs 0.0094) at ~80 % of q4
+long-ctx decode; **turbo2** is the smallest logit-equiv KV that exists (61 MiB, 56 tps) — the 256k play.
+turbo8/turbo4 stay as ladder-rung / materialize fallbacks; turbo8 is **not** fused.
+
+**Reference (`animehacker/llama-turboquant`, investigated #395):** its headline "tb3 ≈ q4_0" is a **regime
+artifact** (Qwen-0.8B, tg128, Strix-Halo UMA APU → KV cost negligible); reproduced on our rig at low ctx
+(turbo3 64.1 vs q4 70.2 = 91 %). Its kernel does the **same float centroid-LUT dot** we do; the gap at long
+ctx is **intrinsic centroid-decode compute** (turbo3 reads *less* KV than q4 yet decodes slower). No kernel
+unlocks q4-parity at long ctx.
+
+**L4.5 optimization round:** **prefill-hybrid = WINNER, shipped** (default `TBV_FUSE_MAX_NQ=8`). **int8-dp4a**
+(restructure the KQ dot into integer `__dp4a` + int8 centroid codebook) — correctness **de-risked viable**
+(`.opencoti/ab-dp4a-unit2.cu`: softmax TV 0.0005), but gather-bound (dp4a speeds the MAC, not the centroid
+LUT lookups, nor the float V path) and needs re-enabling the q8_1 Q path through the L4-green kernel —
+deferred to **#397** (flag-gated follow-on, correctness already proven). AB1 (fold-WHT-into-FA) costed at
+~1–2 % (60 tiny launches/step) — subsumed by #397's q8_1 path; AB2/AB4/AB5 dropped (centroid-LUT is the right
+primitive per the reference). ⚠️ **Correction (user, 2026-06-07):** the original AB4 write-off —
+"turbo8/4 demoted so D=512 turbo-vec moot" — was a **non-sequitur**. Head_dim-512 coverage is orthogonal
+to *which tiers* survive: turbo8/4 being demoted says nothing about whether the 512-dim global layers need a
+fused turbo path. They do — for turbo2/turbo3 (see below). AB4 should never have been dropped on that basis.
+
+#### S2 perf-path D=512 fused turbo-vec for turbo2/turbo3 ✅ DONE + GATED (2026-06-07, #411) — ships as `0075-d512-turbo-vec`
+
+**Done.** The "D=512 turbo-vec moot" verdict above was scoped to turbo8/4 (demoted in favour of native
+q8/q4). It did **not** hold for the sub-4-bit tiers: **turbo3/turbo2 are the 256k play**, and at head_dim
+512 (Gemma-4 global layers, `n_head_kv=2`, ~1 in 6 layers) they fell back to the **Level-A materialize→MMA**
+path — the fused gate was `head_dim∈{128,256}` only. At long ctx that whole-cache `ggml_cast(turboN→F16)`
+per step is the decode tax that scales linearly with `n_kv`, biting hardest at 256k (our primary case).
+
+**The assumption that a flat D=512 would blow the VEC register budget (the original reason upstream caps
+`can_use_vector_kernel` at ≤256) was WRONG for the turbo path** — no split / tiling was needed. The VEC
+kernel is already **D-generic** (`nthreads`/`nthreads_V` derive from D arithmetically; at D=512, ncols=1:
+`nthreads_V=min(D/4,32)=32`→`V_cols_per_iter=1`, KQ smem `max(ne_KQ, nwarps·1·512)` halfs ≈ 4–8 KB on the
+Ampere half2 path — well under the 48 KB static-smem cap — and VKQ ≈ 8 half2 ≈ 16 regs). A plain
+`DECL_FATTN_VEC_CASE(512, turbo2/3)` instance compiles + runs clean. Ships as **`0075-d512-turbo-vec`**:
+`fattn-vec.cuh` extern decls + the two turbo template instances (+ `generate_cu_files.py` autogen line),
+`fattn.cu` kernel-selection relax (`Q->ne[0]==512`→VEC) + dispatch, `llama-graph.cpp` `fused_turbo_512`
+fuse-gate extension. turbo4/turbo8 @ 512 keep Level-A (no D=512 instance).
+
+**Gate (`.opencoti/p411-d512-gate.sh` + `p411-perf.sh`):** correctness = **Parseval-EXACT** (the WHT is
+block-diagonal over 128-blocks → D=512 = 4 independent Parseval-exact 128-blocks): fused-vs-materialize
+logit-equivalence **`real_frac=0.0` for BOTH turbo3 and turbo2** (bit-equivalent, not merely close). Perf:
+turbo3 @ ctx 27.7k, **fused 41.0 vs materialize 29.7 tok/s = +38%**, scaling with `n_kv` (larger at 256k).
+No new ABI / flag (auto-engages via the existing `TBV_FUSE_MAX_NQ`=8 decode gate). This unblocks the long
+RULER re-run.
+
+- Patch capture: the Level-B edit surface joins **`0073-turboquant-kv`** at S3 (#372) — captured +
+  byte-identical-proven (vendored source stays uncommitted per policy; DSO sha `19168f6a`, both `/root`
+  paths restamped byte-identical).
+
+### S4 — Speculative MTP draft head (`gemma4_assistant`) for Gemma-4 A4B — adopt AtomicBot fork (HIGH RISK, NEW 2026-06-05)
+Pairs with S2 (same fork, same `turbo3` KV). Gives **ours** llamafile a working **Multi-Token
+Prediction** speculative-decode path for Gemma-4 A4B: a ~0.4B **assistant draft head** predicts
+`B-1` future tokens from the target's last hidden state, the 27B-MoE target verifies them in
+parallel → **same output distribution** (lossless), ~85-88% accept / claimed +30-50% short-prompt
+throughput on dense targets. The user frames the head as **"EAGLE3 weight layout with low/mid/high
+feature taps"**; the fork ships it as the custom **`gemma4_assistant`** architecture (EAGLE-family;
+`eagle3` is a sibling `--spec-type`).
+
+**Draft artifact (published, ready to consume):**
+[`AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF`](https://huggingface.co/AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF)
+— F16 (816 MB) / Q8_0 (440) / Q5_K_M / **Q4_K_M (310, recommended draft)** / Q4_K_S. Target = any
+Gemma-4 26B-A4B-it GGUF (our `gemma-4-A4B-98e-v6-coder-it-Q4_K_M` or unsloth's). These GGUFs use the
+`gemma4_assistant` arch and **will NOT load in stock llama.cpp** — that's exactly the gap S4 closes.
+
+**Fork usage we must reproduce (CUDA/Linux):**
+```
+llama-server -m TARGET.gguf --mtp-head ASSISTANT.gguf --spec-type mtp \
+   --draft-block-size 3 --draft-max 8 --draft-min 0 -ngl 99 -ngld 99 \
+   -ctk turbo3 -ctv turbo3 -ctkd turbo3 -ctvd turbo3 -fa on -c 16384
+```
+
+**What S4 must adopt/port into our vendored cosmocc-built `llama.cpp` (study-and-adapt, not
+cherry-pick — fork base ≠ our base, and CUDA lives in our DSO):**
+1. **`gemma4_assistant` architecture** — arch enum + hparams + `models/gemma4-assistant.cpp` graph
+   builder (the EAGLE3 low/mid/high feature-tap head + centroid LM head for E2B/E4B; dense tied head
+   for 26B-A4B). Loader for the assistant GGUF's tensors.
+2. **In-context MTP scheduler** — the fork loads the draft **into the target context** (no second
+   `llama_context`, **no second KV cache**), on a dedicated scheduler; the head consumes the target's
+   last hidden state, drafts a block, the target batch-verifies. This composes with M7 residency and
+   S1 unified KV — design the seam carefully (one cache, one slot, draft tokens appended/verified).
+3. **Runtime flags** (`common/arg.cpp`): `--mtp-head` (+ `-md` alias), `--spec-type mtp`,
+   `--draft-block-size`, `--draft-max`, `--draft-min`, `-ngld`, and draft KV `-ctkd/-ctvd`
+   (depends on S2 for `turbo3`; until S2 lands, draft KV can ride q8_0/f16).
+4. **TS adapter / inference-engine surface** — expose the draft-head path through the opencoti
+   tiered-inference config (later; not S4 core).
+- **Gate:** with the assistant draft loaded, decode is **distribution-identical** to no-draft greedy
+  (teacher-forced KL ≈ 0 / top-1 agreement — reuse the #355 logit-equivalence harness), measured
+  accept-rate in the published 70-88% band, and net **throughput up** vs no-draft on Gemma-4 A4B at
+  RTX-3090. Draft-off byte-identical. NEVER greedy-needle alone.
+- **Risks:** (R-S4a) fork base divergence — the arch/scheduler may touch files heavily rewritten by
+  our 0030-0042/0070 patches; budget a real port. (R-S4b) "no second KV cache" claim must be
+  reconciled with our M7/S1 single-unified-cache model — verify the draft's appended tokens don't
+  corrupt the shared/rolling cache. (R-S4c) cosmocc has no nvcc in the host path; any new CUDA kernel
+  (TurboFlash for the draft) goes in the DSO build, not `:make`.
+
+### S3 — Compose-with-glue + bench + patches + docs ✅ DONE + SHIPPED (2026-06-06, #372)
+The ship: prove PolyKV composes on the M5 advanced-KV stack, capture the vendored delta into
+reproducible patches, wire the user-facing TS knob, and land bookkeeping. **S4 (MTP draft) is split
+out as its own future milestone — its `0062` patch is NOT part of this ship.**
+
+**Adapter knob (P1).** The PolyKV user-facing KV-type selector is `-ctk`/`-ctv`, surfaced on the
+`opencoti-llamafile` adapter as `config.ctypeK`/`ctypeV` (env `OPENCOTI_LLAMAFILE_CTYPE_K/_V`,
+validated against `KNOWN_KV_TYPES` incl. `turbo{2,3,4,8}`), emitted in `buildServerArgs` after
+`--kv-residency-mode` / before `--neo-pipeline`. Default-unset ⇒ no flag emitted (byte-identical to
+prior arg vector). S1's pool selection is a **per-request** JSON param (`shared_pool_slot` /
+`shared_prefix_n_tokens`), server-side, not a CLI flag. 98 adapter tests PASS + typecheck clean.
+
+**Compose bench (P1, `perf/llamafile/advanced-kv-stack.bench.ts`).** Added config **A4** = baseline
++ `-ctk q8_0 -ctv q4_0` (S0 native asymmetric, greedy-needle-safe) and **S′** = the full F5 stack +
+asymmetric KV. Live (Qwen 0.5B) result **11/12 hard PASS**: A4 boots + recovers needle, A4 CUDA KV
+4.89 vs baseline 12.00 MiB (0.41× ≤ 0.6× gate), S′ `cached_n>0` ∧ needle_recovered, S′ per-cell mem <
+S per-cell (soft W3 PASS). The sole hard FAIL (C9, S 51.92 vs B 336.05 tps) is the **pre-existing M2
+head-split perf floor**, not PolyKV — A4-alone decodes 128 tps, proving the asymmetric KV path itself
+is fast; C3/C7 SKIPPED (workload forced no shift this run).
+
+**Live turbo3 compose gate (P1b, `.opencoti/m6-s3-glue-gate.sh`) — the PolyKV ↔ HeadInfer-moot
+evidence.** turbo3 **and** turbo2 K/V under the live F5 stack flags (`--kv-unified --cache-reuse 256
+--rest-kv-eviction --rest-kv-recent 256 --rest-kv-layer -1`, `--parallel 2`, Gemma-4 A4B,
+InnerQ-calibrated) vs the **same-stack f16 reference** (isolates the KV-type effect, not the stack
+effect). Both PASS: turbo3 `real_frac=0.0` mean_tv 0.0164, turbo2 `real_frac=0.0` mean_tv 0.01784,
+`recall_mismatch=0`. **PolyKV ↔ HeadInfer conflict is MOOT:** M7 retired the head-split default
+(`kvResidencyMode auto` = *window*, not the M2 head axis), so TurboQuant's Level-B fused FA-VEC never
+collides with the M2 head-axis split — they cannot both be active. The gate is the proof: turbo fused
+decode composes cleanly with window residency + multi-slot reuse + retention eviction.
+
+**Patches (P2/P3) — captured + byte-identical-proven.** The vendored PolyKV delta over the
+`0001-0042 + 0070 + 0071` baseline (snapshot-diff of a faithful `reset → bootstrap → apply` baseline
+vs the live tree) splits into two patches:
+
+| Patch | Stage | Touches | Acceptance |
+|-------|-------|---------|------------|
+| `0072-poly-kv-pool` | S1 SharedKVPool (#370) | 3 server files (`tools/server/server-{task.h,task.cpp,context.cpp}`) | default-off byte-identical; gate `m6-s1-shared-pool-gate.sh` |
+| `0073-turboquant-kv` | S2 TurboQuant (tiers+InnerQ+Level-A+Level-B, #371/#374–396) | 32 files (13 NEW); `GGML_OP_TURBO_WHT` op; carries the outer `llamafile/build-functions.sh` host-source add | logit-equiv vs f16 (#355); gates `s2-tier-family-gate.sh` + `m6-s3-glue-gate.sh` |
+
+Both numbered **`0072`/`0073` (NOT the plan's stale `0060`/`0061`)** because the build pipeline applies
+patches in **pure lexical sort** order (`build-pipeline.ts`, `.sort()`), and the TurboQuant FA-VEC edits
+sit **on top of** `0070`'s `fattn.cu`/`llama-graph.cpp`/`ggml-cuda.cu` rewrite — `0060/0061` would sort
+*before* 0070 and break the chain (M7 shipped before M6, so the original labels predate it). S0
+asymmetric q8/q4 needs **no patch** (rides M7 dequant-on-lift natively). **Proof:** both reverse-apply
+clean against live (`git apply --check -R`), and a forward-apply onto a fresh baseline copy
+**reproduces the live tree byte-for-byte** (`diff -r` = 0 differing / 0 only-in-live / 0 only-in-fwd,
+outer `build-functions.sh` identical). The full `0001-0042+0070+0071+0072+0073` chain is strict-apply
+proven by composition (#366 + the baseline `:apply` + the `0072/0073` `--check`). Vendored source stays
+uncommitted per policy — only the `.patch` files commit. (DSO sha `19168f6a`; both `/root` paths
+restamped byte-identical.)
+
+#### S2 perf-path — turbo-MMA verify kernel + scalar-K/turbo-V fused decode ✅ SHIPPED OPT-IN default-OFF (2026-07-09, #628/#630, WS1, `0105-0107`)
+
+WorkStream-1 = the **asymmetric strong scalar-K (q8_0/q6_0/q5_0/q4_0) + cheap turbo/TCQ-V** decode
+family — K read as scalar (true-space), V read in-register as rotated turbo/TCQ (no f16 materialize);
+the graph `build_attn_mha` `wht_q`/`wht_o` split applies the OUTPUT inverse-WHT only (K is true-space,
+V is FWHT-rotated). Two decode regimes:
+
+- **Fused-VEC decode (n_q=1)** — the foundation, unconditional. 16 new FA-VEC instances
+  (`fattn-vec-instance-q{4,5,6,8}_0-turbo{2,3}_{0,tcq}.cu`) + `FATTN_VEC_CASES_ALL_D` dispatch + K≠V
+  selector admission + `build-functions.sh` 3e default-build inclusion. Invariant K bit-width > V;
+  reverse (turbo-K/scalar-V) not built. Ships in `0107`.
+- **turbo-MMA verify (n_q>1)** — a turbo-aware tensor-core kernel that dequants q8_0-K + turbo2-V into
+  the MMA smem tile (rotated V; graph `wht_o` keeps it exact) for MTP verify batches at head_dim **128**
+  (Qwen full-attn), **256** (Gemma SWA-local), **512** (Gemma global). Three per-head-dim knobs
+  `WS1_TURBO_MMA_D128/D256/D512`, **all default-OFF** — off ⇒ verify keeps the fused-VEC routing (the
+  pre-#628 default + logit-equiv reference). `WS1_DEROTATE` (inverse-FWHT V in-load ⇒ graph skips
+  `wht_o`) is a separate default-OFF experiment. bug-2130: `!is_quant_kv` for `need_f16_K/V` so
+  `launch_fattn` reads raw quant blocks (f16 keeps true,true = byte-identical). Ships in `0107`.
+
+**VERDICT (#630 — 3-model crossover map on bs2, A4B / omni27 / qw35a3, decode-tps + teacher-forced TV +
+niah, ctx 32k/128k/256k):** turbo-MMA is **mid-ctx-specialized and earns NO default-on anywhere.**
+
+| model (attn) | 32k | 128k | 256k | fidelity |
+|---|---|---|---|---|
+| **A4B** Gemma iSWA, D256-local + D512-global | +4.2% (241.4 vs 231.6) | **+6.4%** (249.1 vs 234.1) | **−14.3%** (190.3 vs 222.0) | niah 100 |
+| **omni27** Qwen3.6-27B dense, D128 full | −0.1% (99.1 vs 99.2) | +0.3% (92.2 vs 91.9) | +0.1% (87.2 vs 87.1) | niah 100, real_frac 0 |
+| **qw35a3** Qwen3.6-35B-A3B MoE, D128 full | +1.2% (204.8 vs 202.4) | +0.5% (187.7 vs 186.7) | +0.4% (168.0 vs 167.3) | niah 100, real_frac 0 |
+
+**Named mechanism (A4B, nsys `--cuda-graph-trace=node`):** *windowed-D256 cast-saving swamped by the
+ctx-growing D512 global-cast tail.* turbo-MMA eliminates the `cpy_turbo2_f16` materialize cast on the
+D256 **SWA-windowed** V-cache, paying instead ~2× slower D256 attention (in-kernel re-dequant) — both
+ctx-INDEPENDENT constants (windowed). The saving it must beat is the **D512 GLOBAL** materialize
+(`cpy_turbo2_f16`, still paid — D512 knob off), whose per-cast cost doubles with ctx (123µs@128k →
+242µs@256k). So the eliminated (windowed) cast stays fixed while the total step balloons with the D512
+tail: net-win @128k (cast eliminated ≈ 2744ms), net-loss @256k (only ≈ 211ms left to eliminate). And it
+is **un-gatable** — a per-op `k->ne[1]` context-gate sees the SWA WINDOW length, not session ctx
+(bug-2133). On **Qwen D128 full-attention** the per-op gate WOULD work (uniform n_kv), but there is
+nothing to gate for: turbo-MMA is bit-exact yet tps-flat because the dense/MoE **FFN dominates the
+verify step** — attention acceleration has no leverage.
+
+**Ship:** OPT-IN, **default-OFF** (`0107`); the D128/D256/D512 instances are additive/correct/harmless.
+WS1's shipped decode wins are `0106` (bug-2126 boot correctness) + **`0105` bug-2127 DCA per-tensor
+defer = the ~2× decode win** — independent of turbo-MMA. Chain 0001..0107 strict-applies from clean
+0.10.3 and reproduces the FINAL tree byte-identically (verified 2026-07-09).
+
+> **This WS1 null does NOT transfer to DCA-on — see WS2 (#629).** The turbo-MMA "no default-on
+> anywhere" verdict is a **noDCA, full-attention verify-batch** result: the D512-global
+> `cpy_turbo2_f16` cast it had to beat was *still paid* (D512 knob off), and on Qwen the FFN
+> swamps the attention win. **WS2 (#629) is the opposite verdict on the DCA-on decode path:** reading
+> the rotated turbo/TCQ-V **in-register inside the multichunk DCA fused flash-attn kernel** (n_q≤16
+> gate) *eliminates* that context-growing whole-cache cast, and is a **validated decode WIN of
+> +68–95% @256k** across A4B / 31B (D512) and omni27 / qw35a3 (D256), every turbo/TCQ tier,
+> retrieval-lossless (niah 100) — now **DEFAULT-ON**. Full matrix + caveats:
+> [context.md §5x](../evaluations/context.md). WS1 turbo-MMA (verify-batch, noDCA) and WS2
+> in-register turbo-V (decode, DCA-on-fused) are distinct kernels with distinct verdicts.
+
+### S4 — Speculative MTP draft head ✅ DONE + SHIPPED (2026-06-07, #373/#404–410, `0074`)
+The ship: the Gemma-4 A4B `gemma4_assistant` Multi-Token-Prediction draft head, loaded **into** the
+target (no second `llama_context` / KV cache), proven a **decode-throughput win** AND logit-equivalent,
+captured as `0074-speculative-mtp.patch`, surfaced on the adapter, bookkept.
+
+**The binding requirement was throughput, and it is MET.** MTP-on must BEAT MTP-off in decode tok/s —
+not just be correct. Final gate on the capture-candidate binary (`.opencoti/p4-fused-gate.sh`,
+Gemma-4 A4B v6-coder Q4_K_M @ RTX 3090, 200 tok × 2 prompts, temp 0):
+
+| mode | decode tok/s | vs off | draft accept |
+|------|-------------|--------|--------------|
+| off (plain greedy) | 78.34 | — | — |
+| seq (per-step MTP) | 113.49 | **1.45×** | 0.769 / 0.606 |
+| fused (single-graph MTP) | 114.03 | **1.46×** | 0.769 / 0.606 |
+
+`seq == fused` byte-identical (the permitted refactor-equivalence use of byte-equality — two
+implementations of the same MTP computation). `off != seq` text divergence is **NOT** a regression: it
+is the documented benign **batch-vs-sequential near-tie** numerical artifact (bug-270) — plain greedy is
+bit-deterministic (verified off-vs-off A==B, 761 B identical), so the divergence is the deterministic
+float difference between the spec-decode's batched target-verify and sequential decode flipping an
+argmax within tolerance. The directive-mandated correctness metric is **logit-equivalence**, not greedy
+byte-equality: `.opencoti/s4-p3-logit-equiv.sh` (8 needle prompts, top-20 teacher-forced) gives
+`real_frac=0.0`, `frac_full_agree=1.0`, 100% draft acceptance (the `VERDICT:FAIL` string is the known
+`mean_tv` artifact for spec-accepted empty-`top:[]` tokens — judge on `real_frac`, per
+[[project_mtp_s4_p3]]).
+
+**Root cause of the early "MTP is ~19% SLOWER" result (bug-419) was weight placement, not
+cuda-graph.** Stage-0 measurement (phase-split timers: submit=22 ms, sync=0.01 ms; `GGML_SCHED_DEBUG=2`)
+showed the assistant's **dense LM head ran as a CPU island** mid-graph, stalling the GPU ~95% idle. The
+`gemma4_assistant` `tok_embd` (~277 MiB) is the weight-tied dense output head, but llama.cpp classifies
+`tok_embd` as an INPUT tensor and `dev_input` is hard-pinned to the CPU buft — so the per-draft-step
+full-vocab (262144) output mul_mat executed on CPU. **The fix is one loader edit:** route the
+`gemma4_assistant` `tok_embd` through the OUTPUT buft (`dev_output`, GPU when the output layer is
+offloaded). cuda-graph replay was MOOT (sync already free); the originally-planned custom fused MTP-head
+kernel was unnecessary. See [[project_dso_cache_poisoning]] for the parallel DSO gotcha class.
+
+**Adapter knob.** `config.mtpHead` / `specType` / `draftNGpuLayers` (env
+`OPENCOTI_LLAMAFILE_MTP_HEAD` / `_SPEC_TYPE` / `_DRAFT_NGL`), emitted in `buildServerArgs` as
+`--mtp-head <path>` / `--spec-type mtp` / `-ngld <n>` after `--fused-moe-up-gate`, before `extraArgs`.
+`-ngld` is REQUIRED for the win (a CPU-resident draft head re-seeds the stall). Default-unset ⇒ no draft
+(plain decode, byte-identical arg vector). 31 adapter tests PASS + typecheck clean.
+
+**Patch (`0074-speculative-mtp`) — captured + byte-identical-proven both directions.** Snapshot-diff of
+the live tree vs a faithful `base-mtp` (= `base-poly` + `apply(0072,0073)` = `0001-0042+0070+0071+0072+
+0073`, no MTP): 24 files, 1 NEW (`src/models/gemma4-assistant.cpp`), 0 outer-vendor delta. Forward-apply
+onto a fresh base-mtp reproduces the live tree byte-for-byte (0/24 differ); reverse-apply onto live
+reproduces base-mtp (0/23 differ + new file deleted). Three surgical upstream edits (marker
+`opencoti F5 M6-S4 mtp`) + additive surface — registered in `docs/protocols/UPSTREAM_SYNC.md`. The
+reverted sparse-argmax centroid fast path (dead for this dense `use_ordered_embeddings=false` GGUF) is
+banked at `.opencoti/banked/mtp-sparse-argmax.md`, not shipped (verification discipline: never ship an
+unexercised path). Vendored source stays uncommitted per policy — only the `.patch` commits.
+
+**P5 (#409) — POSITION_WINDOW MTP perf path: characterized, NOT implemented (route inapplicable).**
+The task hypothesized routing the MTP cross-read through `build_attn_mha_position_window` if the
+`get_k` streamback under window residency is a bottleneck. Measurement (forced `--vram-target 600`,
+n_kv~2919) shows Gemma-4 A4B's iSWA cache is **POSITION_WINDOW-ineligible** (`swa_type=1`; eligibility
+requires `swa_type==NONE`) — overflow relief is **GPU_STREAM**, so the proposed window route never
+engages. Under GPU_STREAM overflow MTP is **throughput-negative** (MTP-on 30.3 vs off 46.3 tok/s;
+correct — needle recovered, accept 0.786). Refused to ship a dead position-window MTP route (same
+discipline as the reverted sparse-argmax). DISPOSITION: the supported long-context MTP path is
+**turbo-KV-resident** — S2 Level-B + the D=512 fused turbo instances (#411) keep the 256k cache in
+VRAM → `GPU_RESIDENT` → MTP keeps its 1.46× win; f16-overflow-streaming is a config to avoid (f16 KV
+doesn't fit 256k anyway). See [[bug-419]], bug-426.
+
+### Phase-0 fork recon (2026-06-05) — findings that shape S2 + S4
+Shallow-cloned the fork (`/srv/dev-disk-by-label-opt/dev/_recon/atomic-llama-cpp-turboquant`) and
+diffed its TurboQuant + MTP surfaces against OURS. **Headline: the fork's base is contemporaneous
+with ours, not years apart** — both end the ggml type table at `MXFP4=39`, `GGML_TYPE_COUNT=42`; both
+carry the same recent model set (`gemma4-iswa`, `minimax-m2`, `qwen35*`); OURS already has
+`LLM_ARCH_GEMMA4` and `COMMON_SPECULATIVE_TYPE_EAGLE3` + a full ngram/draft speculative family. So
+both milestones are *additive onto a near-identical base*, far cheaper than a cross-era port.
+
+**S2 (TurboQuant) edit surface:**
+- ggml types append cleanly at 42-46 (`TURBO2_0/3_0/4_0`, `TQ3_1S/TQ4_1S`), `COUNT 42→47` — fork
+  `ggml.h:432`, traits `ggml.c:737`. Block structs in `ggml-common.h` (TURBO3_0 = 14B/128-blk:
+  fp16 norm + 2-bit qs[32] + 1-bit signs[16]).
+- CPU quant/dequant is ONE new file `ggml/src/ggml-turbo-quant.c` (~1026 lines: seeded WHT/Hadamard
+  rotation `TURBO_SEED_ROTATION=42` + PolarQuant/Lloyd-Max codebook) + dispatcher cases in `ggml.c`.
+  **Self-contained, low-conflict.**
+- CUDA (DSO): `turbo-wht.cu`, `turbo-quant.cuh`, `mmvq-tq.cu`, `vecdotq.cuh`, + **12 `template-
+  instances/fattn-vec-instance-turbo*.cu`**. The fattn hook is the **only CRITICAL conflict** (OURS
+  0070 heavily rewrote `fattn.cu`).
+- CLI: `kv_cache_types[]` in `common/arg.cpp` maps `"turbo3"` → enum (keep the spelling for fork-GGUF
+  compat).
+- **Bring-up order:** Gate-1 = TURBO3_0 **CPU-only** (enum + struct + `ggml-turbo-quant.c` + traits +
+  dispatcher + CLI, NO CUDA, NO fattn) → validates WHT/codebook + type registration with zero
+  fattn.cu risk. Gate-2 = CUDA dequant w/ dequant-then-compute fallback (reuse our S3d lift). Gate-3
+  = fused TurboFlash templates (the 0070 rebase).
+
+- **Gate-2 design (2026-06-05, decided).** The fork's `convert.cu` turbo3→f16 emits **rotated** f16
+  (it relies on the fused kernel to rotate q on-the-fly). Our S3d "dequant-on-lift → standard f16
+  FA" path does NOT rotate q, so the correct fallback un-rotates K/V instead:
+  **dequant turbo3 → rotated f16 → `ggml_cuda_turbo_wht` (direction=1, inverse) → true-space f16
+  K/V → unmodified standard f16 FA.** Minimal additive port set: `turbo-quant.cuh` + `turbo-wht.cu`
+  /`.cuh` (verbatim) → `dequantize.cuh` turbo3 accessor → `convert.cu` to_fp16 cases → graph
+  insertion of the inverse-WHT pass on the lifted K/V → `supports_op`/type registration in
+  `ggml-cuda.cu` → CUDA DSO rebuild → GPU cosine/needle gate vs vanilla f16.
+  - **InnerQ stub-disabled for Gate-2 (decided, user).** `turbo-quant.cuh` couples to InnerQ
+    device globals (`d_innerq_scale*`, per-channel Q/V equalization) defined in `turbo-innerq.cu`.
+    For the Gate-2 fallback we compile with **identity InnerQ** (scales = 1, `d_innerq_active = 0`)
+    — either by porting `turbo-innerq.cu` in its dormant/identity state or by stubbing the globals
+    — so K/V un-rotation is the plain inverse WHT. **InnerQ is re-enabled in Gate-3** (the fused
+    TurboFlash path), where the q-rotation + channel-scale fusion actually pays off. Identity
+    InnerQ is numerically exact for the un-rotation (it only rescales channels pre-quant; with
+    scale=1 the round-trip is the bare WHT codebook path Gate-1 already validated at cosine 0.983).
+  - **Gate-2 is two halves (both mechanism-matched to our tree):**
+    - **READ (dequant → true f16) — source DONE 2026-06-05.** New `ggml-cuda/turbo-dequant.cuh`
+      (InnerQ-free; folds the inverse WHT into the per-block dequant since turbo3 block=128=WHT
+      group) + `convert.cu` `to_fp16_cuda` & `to_fp16_nc_cuda` TURBO3_0 cases (use
+      `dequantize_turbo3_0_inv`) + `QR_TURBO3` define in ggml-common.h. Routes through our
+      **type-agnostic `GGML_OP_STREAMING_FLASH_ATTN`** (supports_op gates head_dim only; the op
+      dequants K/V to f16 on the S3d lift at fattn.cu:748) → **no standard-FA supports_op change,
+      no graph node added.** The gate must force the streaming path (residency-mode / low
+      `--vram-target`) so turbo3 never hits the stock `FLASH_ATTN_EXT` (which lacks a turbo3 vec
+      kernel until Gate-3).
+    - **WRITE (f32 → turbo3) — NEXT.** Our KV store uses `ggml_set_rows` (cpy_k/cpy_v →
+      `ggml_set_rows`, llama-kv-cache.cpp:3091/3188/3204) — the SAME mechanism as the fork's
+      `set-rows.cu`. Port `k_set_rows_turbo3` + `k_set_rows_turbo3_tail` + `set_rows_cuda_turbo3`
+      (~300 lines: group forward-WHT via warp shuffle, `__ballot_sync` sign pack, corrected-norm)
+      into our `set-rows.cu` (which today has 0 turbo refs), + `GGML_OP_SET_ROWS` supports_op /
+      dispatch for a turbo3 dst. Then ONE CUDA DSO rebuild + GPU cosine/needle gate vs vanilla f16.
+
+- **Gate-1 — DONE (2026-06-05).** Host build clean (`ggml-turbo-quant.c.o` linked; two splice bugs
+  fixed: bug-333 truncated TQ4_1S `type_traits` entry, bug-334 decls trapped inside
+  `#ifdef __cplusplus`). Registration proven by the boot line
+  `llama_kv_cache: … K (turbo3): 1.37 MiB, V (turbo3): 1.37 MiB` + `attn_rot_k = 1` on Qwen-1.5B
+  CPU (`-ngl 0 -ctk turbo3 -ctv turbo3`) — i.e. `-ctk turbo3` accepted, type name resolves, cache
+  allocates at the compressed block size (~5× vs f16's ~7 MiB). CPU then **segfaults at warmup** —
+  EXPECTED: `dequantize_row_turbo3_0` is a rotated-space stub (no inverse WHT; the GPU FA kernel
+  rotates q so `(Rq)·(Rk)=q·k`), so turbo3 *attention compute* is Gate-2/3 (CUDA) scope, not a
+  Gate-1 fail. Gate-1c math validated out-of-band: vendored kernel primitives
+  (`turbo_cpu_fwht`/`_inverse`/`nearest_centroid_3bit`/`CENTROIDS_3BIT`) are byte-identical (modulo
+  `GGML_API` macro + a comment) to `_recon/turbo3_selftest` (cosine 0.983 round-trip). Gate script
+  `.opencoti/s2-gate1.sh`. **Next: Gate-2 (CUDA dequant-then-FA via S3d lift) — requires DSO rebuild.**
+
+**S4 (MTP draft head) edit surface:**
+- `gemma4_assistant` arch = new file `src/models/gemma4-assistant.cpp` (clean) + enum/hparams in
+  `llama-arch.{h,cpp}` (`n_embd_backbone`, `n_centroids`, `centroid_top_k`, `use_ordered_embeddings`,
+  `requires_target_arch`). EAGLE3 feature taps = target token-emb + prev backbone hidden →
+  `mtp_pre_projection` → transformer → backbone hidden → (centroid head for E2B/E4B | dense head for
+  A4B). Greedy argmax on device.
+- **"No second KV cache" CONFIRMED in code:** the assistant is loaded INTO the target via
+  `llama_model_load_mtp_from_file()` (tensors prefixed `mtp.`, stored as `target->mtp_assistant`);
+  there is no second `llama_context`. MTP cross-attn reads the target's existing KV at the **last
+  full/SWA target layer** (`build_attn_mtp()` → `gemma4_mtp_kv_layer_last_in_range`). Scheduler =
+  `common_speculative_state_mtp` (`common/speculative.cpp`) + `llama_decode_mtp_async/wait`
+  (`llama-context.cpp`), async depth-2 with lazy-wait overlap; MTP only READS target KV, never writes.
+- Flags additive in `common/arg.cpp`: `--mtp-head`/`-md`, `--spec-type mtp`, `--draft-block-size`
+  (default 3), `--n-draft-max/min`, `-ngld`, `-ctkd/-ctvd`.
+- **#1 integration risk (R-S4b, confirmed):** `build_attn_mtp()` must read target KV through OUR M7
+  rolling-KV (POSITION_WINDOW) + S1 accessor dispatch, not the fork's direct `cache_k/cache_v`; and
+  S1/M7 must not mutate the target prefix mid-MTP-snapshot. Mitigation: route MTP's attn through the
+  same accessor as the regular forward; keep the slot in decode-state (no compress) while an MTP
+  draft is pending. Fallback: dequant-on-lift the window into a temp buffer for the MTP read.
+- **Bring-up order:** P1 arch+loader (smoke-load the assistant GGUF, no decode) → P2 flags/params →
+  P3 scheduler skeleton (async no-ops) → P4 KV-accessor merge (single-token MTP forward, cosine vs
+  naive) → P5 async + S1 race-guard (turn-2 latency + logit-equivalence) → P6 docs/patch 0062.
+
+## Critical files (anticipated)
+- `src/llama-kv-cache.{cpp,h}` — pool/shared-prefix storage, type-asymmetric K/V, compression-ratio
+  into the M7 tile-sizing.
+- `ggml/src/ggml-cuda/fattn.cu` + S3d dequant-on-lift — read compressed tiles (S0 reuses; S2 adds
+  the TBQ8_0 / TBQ4_0 / TBQ3_0 dequant kernels).
+- `ggml` core — `TBQ8_0` / `TBQ4_0` / `TBQ3_0` type registration (S2).
+- `common/arg.cpp` — pool/agent + (later) `--poly-kv` knobs; `packages/opencoti-llamafile` adapter.
+- Patches `0072-poly-kv-pool` (S1) / `0073-turboquant-kv` (S2); gates `.opencoti/m6-s*-*.sh`;
+  `perf/llamafile/advanced-kv-stack.bench.ts`.
+
+## Verification
+1. S0: single-tenant cosine/RULER == vanilla at asymmetric q8_0/q4_0; ~2.4× KV; M7 overflow path OK.
+2. S1: 2-agent shared-prefix recall; O(1) shared-prefix memory.
+3. S2: cosine/RULER holds per TurboQuant tier (TBQ8_0 / TBQ4_0 / TBQ3_0); TBQ3_0 ~2.9× compression.
+4. S3: glue bench (A4/S′ + turbo3/turbo2 logit-equiv compose); patches `0072`/`0073` strict-apply in
+   series after `0071`, reverse-apply clean + forward-reproduce byte-identical; hooks registered; reap/DSO hygiene. ✅
+
+## Standing ops
+`OPENCOTI_NO_CCACHE=1`; `:cuda` rebuilds DSO (~24min) only when S2 lands its kernels — S0/S1 are
+host-side; vendor-backup before any destructive vendor op; reap `llamafile.*--server` (never
+embedder 38092 / SSH); DSO restamp before each ours boot; GPU ≥10 GiB; commit only when asked;
+soft-fork additive + `opencoti-hook:` markers registered; RULER acceptance via cosine/semantic/RULER,
+never greedy byte-equality.

docs/features/rolling_kv.md

@@ -0,0 +1,1004 @@
+# F5 M7 — Rolling KV: position-windowed KV residency
+
+**Status:** SHIPPED (2026-06-04, `dev`).  
+> The design below (locked 2026-05-29) describes the head-axis streaming
+> double-buffer pipeline. **That approach was superseded mid-flight** — on
+> Gemma-4's 2-KV-head global layers the head axis can only express a binary
+> 0/50/100 spill, and the streaming op was inert under attention softcap. The
+> **shipped** design decomposes residency by **KEY POSITION**: a device-resident
+> window `[0, wc)` + a pinned-host tail `[wc, n_kv)` on plain flash-attention,
+> merged by an online-softmax combine (keystone: LSE-emitting `launch_fattn`).
+> The tactic auto-selects at cache construction (eligibility predicate +
+> measured PCIe-bandwidth/compute crossover; host-agnostic, nothing hardcoded)
+> and is driven by **`--kv-residency-mode {auto,head,window}`** (default `auto`).
+> The dormant CPU_FA_TAIL tactic stays in-tree for sub-x1 PCIe links but is
+> gated out of auto-select. Acceptance is **logit-distribution equivalence vs
+> vanilla** (`.opencoti/rolling-kv-equiv-gate.sh`), not greedy byte-equality.
+> Ships as `0070-rolling-kv` (29 files); the orthogonal cosmocc state-IO fix
+> (bug-269) ships as `0071-state-io-cosmocc`. See `STATE_SUMMARY.md` ("W4 — M7
+> Rolling KV … SHIPPED") and the `0070`/`0071` rows in
+> `vendors/patches/llamafile/README.md` for the as-shipped record.
+
+**Patch slot:** `0070-rolling-kv.patch` (was LMCache, repurposed —
+LMCache stays deferred under a future slot if it returns).  
+**Prerequisites:** #289 pinned host buffer **(SHIPPED, 0034)**, #290 CPU
+FA TG threading, #293 rebar / PCIe probe **(SHIPPED, 0036 — runtime
+consumption + PCIe-link autodetect; see M7-B)**.  
+**Replaces:** the static GPU/CPU partition in F5 M2
+(`--headinfer-gpu-heads-frac`) — M2 becomes a runtime tactic the
+scheduler picks per-layer, not a user-facing flag.  
+**Sister docs:** [`advanced_kv.md`](advanced_kv.md) (the M0-M6 series
+this builds on).
+
+## Why this exists
+
+opencoti's M2 puts a static fraction of attention heads on GPU and the
+rest on CPU. At Gemma 4 A4B + Q4_K_M + 256k + q8_0 KV that pins ~19 GB
+on a 24 GB card and leaves 4-5 GB GPU idle for the entire decode while
+half the KV computes on CPU through a slow CPU-FA path. The card has
+spare capacity it can't reach because the policy is static, the CPU
+side never returns to GPU compute, and there is no streaming staging
+between host and device. The result observed today (RULER vt@262144,
+F5 stack): 0.0 score, semantic collapse — task tracked separately as
+bug-226, but the deeper architectural gap is what M7 fixes.
+
+The right design for sequential, memory-bandwidth-bound attention is
+classic double-buffered streaming: hold all KV in pinned host memory,
+keep a small pool of GPU ping-pong slots, run compute on one slot
+while DMA loads the next. Both PCIe and SMs are always active. VRAM is
+always hot.
+
+## Decisions locked (2026-05-29)
+
+1. **Tile axis = composite (PCIe-auto-adapted).**
+   Per-layer would have been the cleanest mapping, but on solidPC
+   (PCIe 3.0 x8, ~6.5 GB/s effective) a single layer's KV at 256k
+   q8_0 is ~530 MB, transferring in ~80 ms while per-layer compute is
+   ~5-15 ms. Per-layer pipeline is PCIe-bound 6-15×. The auto-adapt
+   loop at server boot picks the tile axis: per-layer when compute
+   ≥ transfer; position-window sub-tile otherwise. solidPC will land
+   on position-window; a future PCIe 4.0 x16 host falls into
+   per-layer automatically with no flag changes.
+
+2. **M2 retires as a flag, lives on as a runtime tactic.**
+   `--headinfer-gpu-heads-frac` is removed. The pipeline scheduler
+   picks per-layer between two tactics:
+   - **GPU-stream tactic** (new) — fetch this layer's tile via the
+     ping-pong pool, compute FA on GPU
+   - **CPU-spill tactic** (M2-style) — keep this layer's KV pinned
+     on host, run CPU FA inline
+   User sets `--vram-target` (e.g. `--vram-target 22000` for 22 GB);
+   the scheduler aims to saturate that without exceeding it.
+
+3. **PCIe 3.0 x8 is permanent on this host class.**
+   solidPC has a Ryzen 5600G; the iGPU absorbs 8 lanes and the board
+   caps the discrete slot at gen3. This is common (5600G/5700G class).
+   Auto-adapt is permanent infrastructure, not edge-case handling.
+
+4. **Default tactic is GPU_RESIDENT; relief is load-adaptive and reversible.**
+   *(locked 2026-05-30 — supersedes the boot-immutable tactic model below;
+   motivated by the W2 RULER ON-vs-OFF run, where iqk speeding the CPU-half FA
+   moved wall-time ~0% because the CPU split should not have been engaged at all
+   with 23 GiB VRAM free.)*
+   The engine **maximizes VRAM by default** and only sheds load to slower
+   tactics when it must:
+   - The server starts a fixed pool of N parallel slots (`--parallel N`, e.g.
+     4/8/…) but allocates KV **dynamically**, never at a fixed CPU/GPU split.
+   - With 1–2 active requests whose KV fits, **every layer is GPU_RESIDENT** —
+     KV lives on the card, zero DMA, zero CPU FA: vanilla's fast path, using all
+     available VRAM natively.
+   - As concurrency rises and the working set approaches the VRAM budget, the
+     engine engages graduated **relief efforts, in order**, only for the
+     layers/streams that no longer fit: (a) tile streaming (ping-pong
+     double-buffer, `GPU_STREAM`), then (b) sub-tile streaming
+     (`GPU_STREAM_SUBTILE`), then (c) CPU/GPU KV split (`CPU_SPILL`, M2-style,
+     consuming the #290 iqk engine).
+   - As requests drain, the engine **transitions back** toward GPU_RESIDENT,
+     reclaiming VRAM for the survivors.
+   Residency is therefore a runtime function of *(free VRAM, context size,
+   active-request count)*, re-evaluated as load changes — **not** a boot-time
+   immutable table. `--vram-target` is a **cap** (leave headroom for siblings),
+   defaulting to "all free VRAM − reserve" (maximize). The fixed
+   `--headinfer-gpu-heads-frac` split is never the default; CPU spill is the
+   last resort, reached only under genuine VRAM pressure.
+
+## Architecture
+
+### Storage layer
+
+All per-layer KV lives in **pinned host memory**
+(`ggml_backend_dev_host_buffer_type(cuda_dev)` →
+`cudaHostAlloc(cudaHostAllocPortable)`). Pinning unlocks full PCIe
+bandwidth on async DMA and is the foundation #289 builds. With M7 in
+place, M2's CPU-half pinning generalizes to *all* KV: the GPU view of
+each layer is a transient slot, never a permanent home.
+
+A small "hot-tile" reserve at the GPU side can hold the most-recently-
+streamed tiles across tokens (M1-retention-style affinity), avoiding
+redundant DMA when consecutive tokens both attend to the same layer
+range. Default reserve = 0 (start simple), tunable.
+
+### Slot pool
+
+Boot-time computed:
+
+```
+free_vram      = nvidia-smi free at server start, minus user --vram-target slack
+reserve        = model weights + compute scratch + cuBLAS workspace (~1-2 GiB)
+slot_budget    = free_vram - reserve - hot_tile_reserve
+tile_bytes_max = (compute_ms_per_layer × effective_bw_GBps × safety_0.8)
+n_slots        = clamp(slot_budget / tile_bytes_max, 2, 4)
+```
+
+Two slots is the minimum for a ping-pong; three lets the scheduler
+prefetch one ahead of compute; four covers the case where compute and
+copy are unbalanced enough that staggered prefetch helps. Above four
+gives diminishing returns since each extra slot just delays
+host-buffer reuse.
+
+### CUDA stream graph
+
+Two dedicated streams per slot pair:
+- `copy_stream` — issues `cudaMemcpyAsync(slot_i, host_kv_layer_L, ...)`
+- `compute_stream` — runs FA over `slot_i`
+
+Event sync:
+- `copy_done[i]` signals after copy into slot i — `compute_stream`
+  waits before consuming slot i
+- `compute_done[i]` signals after FA over slot i — `copy_stream` waits
+  before overwriting slot i
+
+Forward pass per token (sketch):
+
+```
+for layer L in 0..n_layers:
+    slot = next_slot()
+    if not slot.holds(L):                          # hot-tile miss
+        copy_stream.memcpy_async(slot, host_kv[L]) # ~tile_ms
+        record(copy_done[slot])
+    compute_stream.wait(copy_done[slot])
+    compute_stream.fa(Q[L], slot, V_slot)          # ~tile_ms
+    record(compute_done[slot])
+    # next iteration: prefetch L+1 while compute on L+0 finishes
+```
+
+When `tile_bytes_max < per_layer_kv_bytes`, the inner loop runs N
+position-window sub-tiles per layer with online-softmax accumulation
+across them. FA's running max + denominator carry between sub-tiles
+naturally — flash-attention's exact math, just with the K/V chunk
+source coming from a rotating slot instead of a contiguous tensor.
+
+### M2 as runtime tactic
+
+The scheduler holds a `layer_tactic[L]` table picked at server boot:
+
+```
+for layer L in 0..n_layers:
+    if total_resident_kv + kv_bytes(L) ≤ slot_budget:   # it fits → keep it hot
+        layer_tactic[L] = GPU_RESIDENT       # zero DMA, zero CPU — vanilla's fast path
+    else if tile_bytes_max ≥ kv_bytes_per_layer(L) AND has_slot_headroom():
+        layer_tactic[L] = GPU_STREAM         # ping-pong double-buffer
+    else if cpu_fa_throughput(L) > pcie_throughput(L):
+        layer_tactic[L] = CPU_SPILL          # M2-style — KV pinned, CPU computes (#290)
+    else:
+        layer_tactic[L] = GPU_STREAM_SUBTILE
+```
+
+The table is **re-evaluated as load changes** (active-request count,
+working-set vs VRAM budget) — *not* immutable for the server lifetime
+(superseding the original boot-immutable model, per Decision 4,
+2026-05-30). With 1–2 requests that fit, every layer is `GPU_RESIDENT`
+and the engine runs at vanilla speed on all available VRAM; as
+concurrency grows it demotes the layers/streams that no longer fit
+`GPU_RESIDENT → GPU_STREAM → GPU_STREAM_SUBTILE → CPU_SPILL` in that
+order; as requests drain it promotes them back toward `GPU_RESIDENT`.
+Slot-level adaptivity (which tile in which slot, prefetch ordering)
+rides underneath the tactic-level transitions.
+
+`GPU_RESIDENT` is a first-class tactic (it also absorbs the iswa-window
+layers noted in Open Questions). CPU-spill consumes the work from #290
+(CPU FA TG threading port) and is the **last-resort** tactic, reached
+only under genuine VRAM pressure.
+
+## Auto-adapt loop (server boot, ~100 ms total)
+
+1. **PCIe topology** — read
+   `/sys/bus/pci/devices/<bus>/{current,max}_link_{width,speed}`,
+   compute `effective_bw_GBps = width × gtps × 0.8 / 8`. Cache the
+   value for the server's lifetime.
+2. **Compute microbench** — synthesize one layer's worth of K/V
+   tensors at f16, run a 1-step `ggml_flash_attn_ext` with `seq_len =
+   target_ctx`, measure wall time on `compute_stream`. Yields
+   `ms_per_layer_at_target_ctx`.
+3. **Tile sizing** — apply the formula above.
+4. **Tactic table** — populate `layer_tactic[L]` from per-layer KV
+   sizes (Gemma 4 iswa has uniform `n_kv_heads`; trivial. Mixture
+   architectures with variable per-layer KV are the only case where
+   per-L matters).
+5. **Log** — print the chosen tile axis, size, n_slots, and tactic
+   histogram (`N_layers_gpu_stream, N_layers_gpu_subtile,
+   N_layers_cpu_spill`) so operators can sanity-check.
+
+The probe binary (`perf/llamafile/rebar-probe.sh`) handles step 1's
+discovery once at install / first-boot; the server reads its cached
+JSON. The compute microbench (step 2) runs inside the server itself,
+no external tool. This keeps the boot-time cost bounded and avoids
+shelling out.
+
+## Phase breakdown
+
+### M7-A — Pinned-host KV residency
+Consume #289's pinned-host-buffer-type swap + staging pool. Generalize
+M2's CPU-half buffer to all KV: every layer's K/V starts in pinned
+host memory regardless of tactic.
+
+### M7-B — Auto-adapt loop
+Implements PCIe topology read + compute microbench + tile sizing
+formula + tactic table population at server boot. Logs decisions.
+
+> **Step 1 (PCIe topology read) is already shipped** by W1 / patch 0036
+> (#293). At server boot `pcie_profile_init()` (`common/pcie-profile.{h,cpp}`)
+> resolves `effective_bw_gbps` via the 4-tier cascade (manual override →
+> measured probe JSON → `nvidia-smi` → conservative PCIe3 x8 default) and
+> logs it. M7-B consumes the resolved `pcie_profile_get().effective_bw_gbps`
+> directly — no new topology code needed; M7-B adds only the compute
+> microbench (step 2), tile sizing (step 3), and tactic table (step 4).
+
+### M7-C — Tile streamer + slot pool
+Allocates `n_slots` GPU buffers of `tile_bytes_max`. Owns the slot
+rotation policy (round-robin with hot-tile affinity if reserve > 0).
+
+### M7-D — Compute/copy CUDA stream graph
+Two CUDA streams per slot pair, event-based sync, async DMA enqueue,
+graph captured into ggml's existing compute backend so FA consumers
+see no change.
+
+### M7-E — M2 retirement / runtime-tactic scheduler
+Removes `--headinfer-gpu-heads-frac` from `common/arg.cpp`. Adds
+`--vram-target` (number, MiB). Plumbs the tactic table through
+`build_attn_mha` so per-layer kernel dispatch picks GPU stream vs CPU
+spill from the table, not from a flag.
+
+### M7-F — TS adapter + config schema
+`packages/opencoti-llamafile/src/config.ts`: drop `headinferGpuHeadsFrac`,
+add `vramTargetMiB`. `buildServerArgs` emits the new flag. Update
+adapter tests.
+
+### M7-G — Verification
+1. **Cosine ≥ 0.999 unified-mode equivalence** vs pre-M7 binary at
+   small ctx where tile sizing doesn't activate (M5 stack bench unchanged).
+2. **256k+q8_0+Gemma 4 quality ≥ vanilla−2pp** — RULER vt@262144,
+   `-ctk q8_0 -ctv q8_0`, ours-streaming vs vanilla. NOTE (2026-06-01,
+   corrected): this is a *vanilla-q8_0 parity* gate, NOT a "bug-226 fix"
+   gate. bug-226's real defect (concat_q block-scrambling) was already fixed
+   by 0035; q8_0 is confirmed coherent on this Gemma at small ctx (vanilla,
+   resident, AND S3d streaming n_head_cpu=4 all = 100% on RULER vt@4k,
+   templated). So this cell tests that the streaming pipeline holds q8_0
+   parity *at 256k* — the only unconfirmed cell — and is gated behind a 256k
+   f16 canary per the standing "no 256k re-run until M7 lands + canary"
+   instruction. A reference vanilla-q8_0@256k baseline must be captured here
+   (it was never run — the kvtype-split confound). Do NOT treat ours≈vanilla
+   as a tautology: vanilla q8_0 scores ~100% at coherence-checkable ctx.
+3. **VRAM target enforced** — server holds within `--vram-target` ±
+   200 MiB across the run.
+4. **Throughput ≥ vanilla at 256k** — at concurrency=1, M7 should at
+   least match vanilla (PCIe-bound floor) and ideally beat it through
+   better VRAM utilization (more GPU compute, less CPU spill).
+5. **Hook count audit**: removes M2's hook surface (frac flag);
+   adds M7's vram-target hook. Net delta tracked in
+   `docs/protocols/UPSTREAM_SYNC.md` Registry.
+6. **Dynamic residency transition** (Decision 4) — drive a load ramp:
+   1 request (expect all-`GPU_RESIDENT`, throughput ≈ vanilla, VRAM near
+   `--vram-target`), ramp to `--parallel N` saturation (expect graduated
+   relief: stream → subtile → cpu-spill engages, VRAM held within target),
+   then drain back to 1 (expect promotion back to `GPU_RESIDENT`,
+   throughput recovers). Assert the tactic histogram shifts with load in
+   both directions and never exceeds the VRAM cap. This is the core test
+   of the maximize-VRAM-by-default behavior.
+
+### M7-H — Substance commit + docs ship
+Snapshot-diff `0070-rolling-kv.patch`. Update
+`advanced_kv.md` M7 section (was LMCache → now Rolling KV). Move
+`headinferGpuHeadsFrac` to a deprecated config field with a clear
+upgrade note. Update STATE_SUMMARY, `.wolf/anatomy.md`,
+`.wolf/memory.md`. Task #296 → completed.
+
+## Rung 2 implementation blueprint (code-grounded, 2026-05-30)
+
+Rungs 0 (M7-A auto-residency) and 1 (M7-B tactic table) have shipped on
+`dev` (commits `7fe88b8`, `8957fc9`; patch `0070-rolling-kv.patch`). The
+inert per-layer `headinfer_tactic` table + `rolling_kv_plan` + `get_layer_tactic(il)`
+are in place. Rung 2 wires the **GPU_STREAM** tactic — and the exploration
+below pins down what that actually requires before any kernel is written.
+
+### The load-bearing finding: GPU_STREAM is inseparable from online-softmax
+
+A "single-tile GPU_STREAM" (copy the whole layer's KV to one slot, run a
+standard `ggml_flash_attn_ext`, no running-max accumulation) **does not
+exist as a useful tactic**. GPU_STREAM only has value when a layer's KV
+*exceeds* what fits resident on the card — if it fit one slot, the correct
+tactic is GPU_RESIDENT (keep it; don't re-DMA every step). So GPU_STREAM
+*always* means KV > slot ⇒ multiple tiles ⇒ **cross-tile online-softmax**
+(running max + denominator in F32). There is no low-risk subset of the
+streaming win; the precision-critical kernel is the whole deliverable.
+This is why Rung 2 is gated separately and why the plan flags it HIGH-risk.
+
+### The discrete-GPU linchpin ⇒ a custom op, not graph nodes
+
+On the RTX 3090 (non-integrated), CUDA `supports_buft` returns FALSE for
+pinned-host buffers (`ggml-cuda.cu` ~5144), so if GPU_STREAM were expressed
+as ordinary graph nodes (KV on a pinned-host CPU buffer feeding
+`ggml_flash_attn_ext`), the ggml scheduler inserts a **sequential**
+host→device copy (`ggml-backend.cpp` ~1269) — no overlap, no win over
+CPU_SPILL. Therefore M7-D must be a **custom streaming-FA op** that owns its
+own `cudaMemcpyAsync` on a dedicated copy stream. It cannot be plain nodes.
+
+### Integration points (mapped)
+
+- **Dispatch site:** `src/llama-graph.cpp:2306`. Today:
+  `if (cparams.neo_pipeline_mode != 0 && mctx_cur->headinfer_split_active(il))`
+  → `build_attn_mha_neo` (two-FA GPU+CPU); else → `build_attn_mha` (single
+  FA). Rung 2 reads `mctx_cur->get_layer_tactic(il)` here and adds a third
+  branch for GPU_STREAM. **Requires** exposing `get_layer_tactic` on
+  `llama_kv_cache_context` (delegating to the cache, like
+  `headinfer_split_active` at `llama-kv-cache.h:514`).
+- **Stream/event mechanism to extend:** the M3 NEO orchestrator
+  (`ggml/include/ggml-neo-pipeline.h` + the CUDA `graph_compute` hook at
+  `ggml-cuda.cu:4067-4191`) is the exact precedent. It already: registers an
+  op (`register_pair`), routes it to a non-default stream
+  (`curr_stream_no=1`, `cuda_ctx->stream(device,1)`), records a
+  `cudaEvent` (`cudaEventRecord`, 4189), and makes downstream consumers
+  `cudaStreamWaitEvent` on it (4099-4101). M7-D needs the *copy* analogue:
+  a `copy_stream`, `cudaMemcpyAsync(slot[i], host_kv_tile)`, a `copy_done[i]`
+  event the `compute_stream` waits on, and a `compute_done[i]` event so the
+  next tile's copy can reuse the slot — i.e. a per-slot ping-pong of the
+  same `cudaEventRecord`/`cudaStreamWaitEvent` primitives already in the file.
+- **Bandwidth for tile sizing:** the graph context already holds `cparams`
+  (it reads `cparams.neo_pipeline_mode` at 2306). So `cparams.pcie_bw_gbps`
+  (resolved from `pcie_profile_get().effective_bw_gbps` in
+  `common.cpp:common_context_params_to_llama`, threaded `cparams.h` →
+  `llama.h` → `llama-context.cpp` like `vram_target_mib`) is readable at
+  dispatch — **no kv-cache ctor plumbing needed**, the kernel sizes tiles
+  itself. This 4-file ABI thread lands *with* the kernel, not before
+  (cohesion; nothing reads it until the kernel exists).
+- **New op:** add `GGML_OP_STREAMING_FLASH_ATTN` to `ggml.h` + a CUDA
+  forward impl (the tile loop + online-softmax) + `supports_op` (CUDA-only;
+  CPU aborts, like 0042's fused-MoE). Slot pool (M7-C) allocates `n_slots`
+  GPU buffers of `tile_bytes_max` in the CUDA backend context, lifecycle
+  paired with the events.
+
+### Decomposition (each its own DSO rebuild + cosine gate)
+
+1. **R2-a — op + slot pool + dispatch, NO overlap.** New op, single
+   copy_stream, sequential tile loop with online-softmax, dispatched from
+   `get_layer_tactic(il)==GPU_STREAM`. Assign GPU_STREAM in
+   `build_rolling_kv_plan` when KV doesn't fit but `--vram-target` allows a
+   slot pool (else CPU_SPILL). Gate: cosine ≥0.999 at small ctx (GPU_STREAM
+   never assigned — inactive), and **correctness of the online-softmax at
+   large ctx** (the precision gate — vs CPU_SPILL output on the same prompt).
+2. **R2-b — double-buffer.** Add `copy_done`/`compute_done` events so tile
+   L+1's DMA overlaps tile L's compute. Pure latency; output must stay
+   bit-stable vs R2-a. Gate: same cosine + throughput delta.
+3. **R2-c — M7-E scheduler + load ramp.** Promote/demote tactics as
+   `--parallel` load ramps (Decision-4 dynamic residency). Gate: M7-G load
+   ramp + RULER vt@262144 (bug-226) + VRAM cap.
+
+The cparams `pcie_bw_gbps` thread + `get_layer_tactic` on the context land
+in R2-a (smallest cohesive unit that compiles + runs). M7-C's slot pool is
+folded into R2-a (it has nothing to hold before the op exists — see task #303).
+
+### R2-a actual-state + streaming-kernel refinement (2026-05-31)
+
+What **actually shipped** as R2-a (commits `7f73270`, `2489317`; canonical
+`0070`) is only the **op-dispatch shell**: `GGML_OP_STREAMING_FLASH_ATTN` exists,
+dispatches at `get_layer_tactic(il)==GPU_STREAM`, and its CUDA forward is
+`ggml_cuda_flash_attn_ext(ctx, dst)` **verbatim** — `n_slots == 0`, KV still
+device-resident, **no tile loop, no copy stream, no inter-tile online-softmax.**
+It is byte-identical to GPU_RESIDENT precisely because it is monolithic FA on a
+resident tensor; the "online-softmax retired by reuse" insight covers only
+`launch_fattn`'s **intra-tensor** `parallel_blocks` combine, *not* the inter-tile
+combine across host-streamed slots. So the precision core is **unwritten**.
+
+**The load-bearing architectural finding** (grounded: `build_rolling_kv_plan`
+`llama-kv-cache.cpp:2197`; dispatch `llama-graph.cpp:2372-2389`):
+
+- GPU_STREAM is assigned **iff `layers[j].gpu_heads > 0`** — i.e. exactly when the
+  layer already has an **M2 head-split**, so its spilled heads' K/V are **already
+  in pinned host** (`k_cpu`/`v_cpu`, allocated with the pinned `cpu_buft` at
+  `llama-kv-cache.cpp:461-528`). **There is no whole-layer-host residency reorder**
+  — Rolling KV *rides the existing M2 split*. The ctor-ordering issue
+  (`build_rolling_kv_plan` runs at `:598`, after buffer alloc) is therefore moot
+  for residency: the pinned-host half already exists before the plan runs.
+- Two combine axes, do not conflate them:
+  - **Head axis (M2):** device head-group vs pinned-host head-group are
+    *independent* outputs → combined by a plain **`ggml_concat`** (exactly what
+    `build_attn_mha_neo` already does, `llama-graph.cpp:2382-2389`). **No
+    online-softmax.**
+  - **Key axis (Rolling KV):** the host head-group's long-context KV is streamed
+    in **key-dimension tiles** to a device slot → FA per tile → merged with
+    **running max + denom in F32** (`flash_attn_combine_results`). **This** is the
+    precision-critical online-softmax; it engages only when the host head-group's
+    KV exceeds one slot (large ctx). One tile ⇒ inactive ⇒ byte-identical.
+
+So the real M7-D kernel is **`build_attn_mha_neo` with the CPU-half replaced by an
+op-owned, key-axis-tiled, async-DMA GPU FA**: stream the spilled head-group's
+host KV tiles into the slot pool (own `cudaMemcpyAsync` on a `copy_stream`),
+FA each tile on the `compute_stream`, online-softmax-combine across key tiles,
+then head-axis-concat with the device head-group's FA. This supersedes the R2-a
+"feed the full concatenated K/V to one streaming op" framing — that path only
+ever reproduced monolithic FA (the scheduler's sequential host→device copy of the
+concat is the very thing M7-D must replace).
+
+### Implementation decomposition, refined (S1 → S2 → S3, each a DSO rebuild + gate)
+
+- **S1 — structural split, byte-identical. ✅ DONE 2026-05-31.** Routed GPU_STREAM
+  to the **split** accessors (`get_k_gpu`/`get_v_gpu`/`get_k_cpu`/`get_v_cpu` +
+  `headinfer_gpu_heads`, like NEO), not `get_k`/`get_v`. `build_attn_mha_streaming`
+  restructured to the NEO shape: device head-group via resident `ggml_flash_attn_ext`
+  + host head-group via the `GGML_OP_STREAMING_FLASH_ATTN` op (forward still plain FA
+  at S1) + head-axis `ggml_concat`; no NEO pair-registration. **Slot pool + op-owned
+  DMA deferred to S2** — they have no correctness purpose at a single tile and belong
+  with key-tiling, which keeps S2 as pure precision math. Binary-only `:make` (op
+  forward untouched ⇒ no DSO rebuild). Gate **PASS**: GPU_STREAM
+  (`OPENCOTI_M7_STREAMING=1`) ≡ NEO byte-identical greedy decode on Qwen2.5-1.5B at
+  frac 0.5 (`GPU_STREAM=28` layers engaged; A==B==C all byte-identical). Owning the
+  path surfaced + fixed **bug-167**: the `--flash-attn auto` resolver
+  (`llama-context.cpp`) `GGML_ASSERT`ed on the M2/NEO/M7 split FA naming
+  `__fattn___g-<il>` (latent since M3 NEO) → rewritten to skip non-plain FA ops.
+- **S2 — key-axis tile loop + inter-tile online-softmax. ✅ DONE 2026-05-31
+  (byte-identical).** The open sub-design was resolved by a hard CUDA finding:
+  the **mma kernel our 3090 selects does NOT use `flash_attn_combine_results`** —
+  it normalises in-kernel and combines via the `stream_k` `dstk_fixup` path
+  (only tile/vec/wmma use the `parallel_blocks`/`dst_tmp_meta` machinery; fattn.cu
+  dispatches head_dim=128 to MMA on any `turing_mma_available` GPU, VEC fast-path
+  gated `cc ≥ ADA_LOVELACE`). So extending `launch_fattn` to reuse its combine was
+  invalid for our path. **Resolution (user-approved): LSE-weighted host-tile
+  combine.** Per key-tile, run the STOCK `ggml_cuda_flash_attn_ext` (untouched —
+  resident path byte-identical, every kernel family covered) for normalised
+  `O_t`, plus a small `streaming_lse_kernel` for per-row `lse_t = m_t + log d_t`;
+  merge in F32 via `O = Σ_t O_t·exp(lse_t − M)/Σ_t exp(lse_t − M)`, `M = max_t
+  lse_t` (exact online-softmax). Single-tile short-circuits to plain FA (= S1).
+  Lives in `fattn.cu` (`ggml_cuda_streaming_flash_attn` + 2 kernels), dispatched
+  from `ggml-cuda.cu`; env `OPENCOTI_M7_STREAM_TILES` (default 1). Gate
+  `.opencoti/s2-gate.sh`: T2(2-tile)/T4(4-tile) **byte-identical** to NEO ref at a
+  1969-token prompt (n_blk=8, genuine cross-tile), needle recalled, reap clean —
+  stronger than the cosine≥0.999 bar. Speed: zero penalty on any working path
+  (single-tile short-circuits; multi-tile only runs where resident can't hold KV;
+  the LSE pass reuses the already-streamed slot K → no extra DMA). NB: scoring
+  coverage is scale+mask only (max_bias==0, softcap==0, no sinks) → resident
+  fallback otherwise; gate model qwen2.5-1.5b satisfies all three.
+- **S3 (= blueprint R2-b) — double-buffer overlap.** Staged S3a → S3b → S3c so
+  correctness is gated before throughput, throughput before the memory win.
+  - **S3a — slot-pool machinery (DONE, 2026-05-31, byte-stable).** Per-key-tile
+    round-robin GPU slot pool (`OPENCOTI_M7_STREAM_SLOTS=N`); each tile's K/V is
+    lifted into a packed slot and FA reads the slot, proving the slot path is a
+    numeric no-op vs S2. **Single-stream by design**: copies issue inline as
+    `cudaMemcpyAsync(…, cudaMemcpyDefault, ctx.stream())` on the op's assigned
+    stream — NO dedicated copy_stream, NO events. This sidesteps the M2
+    `concurrent_events` stream collision (bug-253): at frac<1.0 under eager exec
+    the head-split forks work onto streams 1..n, and a hardcoded second stream
+    raced it, scrambling output; `ctx.stream()` auto-rides whatever stream the
+    op was forked onto. Zero events ⇒ CUDA-graph-capturable ⇒ the bug-251
+    graph-disable was reverted. Gate `.opencoti/s3a-gate.sh`: T2s1/T2s2/T4s2 all
+    **byte-identical** to T2(S2)==REF(NEO), needle recalled in all 5, reap clean.
+  - **S3b — double-buffer overlap (DONE, 2026-05-31, byte-stable).** Dedicated
+    copy_stream at fixed index `GGML_CUDA_MAX_STREAMS-1` (=7) carries tile L+1's
+    K/V DMA while FA/lse for tile L run on `ctx.stream()`, gated by a
+    pre-created `copy_done[]`/`compute_done[]` event pool (created ONCE with
+    `cudaEventDisableTiming`, never during capture — record/wait ARE legal during
+    capture, so NO graph-disable needed; the audit corrected bug-251 here). Index
+    7 avoids the QKV `concurrent_events` fan-out (streams 1-3, which JOIN before
+    our op runs) and NEO's stream 1 — hardcoding stream 1 was bug-253. Two latent
+    bugs surfaced and were fixed during S3b: **bug-255** (cross-op WAR hazard —
+    slot pool buffers reused across layers; copy_stream not ordered vs the compute
+    stream across op invocations → layer L+1's copy clobbered a slot layer L's FA
+    still read → prefill garbage; fixed with a per-op-entry `cs_sync` event
+    ordering copy_stream behind the compute stream) and **bug-254** (the shared
+    `streaming_lse_kernel` hard-casts K to `half*`; an S3b-0 stop-gap f16 guard
+    keeps any non-f16 K resident until S3d's dequant-on-lift lands — S3b stays
+    **f16-pure**). Gate `.opencoti/s3b-gate.sh`: B2s2 (overlap) and B4s2
+    (overlap + slot-reuse + prefetch) **byte-identical** to S3a==REF(NEO), needle
+    recalled; the `--parallel 2` P2 collision-stress is SOFT (warn-only) — its
+    collision axis is already covered by B2s2/B4s2 byte-identity, and its boot-
+    fail is a pre-existing llamafile graph-node-pool ceiling under `--parallel`
+    (bug-256), gated separately at M7-G. Regression shield green: NEO 4/4 hard,
+    advanced-kv-stack 8/9 (sole fail = pre-existing non-S3b C5, task #280).
+  - **S3c — scheduler bypass (the memory win, ✅ DONE 2026-05-31, byte-stable).**
+    On a discrete GPU the ggml scheduler can't run a `cuda_host` buffer as an op
+    input (`ggml_backend_cuda_device_supports_buft` → false), so it auto-inserts a
+    full sequential host→device copy of the pinned CPU-half KV before the streaming
+    op every step (`ggml-backend.cpp`), then rewrites the op's `src` to the device
+    dup — no memory win, and the S3a/S3b lift was device→device. S3c adds an
+    **op-scoped, env-gated** suppression of that copy (`OPENCOTI_M7_STREAM_BYPASS=1`,
+    keyed on `GGML_OP_STREAMING_FLASH_ATTN` + K/V src + host buffer) so the op's
+    `src` stays the pinned-host **permuted** view, and `fattn.cu` detects the host
+    pointer (`cudaPointerGetAttributes`), forces the slot path, and lifts each tile
+    with a **strided `cudaMemcpy2DAsync`** (`spitch = nb[1] = n_head_cpu·row`, since
+    the permuted host view's key rows are NOT contiguous, unlike the scheduler's
+    device dup). Device srcs (bypass=0) keep the exact 1-D lift → byte-identical
+    bisection lever. **The streaming op never drifts to CPU** (CUDA-only op type).
+    Gate (`.opencoti/s3c-gate.sh`): S3a==REF, B2s2==S3a, B4s2==S3a, **S3c==B4s2
+    byte-identical**, and the **no-copy-node proof** — under bypass=1 the scheduler
+    copy is suppressed for every layer's K/V (instrumentation count >0, all
+    `host=1`), under bypass=0 never (==0). NOTE: the CUDA0 compute-buffer PEAK is
+    insensitive to the suppressed ~CPU-half MiB at 4k (FA softmax scratch dominates
+    + allocator reuse — bug-257), so the steady-state VRAM win is measured at 256k
+    in M7-G, not at this gate. Regression shield: neo-pipeline 4/4, advanced-kv-stack
+    8/9 (sole fail = owned non-S3c C5, task #280 — the bench enables streaming on
+    zero configs, so the entire S3c path is structurally inert there). Still f16
+    (the bug-254 guard stays; dequant is S3d).
+  - **S3d — dequant-on-lift. ✅ DONE 2026-06-01, validated incl. n_head_cpu>1.**
+    The slot is now ALWAYS f16: a quantized K/V is dequantized into the slot on the
+    lift via `ggml_get_to_fp16_nc_cuda` (the exact primitive fattn-common.cuh uses;
+    strides in elements = nb/type_size, packed f16 output), keeping
+    `streaming_lse_kernel` type-agnostic and removing the bug-254 f16 stop-gap. Two
+    safety guards: an unsupported quant type (no nc converter) and a quant type with
+    no slot both stay resident. Gate (`.opencoti/s3d-gate.sh`, Qwen2.5-1.5B): f16
+    anchors byte-identical (S3a==REF, B2s2==S3a, B4s2==S3a, S3c==B4s2 — refactor inert
+    on f16), NOCOPY (896/0), **DEQUANT byte-identical to resident-q8_0** (streaming
+    q8_0 == resident q8_0). Council audit (7-agent) cleared the dequant stride math
+    against the stock resident reference. **n_head_cpu>1 validated** on Gemma-4 A4B
+    (8 KV heads → frac 0.5 = 4 GPU/4 CPU on 25/30 layers, all streaming): RULER vt@4k
+    q8_0 = **100%**, equal to resident-q8_0 ground truth and vanilla — so the
+    latent-stride fix (`Kt.nb[1]=k_row`, lse `k_nb1=k_row`) is load-bearing and
+    correct, not inert (`.opencoti/s3d-gemma-nhead-validate.sh`). **bug-226 reframed:**
+    the real opencoti defect was the concat_q block-scrambling, already fixed by 0035;
+    q8_0 works on this Gemma at coherence-checkable ctx (vanilla, resident, AND S3d
+    streaming all = 100% templated). The earlier "vanilla q8_0 collapse" was an
+    off-template (raw `/completion`) confound, retracted. The only open cell is **256k
+    q8_0 quality → M7-G** (deferred per the post-M7 + 256k-canary instruction); S3d is
+    NOT blocked on it.
+
+  - **E1 — stream-by-default auto-engage. ✅ DONE 2026-06-01 (E1a+E1b+E1c).**
+    Turns the dev-env-gated S-ladder into the product's auto-selected relief
+    tactic. **E1a (host):** `OPENCOTI_M7_STREAMING` is now tri-state — `"1"`→force
+    GPU_STREAM, `"0"`→force CPU_SPILL, **unset → GPU_STREAM** (stream-by-default for
+    any spilled layer); `frac=1.0` → no spill → all GPU_RESIDENT → vanilla
+    byte-identical (flip inert). **E1b (CUDA DSO + host):** the streaming op
+    self-configures — tile count `n_tiles=clamp(ceil(n_kv/2048),1,n_blk)` auto from
+    n_kv, `slots=2` default, overlap default-on at slots≥2, S3c scheduler-copy
+    bypass default-on (host-buffer-guarded). All knobs keep their env override as
+    bisection levers. **E1c (adapter):** `launch.ts` default-emits
+    `--headinfer-gpu-heads-frac auto` + `--vram-target`; config gains
+    `vramTargetMiB` (#332). Gate (`.opencoti/m7e-b-gate.sh`, Qwen2.5-1.5B):
+    RESIDENT==VANILLA + KAUTO==BASE + **FIDELITY AUTO==RESIDENT byte-identical**
+    (the auto streaming faithfully reproduces the no-split reference — the
+    pathological-prompt "degeneration" is correct, matching vanilla's own
+    filler-echo; the old "AUTO must diverge" gate premise was retracted) +
+    MULTI-TILE-ENGAGED (auto n_tiles up to 4) + DETERMINISM + NOCOPY (1400/0) +
+    needles incl. q8_0. Shields held: advanced-kv 8/9 (C9 = the #287 M2 slowdown
+    M7 exists to fix, not a regression), neo 3/4 (C2 = M3 "no observable win"
+    perf-noise; C4 off-path byte-identical passes). **Gemma-4 A4B n_head_cpu=4
+    auto path** (`.opencoti/m7e-gemma-auto.sh`): RULER vt@4k q8_0 = **100%**, equal
+    to vanilla + resident — the full-auto product path is coherent on the
+    bug-226-class multi-CPU-head split. bug-259 (streaming-op warmup/-fit abort
+    under stream-by-default) fixed: CUDA `supports_op` for STREAMING_FLASH_ATTN
+    validates head_dim only (the op repacks K/V to contiguous f16 slots, so raw-src
+    stride-sensitivity was wrong); host-side CPU-supports-op=false + pass-4 CUDA pin
+    kept as inert defense-in-depth. **E2 (dynamic load-ramp under `--parallel`) is
+    next** — deferred behind the M7-G 256k canary (needs runtime active-occupancy
+    infra the codebase lacks; const-after-ctor `layer_tactic[]`).
+
+## Open questions (resolution path during impl)
+
+- **Hot-tile reserve default** — does keeping 1-2 recently-used tiles
+  GPU-resident across tokens give measurable gain, or is the DMA cost
+  already negligible vs compute under the auto-tune? Decide at M7-C
+  first bench.
+- **Sub-tile online-softmax precision** — when running N
+  position-window sub-tiles per layer, the running max + denominator
+  accumulate in F32. Verify f32 accumulator precision holds at 256k
+  (probably fine; FA already does this internally).
+- **iswa interaction** — Gemma 4 uses interleaved sliding-window
+  attention. Some layers attend to a small recent window only. Those
+  layers may not need streaming at all (the working set fits in one
+  slot). Tactic table should detect and route iswa-window layers to a
+  trivial `GPU_RESIDENT` tactic — a third tactic alongside
+  `GPU_STREAM` and `CPU_SPILL`.
+- **PolyKV M6 composition** — if M6 ships first, M7 streams the
+  *compressed* tiles, decompressing in-slot before FA. Tile size math
+  changes (smaller bytes per slot, same compute time → bigger
+  effective tiles). M7's auto-adapt loop becomes
+  `tile_bytes_max × compression_ratio`. Plan to land M7 first if
+  bug-226 forces it; otherwise M6 → M7 is natural.
+
+## Risks
+
+- **Online-softmax across sub-tiles** is the trickiest math. FA
+  already does it within a kernel; we extend it across multiple
+  kernel invocations using the saved max + denom. One precision bug
+  here destroys output quality. Mitigation: keep the M0/M1/M2/M3/M5
+  regression-shield benches; gate ship on cosine ≥ 0.999 at small ctx
+  where the sub-tile path doesn't activate.
+- **Async DMA + ggml scheduler** integration — ggml's existing
+  scheduler doesn't expect "this tensor's data arrives mid-graph from
+  another stream." Need to register events as scheduler dependencies
+  or pre-stage at graph-build time. Risk: race conditions invisible
+  in single-tenant tests but visible under `--parallel`.
+- **Probe + boot ordering** — the auto-adapt loop reads PCIe topology
+  from the cached probe JSON. If the probe hasn't run (fresh install),
+  fall back to a `current_link_speed` read inline; if that fails, use
+  a conservative tile size (1 MiB) and warn. Never fail boot on
+  topology read.
+- **Re-tuning when VRAM availability changes mid-run** — if a sibling
+  process grabs VRAM, M7's slot allocations may need to shrink. v1
+  doesn't handle this; the existing `--vram-target` assumption is
+  static. Document as a known limitation.
+- **M2 flag removal is a user-visible break.** Anyone scripting
+  `--headinfer-gpu-heads-frac` will break. Ship a deprecation cycle:
+  M7 lands accepting both flags (frac → vram-target translation), a
+  release later the old flag warns, a release after that it errors.
+
+## Why land this above PolyKV M6
+
+M7 is the foundation for "KV doesn't have to fit on GPU." PolyKV
+adds compression to that. Without M7, PolyKV's compressed pool still
+has to fit on GPU; with M7, even an uncompressed pool can stream
+arbitrarily large KV. **M7 first lets PolyKV's compression be pure
+gravy on top of unlocked KV capacity, rather than a hard requirement
+to fit.** Sequence: bug-226 root cause → M7 design → M7 impl →
+PolyKV M6 (composed atop M7). Adjust if bug-226 turns out to need M7
+urgently; otherwise the optimization round (#287, tasks #289-#293)
+ships first as M7 prerequisites.
+
+## S0 cliff-mechanism verdict (2026-07-01) — graphs are NOT the cause; redirect #586/#587/#588
+
+Before building the pending rolling-KV perf work (#586 shared staging ring / #587
+cross-layer run-ahead / #588 perf gate), an S0 investigation nailed **what actually
+causes the overflow decode cliff** (resident ~30–46 tps → spill ~0.3–1.6 tps). The
+working hypothesis had been "a host-sourced tail tile disables CUDA-graph capture for
+the whole decode step → all `n_kv/32768` streaming tiles run as eager host-issued
+launches." **That hypothesis is REFUTED.**
+
+**Method:** the binary already emits a native per-slot metric `graphs reused = N`
+(no rebuild needed — this is the "read the target's native diagnostics first" lesson).
+A/B on the 3090 (Qwen3-8B-Q8_0, ctx 16384, ~15k-token prompt so the spill genuinely
+pushes occupied cells into the tail):
+
+| config | split | **graphs reused** | decode tps |
+| --- | --- | --- | --- |
+| resident (vt 20000) | FULLY RESIDENT | **23** | 41.99 |
+| spill (vt 2000) | 3072 resident / 13312 tail (1872 MiB) | **23** | 1.56 (27× collapse) |
+
+CUDA graphs are captured and **replayed 23× in BOTH cases** — the graph is *not*
+disabled by a host tail. So the ~640 ms/token penalty happens **inside the replayed
+graph**: it is real streaming-path *work*, not launch/eager overhead.
+
+**Mechanism (grounded):** a host tail flips the *entire* attention — including the
+device-resident window — from the fast dense-FA path onto the slow per-tile streaming
+path (`two_region` re-tiles window *and* tail, fattn.cu). Each tile pays a strided
+per-head copy (`cudaMemcpy2DAsync` ×`n_head_kv`×2) + the `cs_sync` barrier
+(fattn.cu:~1375) which serializes the copy stream against all prior compute — so tiles
+execute serially even under graph replay. Cost ∝ tile count (matches the earlier
+2-tile 632 ms vs 7-tile 3146 ms scaling), and the device-resident window pays the
+streaming tax it would NOT pay when fully resident.
+
+**Redirect (the levers are structural, none is "graph capture"):**
+- **#586 shared staging ring** — collapse the per-head `n_head_kv×2` strided copies
+  into one bulk transfer (cuts node/copy count).
+- **#587 cross-layer run-ahead** — drop/loosen the per-op `cs_sync` barrier so the
+  tail DMA overlaps compute instead of serializing (was mis-framed as "restore graph
+  capture"; the real target is the barrier).
+- **structural** — when only a tail spills, run the device-resident window as ONE
+  dense FA (the same fast kernel the fully-resident case uses) emitting its lse, tile
+  ONLY the tail, and do one online-softmax combine. Removes the streaming tax from the
+  window (the bulk of the KV); benefit ∝ window/tail ratio (largest when occupied
+  barely exceeds VRAM — the real serving regime; deep overflow stays bandwidth-bound).
+- **#588 perf gate** — re-run the 3090 cliff curve; success = spill decode within
+  ~10–15% of resident at small-tail (occupied ≈ VRAM) configs.
+
+See bug-1342 (0089, the boot OOM that unblocked this curve) and bug-2094 (0090, the
+`--parallel ≥2` multi-session boot fix found while probing tail amortization).
+
+### Multi-session amortization (2026-07-01) — the cliff does NOT amortize under --parallel
+
+With bug-2094 (0090) unblocking `--parallel ≥2` boot, the original question — does batching
+hide the tail penalty? — was answered on bs2 (RTX PRO 6000, 14B-1M-Q8_0, controlled per-slot
+window: p2 vt=2×p1 vt so each slot's window ≈ equal):
+
+| config | host tail | per-slot decode tps | aggregate |
+| --- | --- | --- | --- |
+| p1 (1 session, vt 8000) | 31232 cells | 1.08 | **1.08** |
+| p2 (2 sessions, vt 16000) | 27136 cells/slot | 0.28 + 0.66 | **0.94** |
+
+Two concurrent spilled sessions aggregate to **less** than one (0.94 < 1.08 tps) — batching
+does not recover throughput; the sessions **contend on the same serialized streaming path**
+(shared copy stream + the per-op `cs_sync` barrier), and per-slot tps is asymmetric (0.28 vs
+0.66) from uneven interleave on that one copy stream. **Consequence:** for the multi-session
+serving target ("more users per card"), spilling collapses per-card throughput regardless of
+concurrency, so the #586/#587 streaming-path rework (kill the serialization) is **critical**,
+not a nice-to-have.
+
+## 2026-07-05 — bug-1843 shipped (patch 0098): the spill-decode recompute is dead
+
+The S0-cliff redirect (#586 ring / #587 barrier-drop) proved perf-inert
+(bug-1838); the real lever was the **per-tile `streaming_lse_kernel`
+recompute** that every `head_dim<=256` decode tile paid because `decode_lse`
+was gated on `head_dim > 256` (bug-1843, ~80% of spill-decode wall — 1.14 vs
+6.05 tps NOOP A/B, Qwen3-8B ctx40960/vt14000, 3090). Patch
+`0098-rolling-kv-lse-decode` arms the Stage-3a LSE channel for D≤256 at all
+three streaming-FA sites and recomputes only when
+`opencoti_fattn_dst_lse_written` says no finalize ran.
+
+Shipped gates (3090, DSO df46dc36): spill decode **1.14 → 6.04 tps (5.3×, ==
+NOOP ceiling)** with needle PRESENT; resident 42.5 tps unchanged;
+`OPENCOTI_LSE_NOOP` now speed-inert (6.02 — recompute structurally off the
+path). Correctness: teacher-forced logit-equiv (Qwen3-8B D128,
+ours-resident REF vs POSITION_WINDOW at 256/16384 cells resident):
+real_frac=0.0, frac_full_agree=1.0, mean_tv 0.0068 → PASS. Follow-up: #588
+re-runs the 3090 overflow→tps cliff curve on the fixed binary (go/no-go for
+further rolling-KV investment). Note bug-2115 while gating: bs2-built DSOs
+do NOT load on solidPC (glibc ≥2.38 vs Debian 11) — build the 3090 DSO
+locally or the run silently falls back to CPU.
+
+## 2026-07-06 — #588 perf gate: post-fix cliff curve — spill decode is at the PCIe floor (rolling-KV perf line CLOSED)
+
+Curve on the bug-1843-fixed DSO (`df46dc36`, 3090, Qwen3-8B-Q8_0, ctx 16384,
+~13.1k-token prompt = 2304 MiB f16 KV, needle PRESENT in every cell):
+
+| vt (MiB) | window split | occupied tail | decode tps | ms/token |
+| --- | --- | --- | --- | --- |
+| 22000/16000/14000/13000 | FULLY RESIDENT | 0 | 42.3–42.5 | 23.6 |
+| 12000 | 11008 / 16384 cells | ~2.1k cells ≈ 295 MiB | 15.5 | 64.6 |
+| 8000/4000/2000 | 256 / 16384 (floor) | ~12.9k cells ≈ 1815 MiB | 3.33 | 300 |
+
+**The PCIe model now fits within ~5%:** `t(token) ≈ 23.6 ms + occupied_tail_bytes
+/ 6.5 GB/s` (vt12000 predicted 67 ms vs 64.6 measured; floor predicted 303 ms vs
+300). The S0 anchor cell (vt 2000) improved 1.56 → 3.33 tps and the former
+compute-side ceiling (streaming_lse recompute) is gone — spill decode is now
+purely bandwidth-bound, i.e. at the structural floor for a design that re-streams
+the tail every token.
+
+**Gate verdict: NO-GO for further re-stream perf work.** The "within 10–15% of
+resident at small tails" bar is only reachable for tails ≤ ~25 MiB (~170 cells)
+on a 6.5 GB/s 3090 (~8× more on bs2's 50 GB/s link) — no copy-plumbing lever
+(#586-class) can beat the link itself. What remains valuable:
+
+- **Window mode as graceful-overflow fallback** (shipping default): linear
+  degradation ∝ spilled bytes, correctness clean (needle + logit-equiv
+  real_frac=0), no cliff pathology left.
+- **The real capacity levers are residency-side**, per #582: quant-KV decode
+  reclaim (#620), auto KV-tier boot policy (#621), and position-axis mixed KV
+  with a *compressed VRAM tail* (#622) — which avoids the per-token re-stream
+  entirely instead of optimizing it.
+
+## 2026-07-06 — bs2 tail-spill curve (the real-bandwidth one): bug-2116 bulk-H2D unlocks the 50 GB/s link
+
+The #588 verdict above is on the **3090's 6.5 GB/s link** — where the tail is so
+bandwidth-starved that no plumbing lever can help. bs2 (RTX PRO 6000, ~50 GB/s
+link) is the host where window-mode spill is actually *usable*, and it is also the
+host where the **bug-2116 bulk-H2D fix** (batched staging vs per-tile serial
+`cudaMemcpyAsync`) matters most. Curve on **Qwen2.5-14B-Instruct-1M Q8_0**, ctx
+131072, ~106k-token prompt (KV 9984 MiB f16-resident), needle PRESENT in every
+cell. Both curves share the same VT→window-split mapping, so they are directly
+comparable against the real host-tail byte count:
+
+| host tail | BEFORE (per-tile H2D) | AFTER (bug-2116 bulk-H2D) |
+| --- | --- | --- |
+| 0 (fully resident) | 15.88 tps / 63.0 ms | 15.91 tps / 62.85 ms |
+| 1443 MiB | 13.19 / 75.8 | 13.17 / 75.95 |
+| 2028 MiB | **9.05** / 110 | **12.23** / 81.79 |
+| 2828 MiB | **3.96** / 253 | **10.28** / 97.31 |
+| 7624 MiB | (already collapsed by 2.8 GiB) | 5.53 / 180.84 |
+
+**The fix converts launch-bound into link-bound.** BEFORE, the deep-tail slope is
+~5.5 GB/s effective (2028→2828 MiB costs +142 ms) — i.e. even on bs2's 50 GB/s
+hardware the *serial per-tile launches* throttle spill to 3090-class bandwidth,
+and the curve cliffs at ~2 GiB. AFTER, the deep slope is **~56 GB/s** (2828→7624
+MiB costs +83.5 ms for 4.68 GB) — matching bs2's physical link — so at a 2.8 GiB
+tail you keep **10.28 tps = 65% of resident** (vs 3.96 = 25% before), and a 7.6 GiB
+tail still decodes at 5.53 tps. This is the measured confirmation of the "~8× on
+bs2's link" projection in the #588 section, and it makes window mode a genuinely
+usable graceful-overflow tier on the production host, not just a non-cliff fallback.
+
+Data: `bs2:/srv/ml/opencoti-c1/tailcurve-out/` (BEFORE) and `tailcurve-after-out/`
+(AFTER); decode tps read from `slot print_timing … eval time` in the per-cell
+`tailN.log` server logs (the AFTER driver never aggregated a `run.log` beyond
+tail=200 — the numbers above are recovered from the server logs). **Two AFTER
+cells are truncated** (host tail 1638 MiB / 4426 MiB — run cut off; the log has
+prompt-processing but no eval line); they will be re-run to complete the curve
+when GPU1 frees from the bug-2121 window-mode correctness gate (which exercises
+this same spill path). The AFTER slope is already pinned by the 2828→7624 MiB
+segment, so the verdict does not depend on the two missing points.
+
+### 2026-07-06 — multi-model spill sweep: production KV is compact at 128k (spill barely triggers)
+
+Extending the bs2 curve to four production targets (Gemma-4-31B Q6_K, Gemma-4-A4B-128e
+Q4_K_M, Qwen3.6-27B-Omnimerge Q6_K, Qwen3.6-35B-A3B Q6_K), q8_0-K/q4_0-V, ctx 131072,
+surfaced a physical finding **before** any bandwidth number: at 128k these models simply
+do not have enough KV to spill. Measured resident KV (fully on-GPU) and fully-resident
+decode tps:
+
+| model | attention | resident KV @131072 | resident decode tps |
+| --- | --- | --- | --- |
+| Qwen2.5-14B-1M Q8_0 | full | **9584 MiB** | 15.9 |
+| Qwen3.6-27B-Omnimerge Q6_K | full | 3328 MiB | 29.8 |
+| Qwen3.6-35B-A3B Q6_K | full (MoE) | 1040 MiB | 89.4 |
+| Gemma-4-31B Q6_K | **iSWA** | **488 MiB** (global only) | 25.6 |
+| Gemma-4-A4B-128e Q4_K_M | **iSWA** | (small global) | — |
+
+Two structural reasons the production models barely spill at 128k, both of which the
+14B-1M (the original curve) side-steps: **(1) GQA compactness** — the Qwen production
+quants have few KV heads, so full-attention KV is only 1–3.3 GB at 128k vs the 14B-1M's
+9.6 GB (Q8, more KV heads); **(2) iSWA** — Gemma-4 (both A4B and 31B) keeps only ~1/6 of
+layers global, so the cache that *can* spill is tiny (31B global = 488 MiB). On a 96 GB
+card none of these overflow at 128k, so a natural spill only appears at far longer
+context (256k–1M for the Qwen full-attn models; iSWA Gemma effectively never spills its
+global cache at usable context).
+
+A follow-up sweep (`tailcurve-multi2`) forced spill by capping `--vram-target` below
+(weights + KV), weights measured from `nvidia-smi`, tail ladder = 0/25/50/75/90 % of each
+model's KV. Results (bs2, DSO `8b23afbf`, q8_0-K/q4_0-V, ctx 131072, needle PRESENT in
+**every** cell):
+
+**14B-1M — the one clean forced-spill curve** (full-attn, 9584 MiB KV):
+
+| host tail | decode tps | ms/tok |
+| --- | --- | --- |
+| 1019 MiB | 13.16 | 75.97 |
+| 3515 MiB | 9.36 | 106.87 |
+| 6011 MiB | 6.46 | 154.83 |
+| 8507 MiB | 5.03 | 198.80 |
+| 9984 MiB (all KV on host) | 4.35 | 229.62 |
+
+Deep slope (6011→9984 MiB) = **51.9 GB/s**, i.e. the bs2 host link — the same result as the
+`tailcurve-after` curve above, reproduced by an independent forced-spill trigger. Window mode
+degrades linearly and holds retrieval to the fully-spilled floor.
+
+**The four production targets do not give a spill curve at 128k**, for two distinct reasons:
+
+- **iSWA (Gemma-31B, A4B):** the *global* cache — the only part window mode spills — is just
+  **1536 cells (~5 MiB)**. Every forced-spill cell shows `256 / 1536 resident`, decode stays
+  flat (31B ~21 tps, A4B ~38 tps), needle PRESENT. Spill is a non-event; the sliding-window
+  layers never leave VRAM. **iSWA models effectively cannot be made to spill** at any usable
+  context.
+- **Compact-GQA full-attn (Qwen-27B 3328 MiB, 35B-A3B 1040 MiB):** KV fits so easily that the
+  calibrated low-VT still didn't drop below (weights+KV) — all cells stayed FULLY RESIDENT
+  (flat 29.8 / 89.4 tps). Forcing them to spill would need a hardcoded sub-weights VT; at 128k
+  they simply have no overflow. A genuine spill on these needs 256k–512k context (bigger KV).
+
+(A first attempt, `tailcurve-multi`, was discarded — its calibration read the high-VT budget
+as free-VRAM → negative targets → BOOT-FAILs; only the resident-tps column above survived it.)
+
+**Bottom line for the four requested models:** at 128k only the 14B-1M (large Q8 KV) spills,
+and it hits the ~52 GB/s link exactly like the reference curve. The production targets either
+*can't* spill (iSWA global cache is ~5 MiB) or *don't* (compact GQA KV fits) — so window mode
+is a no-op safety-net for them at this context, and real spill characterization would require
+256k–512k runs.
+
+## 2026-07-12 — #582-P2 SHIPPED: position-axis mixed KV (compressed tail, `-ctkt`/`-ctvt`)
+
+The window-mode spill above re-streams the tail every token at *the boot KV type*, so a big
+Q8 tail is both slow (bytes) and redundant (older tokens are attention-light). **P2 makes the
+spilled tail carry a MORE-compressed type than the resident window** — the position-axis
+complement to P0 (head-axis, #620/#643) and P1 (boot-tier auto-select, #621). Recent tokens
+in the window `[0,wc)` keep their high-fidelity type on-device; the tail `[wc,n_kv)` on pinned
+host is quantized further, so each decode step ships fewer H2D bytes without touching the part
+of the cache attention actually leans on.
+
+**Interface.** Two new CLI flags set the *tail* K/V type independently of the window:
+`-ctkt <type>` / `-ctvt <type>` (`--cache-type-k-tail` / `--cache-type-v-tail`). Unset →
+sentinel `GGML_TYPE_COUNT` → "tail == window", and every code path is byte-identical to the
+shipped uniform-window behaviour (regression leg of the gate, `REG==U8` proven byte-identical).
+Typical use: `-ctk q8_0 -ctv q8_0 -ctkt q4_0 -ctvt q4_0` (q8 window ⊕ q4 tail).
+
+**Where the win lands (decode).** The POSITION_WINDOW streaming FA op (`fattn.cu`) already
+reads window and tail *each in-register at its own type* (the #620/#629 reader path) — there is
+no whole-cache f16 materialise, so a q4 tail is genuinely ~half the H2D bytes of a q8 tail.
+
+**Two correctness-fallback fixes** were needed for the paths that *don't* go through the
+in-register decode op (both host-only, both byte-identical when tail type == window type):
+
+- **bug-2161** — prefill / graph-reserve (n_q > `WS2_NQ_MAX`) reassembles window+tail via
+  `ggml_concat`, which asserts `a->type == b->type`. Fixed with a file-local
+  `poswin_lift_to_f16` (mirror of `dca_lift_to_f16`: scalar quants → `ggml_cast` F32 → F16,
+  byte-exact) applied to both region views only when their types differ.
+- **bug-2162** — an in-memory **context checkpoint** (llama.cpp PR16391, on by default) hits
+  the Stage 3c-6 state-IO window paths, which had assumed the tail row size equals the window
+  row size — so a mixed config asserted on the 2nd prompt. Fixed with a per-region
+  `ggml_row_size(k_cpu->type / v_cpu->type, …)`, symmetric across writer and reader.
+
+**Gate (S7, bs2 Qwen2.5-14B-Instruct-1M-Q8_0, `-c 24576` `--vram-target 18000`, ~21k-token
+prompts → wc ≈ 13312 resident, ~11264-cell host tail; logit-equiv vs the model's own
+full-resident gold — NEVER greedy needle):**
+
+| config | tail type | correctness (real_frac) | advisory mean-TV vs gold | decode tps |
+| --- | --- | --- | --- | --- |
+| U8 (uniform q8 tail) | q8_0 | 0.0 | 0.0069 | 26.4 |
+| **MIX (q8 window ⊕ q4 tail)** | q4_0 | **0.0** | **0.0110** | **28.1** |
+| U4 (uniform q4) | q4_0 | 0.0 | 0.0118 | — |
+
+Mixed sits **strictly between** uniform-q8 and uniform-q4 on fidelity (closer to gold than
+uniform-q4) at **+6.4% decode tps** over uniform-q8 �� the tail's fewer H2D bytes convert
+directly to throughput, and the recent window keeps quality above the uniform-q4 floor. The
+gate exercises checkpoints, so it doubles as a bug-2162 checkpoint round-trip test.
+
+Shipped as patch `0120-p2-mixed-kv` (marker `opencoti-hook: P2 mixed-kv`, 12 nested-llama.cpp
+files, additive; chain state → 66 patches). See UPSTREAM_SYNC.md registry entry for the full
+5-point plumbing surface.
+
+### 2026-07-12 — #653 P2 productized: TS adapter flags + P2-auto boot policy
+
+Two follow-ons turn the raw `-ctkt`/`-ctvt` mechanism into a first-class, opt-in
+opencoti feature.
+
+**Part 1 — adapter (`packages/opencoti-llamafile`).** The mixed-tail flags are now
+typed config fields, so a caller sets them like any other KV knob instead of hand-passing
+`extraArgs`:
+
+- `config.ts` — `ctypeKTail` / `ctypeVTail: string | undefined` (env
+  `OPENCOTI_LLAMAFILE_CTYPE_K_TAIL` / `_V_TAIL`), defaulting to `undefined` = server
+  default = "tail == window".
+- `launch.ts` — `buildServerArgs` emits `-ctkt`/`-ctvt` only when set, ordered right
+  after `-ctk`/`-ctv` (the window types they refine), before `--neo-pipeline`. Unset →
+  nothing emitted → byte-identical argv.
+- `test/launch-args.test.ts` — 4 new cases (omit-when-unset, symmetric q4_0 tail,
+  independent side, ordering); suite 49 pass / 0 fail.
+
+**Part 2 — P2-auto boot policy (patch `0121-p2-auto-tail`).** Extends the P1 boot
+auto-tier so that when it is *forced to spill* (the auto-selected window still overflows
+the VRAM budget), it can auto-pick a compressed tail rather than re-streaming the whole
+tail at the window type. Policy is **"q4_0 floor when window > q4_0"**: per axis, a
+window higher-bit than q4_0 gets a q4_0 tail; a window already at/below q4_0 keeps
+tail == window. It fires only on the two spill exits of `opencoti_auto_select_kv_tier`
+(never on the fully-resident fast path), and an explicit `-ctkt`/`-ctvt` is always honored.
+
+Gate opt-in is **separate from P1** — `OPENCOTI_KV_AUTO_TIER_TAIL=1` on top of
+`OPENCOTI_KV_AUTO_TIER=1` — so a P1-only boot stays byte-identical:
+
+| leg | envs | tail line | correctness |
+| --- | --- | --- | --- |
+| P1AUTO | `OPENCOTI_KV_AUTO_TIER=1` | *(none — tail == window)* | `real_frac 0.0` |
+| **P2AUTO** | `+ OPENCOTI_KV_AUTO_TIER_TAIL=1` | `auto KV tail = q4_0/q4_0` (~189 MiB / 3072-cell spill) | `real_frac 0.0`, mean-TV 0.0087 |
+
+The window was `q5_1/q4_0` at the **second-pass binding budget** — the bug-1342 two-pass
+compute reserve runs the auto-tier twice; the first (looser) pass fits, the binding pass
+spills, and the tail policy fires there. HOST-ONLY (`src/llama-kv-cache.cpp` is libllama,
+not the CUDA DSO). Marker: reuses the 0116 `opencoti-hook: kv-auto-tier` site (like 0118,
+no new hook). Chain state → 67 patches.
+
+## 2026-07-14 — bug-2172 SHIPPED (patch 0125): rolling-KV window on a HYBRID attention sub-cache
+
+Until now the position-window armed cleanly only on caches whose cell placement is
+**append-only** — dest cell idx == key position, monotone in batch order (every dense / iSWA
+/ unified `llama_kv_cache`). A **hybrid** model (qwen35moe / qwen3next: gated-delta-net
+recurrent ⊕ full-attention ⊕ MoE ⊕ NextN) routes its attention sub-cache (`mem_attn` inside
+`llama_memory_hybrid`) through the **recurrent** batch pipeline (`split_seq` +
+`[TAG_RECURRENT_ROLLBACK_SPLITS]` + `find_slot` cell-reuse/wrap), which hands the scatter
+**non-monotone** destination idxs. Under tight VRAM / large ctx the window armed and the
+append-only scatter's monotone `GGML_ASSERT` in `cpy_k` fired at boot.
+
+**Root cause** — not a bandwidth or a residency bug (those were #2142 / #2144); the append-only
+window scatter *assumed* a contiguous window-prefix / tail-suffix partition of the batch, which
+a recurrent-rollback batch violates.
+
+**Fix (host-only, one TU + two one-liners; NO CUDA / NO DSO rebuild):**
+
+- New `llama_kv_cache::append_only` flag (default **true** — every existing cache is
+  byte-identical). Set **false** only by `llama_memory_hybrid` on its `mem_attn`.
+- `!append_only && windowed`: window + tail tensors get **+1 trash sink cell**; the idx
+  tensors become **2-row `[n_tokens,2]`** (row 0 window-route, row 1 tail-route, off-target →
+  trash); `cpy_k`/`cpy_v` scatter the full per-stream K/V into **both** regions via two
+  arbitrary-dest `ggml_set_rows` (drops the monotone assert). `ne[0]` stays `n_tokens`
+  (graph guards hold); fixed 2-set_rows/stream topology (reserve == live).
+- **Read side unchanged** — `get_k/v_window` / `get_k/v_tail` already cover every active
+  scattered cell, and the streaming-FA `KQ_mask` `-INF`-masks the rollback holes by absolute
+  key position. **Mask contract verified**: `GGML_OP_STREAMING_FLASH_ATTN` is per-cell,
+  absolute-key-indexed, softmax-denominator-normalized ⇒ no kernel change.
+- **Scope**: iSWA-hybrid out of scope (no current model is both hybrid AND SWA; the window
+  never arms on `swa_type!=NONE`).
+
+**Validated** (solidPC host `make`): qwen35moe auto-spill boots with **no assert** + correct
+output; dense (Qwen) + iSWA (Gemma) regression shields **byte-identical** (append_only path
+untouched). Marker: `opencoti-hook: hybrid-window-orderagnostic` (3 sites). Chain state → 71
+patches.
+
+**RYS × rolling-KV-window composition confirmed** (2026-07-14, solidPC 3090,
+Qwen_Qwen3.6-35B-A3B-Q3_K_M, `--repeat-layers 5,9 --kv-residency-mode window
+--vram-target 18000 -c 40960 -fa on`): RYS ENGAGED (40 → **44 effective layers**, +4
+duplicated) **and** the window armed on the hybrid attention sub-cache (256 resident / 40704
+host-tail = 874 MiB) **simultaneously**. A 946-token needle prompt (needle sitting in the
+~690-cell spilled tail) → greedy-correct retrieval (`74-ALPHA-purple-9931`), **no assert / no
+NaN**, 16 graphs reused, clean slot release (pp 1452 tps / tg 57.5 tps). The two features that
+each stress the hybrid attention cache — RYS's extra duplicated KV layers and the
+order-agnostic window scatter — compose without interference.

docs/features/rolling_kv_compute_reserve.md

@@ -0,0 +1,126 @@
+# Rolling-KV: two-pass compute-buffer reserve (fix the resident-allocation OOM)
+
+**Status:** SHIPPED 2026-07-01 as `0089-rolling-kv-compute-reserve.patch` (9 files,
+byte-identical-verified). Host-only change (`llama-kv-cache*`, `llama-context.cpp`,
+`llama-model.cpp`, `llama-cparams.h`, `llama-memory-hybrid{,-iswa}.cpp`) →
+`build:llamafile:make` rebuild ONLY (no CUDA DSO, no restamp). Vendor backup:
+`.opencoti/vendor-backups/20260701-123044-two-pass-compute-reserve`.
+
+**Gate result (bs2, RTX PRO 6000 96 GB, 2026-07-01):** the exact repro config below
+(`-c 524288 --kv-residency-mode window --vram-target 88000 -ctk f16 -ctv f16`, KV
+~96 GB) that **OOM'd the pre-0089 binary at `graph_reserve` now BOOTS** and prefills
+with the streaming window engaged — prefill tps falls 4000→~450 tok/s as the host tail
+grows (the expected cliff, per `docs/features/rolling_kv.md`). Two-pass gate conditions
+(`offload && flash_attn && !devices.empty() && kv_residency_mode != head`) are all met on
+this path, so boot-success ⇒ the measured reserve replaced the too-small 1536 MiB fixed
+default. Streaming-window FA math is unchanged (size-only patch), correctness carried by
+the `0070` M7 logit-equiv/RULER gates.
+
+## The bug (bug-1342)
+
+The POSITION_WINDOW resident-window sizer
+(`opencoti_compute_resident_window_cells`, llama-kv-cache.cpp:227) reserves a
+**fixed** `OPENCOTI_HEADINFER_AUTO_RESERVE_BYTES = 1536 MiB` (line 120) for the
+compute scratch that `graph_reserve` allocates AFTER the KV cache. But the
+compute buffer **scales with context** — measured **3.63 GiB (3 800 045 568 B)
+at 262 144–524 288 ctx** on the RTX PRO 6000. So the sizer keeps ~2.1 GiB too
+much KV resident and `graph_reserve` OOMs:
+
+```
+ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 3800045568
+graph_reserve: failed to allocate compute buffers
+llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
+```
+
+Repro: RTX6000, Qwen2.5-14B-1M Q8_0, `-c 524288 --kv-residency-mode window
+--vram-target 88000 -ctk f16 -ctv f16`. 128k boots (KV ~25.7 GiB fits); 512k
+(KV ~96 GiB) does not.
+
+The author KNEW (comment at llama-kv-cache.cpp:114-119): "this reserve only
+covers the not-yet-allocated compute buffers. --vram-target gives the operator
+a tighter knob when this default is wrong." The fix removes the manual-guess
+dependency.
+
+## Fix: two-pass pre-measure (user-chosen 2026-07-01)
+
+Model-agnostic: MEASURE the real compute buffer, don't estimate it.
+
+- **Pass 1 (measure):** `create_memory` with a *measurement flag* → the sizer
+  returns a MINIMAL window (`walign` cells resident, rest host) → tiny device
+  KV → boots for ANY compute size. Run `sched_reserve()` (creates the sched at
+  llama-context.cpp:525, runs the pp worst-case reserve at :697). Then read the
+  actual GPU compute buffer: sum `ggml_backend_sched_get_buffer_size(sched, b)`
+  over backends whose `ggml_backend_dev_type(...)==GPU`.
+- **Pass 2 (real):** re-`create_memory` with `kv_compute_reserve_mib = measured
+  + safety(256 MiB)`. Sizer now sizes the window against `budget − model −
+  measured_compute`. Set `sched_need_reserve=true`; `sched_reserve()` again.
+- **Gate the two-pass** so the common resident case stays SINGLE-pass /
+  byte-identical: only engage when `offload && kv_residency_mode != head(1) &&
+  full-KV-demand + a generous compute margin > free_vram` (overflow possible).
+  Reuse the `kv_demand` vs `free_vram` compare already in the sizer — expose a
+  cheap `bool opencoti_kv_may_overflow(...)` helper. If it can't overflow →
+  today's single pass, no behavior change.
+
+Compute buffer size is INDEPENDENT of the window/tail split (it depends on
+n_ubatch × n_kv × n_embd, and n_kv = full ctx regardless of resident split), so
+the Pass-1 minimal-window measurement equals the Pass-2 compute — one
+measurement suffices.
+
+## Exact edit surface (threading map, verified 2026-07-01)
+
+1. **llama-cparams.h** (struct at :9): add
+   `uint32_t kv_compute_reserve_mib;` (0 = use 1536 default) and
+   `bool kv_window_measure_pass;` (default false).
+2. **llama-context.cpp**:
+   - Init defaults near :104-108 (beside `cparams.vram_target_mib` /
+     `cparams.kv_residency_mode`): `kv_compute_reserve_mib=0`,
+     `kv_window_measure_pass=false`.
+   - Two-pass orchestration wrapping the `memory.reset(model.create_memory(...))`
+     at :385 + the `sched_reserve()` at :451. Backend-init block (:388-449) is
+     memory-independent — keep it before the first `sched_reserve`. After the
+     Pass-1 reserve, sum GPU compute buffer, set
+     `cparams.kv_compute_reserve_mib`, clear `kv_window_measure_pass`,
+     `memory.reset(create_memory(...))` again, `sched_need_reserve=true`,
+     `sched_reserve()`. Guard whole block behind `opencoti_kv_may_overflow`.
+3. **llama-kv-cache.h** (ctor decl :96-145): add `uint32_t
+   compute_reserve_mib` and `bool window_measure_pass` args (beside
+   `vram_target_mib` :123 / `kv_residency_mode` :133).
+4. **llama-kv-cache.cpp**:
+   - `opencoti_compute_resident_window_cells` (:227): take `compute_reserve_mib`
+     + `window_measure_pass`. Replace the fixed
+     `OPENCOTI_HEADINFER_AUTO_RESERVE_BYTES` at :271-273 with `compute_reserve_mib
+     ? (size_t)compute_reserve_mib<<20 : OPENCOTI_HEADINFER_AUTO_RESERVE_BYTES`.
+     If `window_measure_pass` → early-return the minimal window (`walign`, using
+     the same align math at :296-298) so Pass-1 device KV is ~0.5 MiB.
+   - ctor (:316): accept + forward the 2 args to the sizer call at :396.
+   - (optional) expose `opencoti_kv_may_overflow` (reuse the auto-frac budget
+     math) for the context-level gate.
+5. **llama-kv-cache-iswa.{h,cpp}**: thread the same 2 args through the wrapper
+   to both inner `llama_kv_cache` ctors.
+6. **llama-model.cpp** create_memory (:2082 iswa, :2113 plain): pass
+   `cparams.kv_compute_reserve_mib`, `cparams.kv_window_measure_pass` alongside
+   `cparams.vram_target_mib` (:2098/:2128) and `cparams.kv_residency_mode`
+   (:2102/:2132).
+
+## Gates
+
+- **Boot repro (the fix):** the exact 512k config above now BOOTS with a host
+  tail (was OOM). Read the logged `position window = C / kv_size cells resident`
+  + host tail; decode tps > 0, needle held.
+- **Resident unchanged (no regression):** a config that fits resident (e.g. 128k
+  or `--vram-target` high) still logs FULLY RESIDENT, single-pass, and decodes
+  at the same ~38 tps — byte-identical path (measure gate not tripped).
+- **Curve enabled:** with boot fixed, the tps-vs-host-tail curve (#583/#584 on
+  3090) is runnable on RTX6000: fix a ctx, sweep `--vram-target` down to grow
+  the tail, read (host-tail MiB, decode tps). NOTE the binary must LOG the
+  device/host split — the `position window = ...` INFO line at
+  llama-kv-cache.cpp:310 is the measurement source (the bench's grep must match
+  THAT line, not a non-existent `CUDA_Host KV buffer` line).
+
+## Constraints (standing)
+
+Host rebuild only (`bun run build:llamafile:make` from repo, or the make path) —
+NOT a CUDA DSO rebuild, NO restamp (llama-kv-cache/llama-context are host TUs).
+Additive soft-fork + `opencoti-hook:` markers (two-pass reserve) + UPSTREAM_SYNC
+registry. Correctness via boot + niah, never greedy byte-equality. Commit only
+when asked (dev; trailers). Vendor-backup already taken (above).

docs/features/rolling_kv_step_prefetch.md

@@ -0,0 +1,194 @@
+# Rolling-KV: step/graph-level tail prefetch (design)
+
+**Status:** DESIGN (2026-07-01). Supersedes the op-level double-buffer's overlap
+ambition. Gates on user go-ahead — this is an M7-scale CUDA effort (full host
+rebuild + dual-DSO restamp + logit-equiv/RULER-niah gate).
+
+## Problem (measured, 3090)
+
+POSITION_WINDOW streams the host KV tail over PCIe **every decode step**, and the
+tail DMA is ~100% un-hidden → decode collapses on a smooth hyperbola at the raw
+PCIe rate (~6.4 GB/s, `+0.156 ms/MiB`). Confirmed on both model classes:
+
+| model | attn | streaming layers | overflow decode | note |
+|---|---|---|---|---|
+| Gemma-4-A4B-128e | iSWA | ~5 global | 71→19.5 tps @236 MiB tail | graceful hyperbola |
+| Qwen2.5-14B-1M | full | 48 | 26.9→0.86 tps @~1.2 GB tail | ~10× worse (48 streaming layers) |
+
+Qwen is **not** CPU_SPILL and **not** ineligible — it engages POSITION_WINDOW
+(GPU 60–84% busy, CPU ~2.7 cores during overflow decode). It's slow purely
+because full-attention streams a tail on *every* layer.
+
+## Root cause (code)
+
+`fattn.cu` streaming forward runs a per-op, 1-tile-ahead ping-pong: `lift(t+1)`
+on `copy_stream` while tile `t` computes on the main stream. At decode (`n_q=1`)
+one attention op's compute is a sub-ms VRAM read; its tail tile DMA is ~30× that.
+So each tile's DMA can only hide behind the *previous tile's* tiny compute →
+DMA-bound from MiB 1.
+
+The **specific barrier** that prevents cross-layer overlap is the per-op WAR
+resync at `fattn.cu:1379`:
+
+```cpp
+CUDA_CHECK(cudaEventRecord(cs_sync, stream));       // record ALL prior compute
+CUDA_CHECK(cudaStreamWaitEvent(cs, cs_sync, 0));    // copy_stream waits on it
+```
+
+This orders `copy_stream` *after all prior compute-stream work at every op entry*
+(needed because the slot pool is re-handed per op → WAR hazard). It means the copy
+stream can **never run ahead of the current attention op** — so layer L+1's tail
+cannot begin loading during layer L's FFN. The recovery budget is therefore one
+op's compute (~0.1 ms → ~0.7 MiB), not the whole step's (~14 ms → ~90 MiB Gemma).
+
+## The fix: a persistent, step-spanning tail-prefetch ring
+
+Flatten the per-layer tile loop into **one tile stream across the whole step's
+attention layers**, kept `B` tiles ahead on a dedicated copy stream, hiding tail
+DMA behind the *sum* of all intervening compute (attention + FFN of the layers in
+between), not one op's.
+
+Components:
+
+1. **Shared device staging ring** (`B` = 2–4 buffers, each `tile_kv_full` × f16,
+   sized once). Replaces the per-op `ggml_cuda_pool_alloc` slot pool. VRAM cost =
+   `B × k_slot+v_slot` — a few hundred MiB, independent of context length.
+2. **Flat prefetch scheduler** over the ordered sequence of (layer, tile) tail
+   reads for the step. Issues H2D into ring slots on `copy_stream`, staying `B`
+   ahead. A ring slot is refilled only after its `compute_done[slot]` fires (WAR
+   discipline **replaces** the per-op `cs_sync` barrier — that's the edit that
+   unblocks cross-layer run-ahead).
+3. **Op consumes, does not copy.** Each layer's streaming FA waits on its tiles'
+   `copy_done`, reads the already-resident staging tile, records `compute_done`.
+   The window (resident) region stays a plain FA, unchanged.
+4. **Scheduling home.** The scheduler must span op boundaries, so it lives one
+   level up from the op — either (a) a context/graph-level KV-prefetch pass that
+   walks the step's attention nodes and their `k_tail/v_tail` sources, or (b) a
+   persistent per-context streamer object the ops register their tiles with. (a)
+   is cleaner for CUDA-graph capture; (b) is less invasive. Decide in S1.
+
+## Bounded win (be honest)
+
+Perfect overlap gives `decode_ms = max(step_compute, total_tail_DMA)` instead of
+their sum. Consequences:
+
+- **Below ~`step_compute × PCIe_BW`** (~90 MiB on Gemma-A4B, less on the 14B):
+  tail fully hidden → **near-1.0× (resident speed)**. This is the "free zone" the
+  op-level path fails to deliver (currently ~0 MiB).
+- **Above it:** PCIe-bound at `1000/(tail/BW)` regardless — physics. E.g. Gemma
+  @236 MiB → `max(14, 35) = 35 ms` ≈ 28 tps vs today's 19.5 (~1.4×).
+- **Does NOT rescue 1M full-attention** (14B tail = GBs, step_compute tiny → floor
+  stays low). Route (b) low-bit-resident remains the only long-ctx full-attn
+  ceiling-raiser (§5b-14B turbo-KLD, #581).
+
+So this widens the usable-overflow zone and roughly doubles the mid-tail regime;
+it does not defeat PCIe for extreme context. Worth it for the iSWA/short-overflow
+serving band; not a substitute for compression.
+
+## Companion lever: per-layer residency budget (stream fewer layers)
+
+Orthogonal to *how well* each tail is hidden is *how many* tails exist. Today the
+window/tail split is one global `window_cells` applied to every layer. But layers
+differ enormously in how far back they attend:
+
+- **Gemma (iSWA):** already per-layer — sliding-window layers carry NO tail, only
+  the ~5 global layers stream. This lever is fully banked; it's the 10× gap vs the
+  14B. Nothing to do.
+- **Qwen (full-attn):** HAL probe (needle ~1k in 12k niah) → only layers 0–4
+  (+~11, ~47) are truly local; ~13 layers attend the needle at ≥0.9, ~22 more at
+  0.45–0.9 → retrieval is **distributed across ~35/48 layers**. So the safely
+  droppable set is small.
+
+**Bounded win (measured):** window-only the provably-local layers → 48→~41
+streaming ≈ **1.17×**; pushing into the mid-band trades retrieval (StreamingLLM
+wall — the needle drops on the majority; see [[project_qwen_retrieval_distributed_hal]]).
+Hard ceiling if you could window all but the 13 strong retrievers ≈ 3.7×, but
+unreachable without breaking niah. Treat as a **~1.2–1.5× correctness-gated
+refinement that STACKS on the prefetch** (fewer streaming layers × each better
+hidden), NOT a Gemma-style restructure. Cross-layer cache SHARING stays dead
+(orthogonal caches, cos≈0.001) — this only skips a layer's OWN tail when that
+layer is local.
+
+Mechanism: replace the scalar `window_cells` with a per-layer `window_cells[il]`
+(a layer whose profile is local gets `window_cells[il]==kv_size` → no tail =
+resident-cheap). Drive `window_cells[il]` from an attention-locality profile
+(reuse the HAL probe offline, or a cheap online out-of-window-mass estimate at
+prefill). HARD GATE: per-layer niah must hold — never widen a layer's window past
+the point its retrieval mass survives.
+
+## Staged plan (de-risk correctness FIRST)
+
+- **S0 — per-layer residency budget — SHELVED 2026-07-01 (mechanism proven, not
+  shippable as scoped; user pivoted to S1/S2).** Implemented via env-gated
+  `OPENCOTI_KV_LOCAL_LAYERS` in `llama-kv-cache.cpp` (per-layer `window_cells`;
+  a local layer → `window_cells=0` resident sentinel, `layer_window` gates the
+  tail machinery). **Measured on 3090, 14B-1M @20k overflow, `--kv-residency-mode
+  window`:** the *throughput* lever is real and scales with resident count —
+  baseline (0 resident) 0.97 tps, local1 0.96, local5 1.05, **local10 1.24 tps
+  (1.28×)**, matching the 48/38-streaming ratio. **But correctness breaks from
+  N=1:** baseline needle=YES, but local1/5/10 all needle=NO. Root cause
+  (bug-1341): the READ dispatch is per-layer (llama-graph.cpp:3274,
+  `get_layer_tactic==POSITION_WINDOW`; `window_cells==0`→plain FA, correct) but
+  the KV **WRITE** path splits every layer window/tail on a context-uniform
+  assumption — a resident layer routes its whole write to a non-existent tail
+  (tail_c=0) → device KV stays zero → attends zeros → poisons the forward from
+  layer 0. A prior alloc assert was bug-1340 (fixed: `layer_window` gating).
+  **Why shelved:** (a) the fix is write-path surgery in llama-graph.cpp + rebuild
+  + re-gate, not the "single-file host-only cheap win" it was scoped as; (b) even
+  fixed, only the ~5–6 truly-local layers (0–4, needle_max<0.1 per the HAL
+  profile) are safe → realistic ceiling ~1.12×, still needing bs2 256k
+  validation. The step-prefetch below is the bigger, uniform, correctness-free
+  lever — do that first. Resurrect S0 only if a larger justifying win appears;
+  the exact edit recipe is in buglog bug-1340/1341.
+
+
+- **S1 — foundation (low-risk, no math change):** shared staging ring + flat
+  scheduler skeleton that still runs **1-deep** (B forced to current behaviour).
+  Prove logit-equiv byte-identical to today's op-level path. Establishes the new
+  ownership without changing timing.
+- **S2 — cross-layer run-ahead:** drop the per-op `cs_sync` barrier, replace with
+  ring WAR discipline; let the scheduler run `B` ahead across layers. Re-prove
+  logit-equiv (this is the risky reorder — the online-softmax combine must be
+  unaffected; ordering is data-independent so it *should* be identical, gate it).
+- **S3 — perf gate:** re-run the 3090 cliff curve (both models). PASS = the
+  free-zone extends to ~`step_compute×BW` and the mid-tail regime ≈ `max(compute,
+  DMA)`. Quantify vs the current hyperbola.
+- **S4 — tune B + tile size** for the staging-VRAM/overlap tradeoff; auto-size B
+  from `vram_target` headroom. bs2 96 GB validation.
+- **S5 — ship:** additive patch(es) + `opencoti-hook:` markers + UPSTREAM_SYNC +
+  this doc's results + .wolf + pgvector.
+
+## Constraints (standing)
+
+vendor-backup WHOLE tree before any vendor mutation; new kernel/reorder = full host
+`rm -rf o` + CUDA DSO rebuild + restamp BOTH DSO paths byte-identical + `nm -D`;
+correctness via logit-equiv / RULER-niah, NEVER greedy needle; additive soft-fork;
+CUDA-graph capturability preserved (no `cudaEventCreate` inside the op — reuse the
+pre-created event pool pattern from #315). Commit only when asked (dev).
+
+## Postscript (2026-07-05)
+
+S1 (staging ring) and S2 (barrier drop) landed but measured perf-inert on the
+3090 (bug-1838: the PCIe link is saturated; there is no copy-side slack to
+reclaim). The actual spill-decode ceiling was compute-side: the
+`streaming_lse_kernel` recompute on D≤256 decode tiles — fixed by bug-1843 /
+patch `0098-rolling-kv-lse-decode` (5.3× spill decode, see
+`rolling_kv.md`). S3 (this doc's perf gate) proceeds as #588 on the fixed
+binary.
+
+## #638 / bug-2148 — context-shift guard on a spilled window
+
+The spilled position window (GPU window `[0,wc)` ⊕ CPU tail `[wc,kv_size)`) is
+**incompatible with in-place KV re-roping** (server `--context-shift`, self-extend).
+The k_shift graph (`llm_graph_input_k_shift` / `build_rope_shift`) views `k_per_stream`
+over `get_size()` cells, but a windowed layer's `k_per_stream` holds only `window_cells`,
+so a shift over-reads it and asserts (`ggml.c:1840`); the host tail is never re-roped.
+`llama_kv_cache::get_can_shift()` now returns `false` whenever a spilled window is active
+(`window_cells > 0` && populated `k_cpu_per_stream`), so the server disables `ctx_shift` +
+cache-reuse at init and **bounds the request at `n_ctx`** instead of crashing. Fully-resident
+windows and non-window caches are byte-identical and still context-shift; iSWA/hybrid
+wrappers propagate the leaf guard. The no-degradation alternative — a host-side rope-by-delta
+pass over the CPU tail on every shift — is **deferred** (out of scope for the multi-session
+prefix-shared serving target). PolyKV shared-prefix (`seq_cp`/`seq_add`, does NOT set
+`is_fragmented`) composes with spill cleanly and needs no guard — validated in #638.
+Patch `0115-rolling-kv-shift-guard-bug2148`.

docs/features/rys_layer_duplication.md

@@ -0,0 +1,326 @@
+# RYS runtime layer duplication (weight-shared "Repeat-Your-Self")
+
+> Status: **DESIGN + PROTOTYPE** (2026-07-12). Parser/expander prototyped and
+> validated; edit surface mapped; MVP scoped. No source landed yet.
+>
+> Child of [MASTER_PLAN.md](../MASTER_PLAN.md). Additive soft-fork feature —
+> new load-time flag, per-arch surgical hooks in the graph loop + KV alloc.
+
+## 1. What this is
+
+David Noel Ng's **RYS** ("Repeat Your Self", [rys-ii][rys]) shows that
+re-running a contiguous block of *middle* transformer layers — **no weight
+changes, no training** — measurably improves a model (the layers duplicated are
+the ones operating in the format-agnostic "reasoning" band; encoding/decoding
+boundary layers must not be touched). For Qwen3.5-27B the Pareto-optimal blocks
+are tiny: `(33,34)` (+1 layer) already captures most of the EQ gain, up through
+`(26,34)` (+8) for the best absolute score.
+
+The naive way to ship a RYS variant is a **physically merged GGUF** (mergekit
+passthrough) — but that *doubles the duplicated layers' weight bytes* on disk
+and in VRAM. dnhkng is instead working with TurboDerp on **pointer-based**
+duplication in ExLlamaV3: the repeated layers **share the weight tensors** with
+their originals, so *"no additional VRAM is consumed for the parameters
+themselves — you only pay extra for the compute time and KV cache of the
+additional forward passes."*
+
+**This feature ports that pointer-shared idea to opencoti llamafile, driven
+purely by a load-time flag.** No new GGUF, no re-quant, quant-agnostic:
+
+```
+llamafile --server -m Qwen3.5-27B-*.gguf --repeat-layers 26,34   # RYS-XL, +8 layers
+llamafile --server -m Qwen3.5-27B-*.gguf --repeat-layers 33,34   # RYS-S,  +1 layer
+```
+
+Stock llama.cpp has **no** runtime layer-repeat; this is a genuine opencoti
+differentiator and fits our "additive, load-time" ethos.
+
+## 2. Mechanism: a layer execution plan (`eff → src`)
+
+llama.cpp's per-architecture graph builder runs a residual loop
+`for (il = 0; il < n_layer; ++il)` that (a) reads weights from
+`model.layers[il]` and (b) uses `il` to index the KV cache, RoPE, and per-layer
+hparams. We replace the identity iteration with a **plan**: a vector of *source*
+layer indices to execute in order.
+
+- **`layer_plan[eff] = src`** — for each *effective* position `eff`, which
+  source layer's weights to run. Default (no flag) = identity `[0,1,…,n_layer-1]`.
+- A RYS block `(i,j)` (half-open, matching the article) inserts source indices
+  `i…j-1` a second time **right after** their first pass.
+- **`n_layer_eff = layer_plan.size()`** — the *effective* layer count. This is
+  what the KV cache, RoPE loop, and residual loop use. The GGUF's `n_layer`
+  (the *source* weight count) is unchanged — weight loading is untouched.
+
+Forward loop, conceptually:
+
+```cpp
+for (int eff = 0; eff < n_layer_eff; ++eff) {
+    const int src = layer_plan[eff];
+    const auto & layer = model.layers[src];   // WEIGHTS: shared by pointer, zero extra VRAM
+    cur = build_norm(inpL, layer.attn_norm, …, eff);
+    cur = build_layer_attn(inp, cur, inp_pos, …, /*kv_il=*/eff);   // KV/cache index: eff (distinct slot)
+    …
+    inpL = cur;
+}
+```
+
+The duplicated pass reads the *same* weights but writes its *own* K/V at the
+same token positions — exactly the RYS semantics.
+
+### 2a. Prototype (validated)
+
+The plan expander is prototyped and unit-tested against every config in the
+article's Qwen3.5-27B Pareto table (reference:
+`scratchpad/rys_plan.py`; the C++ will mirror it):
+
+| spec       | blocks       | `n_eff` (from 64) | extra | article |
+|------------|--------------|-------------------|-------|---------|
+| `33,34`    | (33,34)      | 65                | +1    | +1 ✓   |
+| `31,34`    | (31,34)      | 67                | +3    | +3 ✓   |
+| `30,35`    | (30,35)      | 69                | +5    | +5 ✓   |
+| `26,34`    | (26,34)      | 72                | +8    | +8 ✓   |
+| `24,35`    | (24,35)      | 75                | +11   | +11 ✓  |
+| `29,34`    | (29,34)      | 69                | +5    | +5 ✓   |
+
+`(26,34)` expands to `…,24,25,[26,27,28,29,30,31,32,33],[26,27,28,29,30,31,32,33],34,35,…`
+— the seam is the duplicated `26..33`. Overlapping blocks are rejected in the
+MVP (disjoint-ascending only; the Pareto winners are all single blocks).
+
+## 3. Parameter grammar
+
+```
+--repeat-layers <spec>
+```
+- `<spec>` = one or more blocks separated by `;`.
+- Each block is `i,j` **or** `i-j` = the half-open range `[i,j)` of **source**
+  transformer-layer indices to duplicate in place. `i,j` matches the article's
+  `(i,j)` notation for copy-paste from the HF configs.
+- Examples: `33,34` · `26,34` · `43,45;28,34` (compose two disjoint blocks).
+- Validation (fail-fast at load): `0 ≤ i < j ≤ n_transformer_layers`; blocks
+  disjoint; the MTP/`nextn_predict_layers` head is **never** duplicable.
+
+Also plumbed through the opencoti TS adapter (`buildServerArgs`) as
+`repeatLayers?: string` so opencode configs can set it.
+
+## 4. Memory / cost model
+
+- **Params: zero extra VRAM** — duplicated layers point at the same
+  `model.layers[src]` tensors.
+- **KV cache: grows by the extra passes** — `n_layer_eff/n_layer`. E.g. `(26,34)`
+  on a 64-layer model = +8 → **+12.5%** KV. Modest; the article calls it out
+  explicitly. (This is why RYS-S `(33,34)`, +1.5% KV, is the low-overhead pick.)
+- **Compute: +one forward pass per duplicated layer** per token — same ratio as
+  KV. The win is quality-per-token, paid in tokens/s.
+
+## 5. What is provably unaffected
+
+- **Weight loading** — untouched; `n_layer` (source) drives `create_tensor`.
+- **RoPE / positions** — positions come from the batch, not the layer. A source
+  layer run at two `eff` positions sees the same token positions; only its KV
+  slot differs. No self-extend-style position hacking.
+- **Off (no flag)** — `layer_plan` = identity ⇒ byte-identical to stock. This is
+  the primary regression gate.
+
+## 6. Composition with opencoti per-layer features
+
+Everything opencoti indexes by layer must consume the **effective** index (or map
+`eff→src` for source-keyed attributes). Scoped incrementally:
+
+| feature | keyed by | MVP posture |
+|---|---|---|
+| plain f16 / scalar-quant KV | `eff` slot | **MVP** — works directly |
+| RoPE / positions | batch pos | unaffected |
+| iSWA (Gemma sliding/global per layer) | **src** (`is_swa(il)`) | Gemma phase — map `eff→src` |
+| DCA (all-KV) | per-layer | later — gate off under `--repeat-layers` first, then make plan-aware |
+| rolling-KV (`window_cells[il]`) | `eff` budget | later — size by `n_layer_eff` |
+| PolyKV / sparse-attn | per-layer | later |
+| MTP / NextN head | excluded | head never duplicated; spec path unchanged |
+
+**Survey nuance (important):** DCA, rolling-KV (`window_cells[il]`), PolyKV, and
+sparse-attn are all keyed through `map_layer_ids` / per-**physical**-layer
+structs — so they **auto-inherit eff-correctness** the moment the `:927` alloc
+loop (§8b#4) is eff-length. The residual risk is *not* the accessors; it's the
+**sizing loops that still say `hparams.n_layer`** and would under-allocate for
+eff slots: `n_layer_kv()`, `opencoti_compute_resident_window_cells`
+(`kv-cache:249/:772`), and the sparse block-sel reserve `8*n_layer_kv`
+(`kv-cache:879`). Plus DCA's `is_swa(il)` in `gemma4:217` needs `src`.
+
+**MVP rule:** ship `--repeat-layers` composing only with plain/quant KV on the
+dense arch first; **hard-error at load** if combined with DCA / rolling-KV /
+sparse / MTP until each sizing loop is made eff-aware (explicit follow-ups). This
+prevents silent under-allocation, which would corrupt rather than error.
+
+## 7. Per-architecture scope
+
+### Phase 1 — Qwen3.5 / Qwen3.6 **dense** (MVP target)
+Uniform layers (no iSWA, uniform head counts/rope) ⇒ `eff→src` is trivial for
+everything except KV-slot allocation. `nextn_predict_layers` (MTP head) excluded
+from the plan. Target models: Qwen3.5-27B dense, Qwen3.6 dense variants.
+Forward loop: `models/qwen35.cpp` (also covers the dense path), `models/qwen3.cpp`.
+
+### Phase 2 — Gemma-4 **dense** 12B and 31B
+Adds the iSWA wrinkle: alternating sliding-window / global layers, per-layer
+query scaling / attn logit-softcap. A duplicated layer must inherit its **source**
+layer's SWA type and scaling ⇒ per-layer hparam lookups map `eff→src`. The KV
+cache's SWA/global split (`kv_local_layer`, the iSWA cache) must size by
+`n_layer_eff` with the src-derived SWA pattern. Forward loop: `models/gemma4.cpp`.
+Targets: `gemma-4-12B-it` dense, `gemma-4-31B-it` dense.
+
+(Gemma-4 A4B is MoE — out of scope for the dense MVP; RYS on MoE is a separate
+question about whether per-expert routing survives block re-traversal.)
+
+## 8. Edit surface
+
+> Line-anchored from the source survey (2026-07-12). All paths under
+> `vendors/sources/llamafile/llama.cpp/`.
+
+**Governing finding:** in this fork the `il` handed to `build_attn(...)` is used
+for BOTH weight lookup (`model.layers[il]` in the model `.cpp`) AND KV-slot
+lookup (`map_layer_ids[il]`). RYS **decouples** them: **weights + architecture
+props (`is_swa`, rope-freq, `n_head`, `n_rot`) index by `src`; the KV-slot index
+handed to `build_attn` is `eff`.** For Qwen **dense** this is a no-op distinction
+outside KV (uniform layers, `is_swa` always false), so `build_attn(eff)` is
+clean. For Gemma the iSWA sub-cache choice must be `src`-driven (see §8b).
+
+### 8a. C-ABI plumbing caveat (load-bearing)
+`struct llama_context_params` (`include/llama.h:336`) crosses the **C ABI** — it
+cannot carry a `std::vector`. So:
+- `include/llama.h:336` — add a **compact int spec** (`uint32_t repeat_start,
+  repeat_end, repeat_count`, or a small CSV string).
+- `src/llama-cparams.h:9` — add `std::vector<int> layer_plan;` to `llama_cparams`
+  (plain C++ struct, vector OK). Insert near the opencoti extension fields (69-80).
+- `src/llama-context.cpp:~133` — **expand** the compact spec into
+  `cparams.layer_plan` here (where `hparams.n_layer` is known); default unset =
+  identity `0..n_layer-1`; set `n_layer_eff = layer_plan.size()`.
+
+### 8b. Minimal MVP edit set — Qwen dense (plain KV)
+| # | site | change | invasiveness |
+|---|------|--------|--------------|
+| 1 | `llama-cparams.h:9` | add `layer_plan` vector | trivial |
+| 2 | `include/llama.h:336` | compact int spec (C-ABI) | trivial |
+| 3 | `llama-context.cpp:~133` | expand spec → `layer_plan`; `n_layer_eff` | trivial |
+| 4 | `llama-kv-cache.cpp:927` (+ sizing `:165/:278/:451/:478`) | iterate `n_layer_eff`; size each eff slot from `src=plan[eff]` (`n_embd_k_gqa(src)`, `has_kv(src)`); `map_layer_ids[eff]=layers.size()` | moderate |
+| 5 | `models/qwen3.cpp:76` (·qwen2.cpp:76·qwen3moe.cpp:86) | loop over `eff`; `src=plan[eff]`; `model.layers[il]→[src]`; `build_attn(…,eff)`; out-ids guard `eff==n_layer_eff-1` | moderate |
+| 6 | `common/arg.cpp:~2780` | `add_opt("--repeat-layers")` (mirror `--override-kv` CSV) | trivial |
+| 7 | `common/common.cpp:~1660` | map `common_params` range → `llama_context_params` spec | trivial |
+
+`qwen35.cpp:169` — the plan is built over `[0, n_transformer_layers)` where
+`n_transformer_layers = n_layer - nextn_predict_layers` (`:167`), so the **MTP
+head is already excluded** by construction; never duplicate a `nextn` block.
+
+**KV precedent:** `llama-kv-cache.cpp` already tolerates a non-identity
+`map_layer_ids` — the `il_share` path (`:946-963`, bug-858 dual-ctx MTP) and
+`il_reuse` (`:1265-1283`, SWA reuse). Both *share* physical slots; RYS is the
+inverse (allocate **more, distinct** slots), so no new indirection type is
+needed — just an **eff-length domain** on the `:927` alloc loop. The 30+ read
+accessors (`get_k`/`cpy_k` via `mli_at_checked(map_layer_ids, il, …)`) are
+unchanged as long as the graph passes `eff` and every `eff` has a map entry.
+
+### 8c. Additional edits — Gemma-4 dense (iSWA), the invasive part
+| site | change |
+|------|--------|
+| `models/gemma4.cpp:194` | loop rewrite splitting `src`/`eff`: `is_swa(src)`, `get_rope_freq_base/scale(cparams,src)` (`:201-202`), `n_rot(src)`, `layers[src].rope_freqs`, `f_attention_scale` by `src`; feed `eff` only to `build_attn` (`:269-283`) + KV routing |
+| `gemma4.cpp:164` iSWA dual-cache | `build_attn_inp_kv_iswa()` + kv-cache filter/reuse must assign each **eff** slot to base-vs-swa sub-cache by `is_swa(plan[eff])` — pass a `plan`-aware filter into the iSWA cache ctor (the invasive step: it currently reads `swa_layers[il]`) |
+| `llama-model.cpp:1929/1933/1937` | `get_rope_freq_base/scale/factors` only ever called with `src` (they branch on `is_swa`) |
+
+### 8d. hparams helpers (index by `src`)
+`llama-hparams.cpp`: `is_swa(il)` `:208`, `n_head(il)` `:30`, `n_head_kv(il)`
+`:38`, `n_rot(il)` `:65`, `n_embd_k_gqa/v_gqa(il)` `:103/:109`, `has_kv(il)`
+`:231`, `n_layer_kv()` `:250`, `swa_layers[]` (`hparams.h:141`). No signature
+changes — the rule is "call these with `src` in the loop." RoPE positions come
+from the batch (`llama-graph.cpp:161`), not the layer, so a `src` layer run at
+two `eff` positions is correct with no position hacking.
+
+## 9. Validation plan
+
+Correctness is *mechanism* correctness (the plan re-traverses exactly the
+intended layers) — RYS quality is the user's block choice, not ours to prove.
+
+1. **Off = identity** — `--repeat-layers` absent ⇒ logits byte-identical to
+   stock (teacher-forced, real_frac=0). Primary regression gate.
+2. **Plan structural** — boot log dumps `n_layer_eff` + the `eff→src` plan;
+   assert it matches the spec (e.g. `(26,34)` → 72 layers, seam at 34/42).
+3. **Deterministic divergence** — `(33,34)` vs base produces a *deterministic,
+   non-degenerate* logit change (greedy, same GGUF) — proves the duplicated pass
+   actually runs and feeds forward, not a no-op.
+4. **KV integrity** — niah retrieval stays 100 at a short ctx with a duplicated
+   block (the extra KV slots are wired correctly; no cross-slot aliasing).
+5. **Known-win spot check** — reproduce the article's *direction*: a small EQ/math
+   probe should move in the reported direction for `(33,34)` / `(26,34)` on
+   Qwen3.5-27B (sanity that we duplicated the reasoning band, not garbage).
+6. **Gemma iSWA** — the src-derived SWA pattern is honored (duplicated global
+   layer stays global); niah 100 + coherent decode.
+
+Correctness always via logit-equiv / niah — never greedy-needle-as-proof.
+
+## 10. Risks / caveats
+
+- **KV growth is real** (§4) — surface it in the boot log; the TS adapter should
+  account for it in VRAM budgeting.
+- **iSWA mapping** — feared to be the main Gemma risk, but **verified correct**
+  (see §10a): a middle SWA layer duplicates coherently, so the dual-cache eff→src
+  wiring is sound. What actually breaks is *boundary* layers, independent of SWA.
+- **MoE out of scope** for dense MVP — but the iSWA path was validated on **both**
+  dense (31B, 60 layers) and MoE (A4B, 26B-A4B-128e) with identical behaviour.
+- **Upstream syncability** — new flag + per-arch loop hooks tagged
+  `opencoti-hook: rys-layer-dup` and registered in `UPSTREAM_SYNC.md`; the
+  `layer_plan` indirection is small and localized to keep future bumps cheap.
+
+## 10a. Boundary-layer fragility (bug-2164) — a MODEL property, not an engine bug
+
+Duplicating the **first or last** transformer layers reliably produces incoherent
+output. This is the well-documented franken-merge / passthrough self-merge
+fragility (mergekit `passthrough`, SOLAR depth-upscaling all DROP the first/last
+*m* layers), NOT an engine defect. The engine duplicates *any* layer faithfully.
+
+**Decisive 2×2** (2026-07-13, native `google_gemma-4-26B-A4B-it-Q4_K_M`, 3090,
+CHAT endpoint — the raw `/completion` greedy path is INVALID on this thinking
+model: its answer lands in `reasoning_content`, so even RYS-off looks like garbage
+there). Garbage is determined **entirely by boundary-ness, independent of
+SWA/global**:
+
+| | SWA layer | GLOBAL layer |
+|---|---|---|
+| **middle** (L4, L10 / L5, L17) | coherent ✓ | coherent ✓ |
+| **boundary** (L0, L28 / L29) | garbage ✗ | garbage ✗ |
+
+- Gemma-4 A4B coherent band ≈ **L4..L24**; L0–L3 (early) and L25–L29 (late) break.
+- Qwen3-8B has a wider tolerant band — only its **last** layer (L35) breaks.
+- Every attention output + final logit stays **finite** (no NaN) even under the
+  garbage config → finite-but-wrong = model-level, not a numeric/cache corruption.
+
+**Guidance: duplicate MIDDLE layers.** The engine emits a boot-time **advisory
+WARNING** (`llama-context.cpp`, RYS engage block) whenever a plan duplicates a
+layer within the first/last `max(3, n_transformer/8)`; it warns but does not block
+(the exact usable band is model-dependent). Correctness gate = coherence smoke via
+the CHAT endpoint (RYS intentionally changes the model, so logit-equivalence vs
+baseline is the wrong bar). Historical note: an earlier pass mis-labelled this an
+"SWA sub-cache engine bug" by testing only L0 (which is *both* SWA *and* first) on
+the noisy raw-greedy path — see [[project_rys_swa_dup_garbage]].
+
+## 10b. RYS × MTP self-spec — the draft context must NOT inherit the plan (bug-2171, 0124)
+
+RYS is a **target-only** capability. With RYS active *and* self-speculation on
+(`--spec-type draft-mtp`, Qwen NextN or the gemma4 assistant), the MTP draft
+`llama_context` is built from `common_context_params_to_llama(params_base)`, so it
+inherited the target's `params.repeat_layers` — and its `cparams.layer_plan` got
+RYS-duplicated too. But the NextN head lives at the canonical `il = n_transformer`
+(`n_layer − nextn_predict_layers`), which the RYS eff→src map never contains, so
+draft-context init aborts with `cpy_k: map_layer_ids MISS il=n_transformer` →
+`failed to create MTP context`. Fix (`0124`, `src/llama-context.cpp`): the RYS
+plan-expansion block gates the spec on context type —
+`rys_spec = (cparams.ctx_type == LLAMA_CONTEXT_TYPE_MTP) ? nullptr : params.repeat_layers`
+(mirrored in `sched_reserve()`), so the **draft runs the plain base stack** while the
+**target keeps RYS**. This is **lossless**: greedy self-spec verifies every drafted
+token against the target, so the drafter's stack depth only moves acceptance/tps.
+In self-spec the NextN head consumes the target's *already-RYS* hidden state via
+shared memory — running the drafter plain is the intended composition, not a
+limitation.
+
+## 11. References
+
+- [dnhkng, *LLM Neuroanatomy II*][rys] — the RYS-II study, Pareto table, pointer-based note ("The Models").
+- [dnhkng/RYS on GitHub](https://github.com/dnhkng/RYS) — scanner, probes, model-builder config grammar.
+- mergekit `passthrough` — the physical-merge convention `(i,j)` mirrors.
+
+[rys]: https://dnhkng.github.io/posts/rys-ii/

docs/features/rys_probe.md

@@ -0,0 +1,204 @@
+# RYS template probe (`--rys-probe`) — design + implementation
+
+Status: **IMPLEMENTED (v1.1 = ΔPPL + graded task-probe → 2 templates + external
+eval hook), host-only, validated on Qwen3-4B** (2026-07-13, #669/#670). Not yet
+captured as a patch / committed — ships on request. Companion to the shipped RYS
+execution engine,
+[rys_layer_duplication.md](rys_layer_duplication.md) (`--repeat-layers`, patch 0122).
+
+## 0. What shipped (v1.1)
+
+A new `PROBE` program mode in `llamafile/rys_probe.cpp` (dispatched from
+`llamafile/main.cpp`; flag detected in `llamafile/args.cpp`). It:
+
+1. Parses args as the `PERPLEXITY` example, then **disables `-fit`** (the probe
+   sizes its own small `n_ctx`; the fit path builds a trial context that we don't
+   want mutating our cparams — it aborted the flash-attn print otherwise).
+2. Loads the model **once** (`common_init_from_params(params, /*model_only*/true)`).
+3. Enumerates contiguous mid-stack blocks `[i,j)` in the band (default
+   `[max(3,n/8) .. n-max(3,n/8))`) × `--rys-probe-widths`. Default widths =
+   **`auto`** = every width `1 .. (band)` (an explicit `W1,W2,…` list overrides).
+4. Scores **base first** (empty plan = reference), then each candidate by
+   rebuilding **only the context** (`llama_init_from_model` with
+   `cp.repeat_layers = "i,j"`; the model/weights are plan-independent), computing
+   a compact self-contained **ΔPPL** over a small corpus (`-f`/`-p`, else a
+   built-in slice) **+ a graded built-in task-probe** (24 arithmetic / word-problem
+   / factual / sequence items, greedy decode with substring-match early-stop; the
+   fraction correct is the `task` score, used both as a rank signal and a guardrail
+   — a candidate that drops below base `task` is discarded).
+5. Prints a ranked table of the surviving wins, then **exactly two ready-to-paste
+   templates**:
+   - **MOST EFFICIENT** — best PPL-drop *per added layer* (largest `ΔPPL/(#dup layers)`).
+   - **MAX GAIN** — largest absolute PPL-drop / lowest PPL.
+   When no wider block beats the efficient one, the two coincide and the probe
+   says so honestly ("the same block is both").
+
+**Flags:** `--rys-probe` (enter probe mode) · `--rys-probe-widths auto|W1,W2,…`
+(default `auto`) · `--rys-probe-band auto|i:j` · `--rys-probe-topk N` (default 10,
+rows in the table). Corpus comes from the standard `-f`/`-p`; `-ngl`, `-c`, `-fa`
+flow through.
+
+**The last mile — external eval `(c)`.** ΔPPL + the built-in task-probe are cheap
+*proxies*, not a verdict. The probe's two templates feed
+[`perf/llamafile/rys-probe-eval.sh`](../../perf/llamafile/rys-probe-eval.sh),
+which boots `llama-server` for BASE + each template (only `--repeat-layers`
+changes between arms) and runs a user-supplied `EVAL_CMD` (e.g. a GSM8K / MMLU
+slice via the omk harness) against each, printing a real task-score comparison
+with deltas-vs-base. That is the analogue of dnhkng's graded benchmark, minus the
+days-long search — the final pick is a downstream score, not a proxy.
+
+**Validation (Qwen3-4B-Q4_K_M, CPU):**
+- band [8,16), widths {2,4}, 12 candidates: base ppl 30.15; wins `9,13`
+  (ppl→24.95, Δ−5.20, 11.1% over) and `10,12` (ppl→27.38, Δ−2.76, 5.6% over) →
+  MAX GAIN = `9,13`, MOST EFFICIENT = `10,12` (−1.38/layer), two distinct frontier
+  points.
+- band [10,16), widths `auto` (all 1..6), 14 candidates: base ppl 30.15 / task
+  0.917; winner `11,13` (ppl→28.03, Δ−2.12, 5.56% over, task 0.958) — sole survivor
+  wide enough, so MOST EFFICIENT == MAX GAIN (honest "same block is both").
+
+Contiguous mid-stack winners, no boundary layers, consistent with §1's structural
+prior. Build gotcha: a `common_params` struct **field addition** needs a **clean
+host rebuild** (`rm -rf o` + make) — incremental `build:llamafile:make` left
+ABI-skewed `common_params` consumers (bug-2160/2166 class); see BUILD_CYCLE.md. A
+later default-**value** change to an existing field (e.g. widths default
+`1,2,4`→`auto`) is ABI-safe and rebuilds incrementally.
+
+Question (user, 2026-07-13): can opencoti-llamafile grow a flag that *probes* a
+GGUF for good RYS layer-duplication templates, without reproducing dnhkng's
+multi-**day** search?
+
+Short answer: **yes, cheaply** — because we can replace the two expensive halves
+of the reference method (a huge candidate space + a downstream-benchmark score)
+with (a) a **pruned** candidate space we already know the shape of, and (b) a
+**perplexity scan** the binary can already run. The result is a *shortlist
+generator* (minutes–~1 h), not a full Pareto search — the user still validates
+the top-K on the real retrieval/benchmark harness.
+
+## 1. What the reference method (dnhkng RYS-II) actually does, and why it's slow
+
+From <https://dnhkng.github.io/posts/rys-ii/> (64-layer stack):
+
+| Stage | Mechanism | Cost |
+|---|---|---|
+| Full scan | every contiguous `(i,j)` block, duplicate `[i,j)`, measure Δ | many single-block evals |
+| Beam search | width 24, depth 3–6, greedily stack blocks | **3,024 candidates, ~14 h** |
+| Surrogate | XGBoost trained on **4,411 measured** configs ranks **~2,000,000** candidates | top **100** fully benchmarked |
+| Validate | Pareto front (better *and* smaller) → 4 optimal configs | large val sets |
+
+Two things make it days-long:
+
+1. **The score is a downstream benchmark** — `Math120` (16–120 math Qs) + `EQ140`
+   (EQ-Bench scenarios). Every candidate needs generation + grading. Small
+   "probe" sets are used for the search, larger sets for finalists, but it is
+   still an accuracy eval per candidate.
+2. **The space is ~2 M** contiguous+stacked blocks, so a surrogate (XGBoost on
+   4.4 k measured points) is needed just to *rank* it.
+
+**Key structural finding we can exploit:** every Pareto-optimal template was a
+**single contiguous mid-stack block**. The cheapest winner was `(33,34)`
+(+1.56% params); the widest useful one `(26,34)` (+12.5%). Nothing near the
+first/last layers ever won — which is exactly our own boundary-layer result
+([rys_layer_duplication.md §10a](rys_layer_duplication.md), bug-2164: duplicating
+early/late layers yields incoherent output).
+
+## 2. What opencoti already has
+
+- The **execution engine**: `--repeat-layers "i,j"` sets `cparams.layer_plan` and
+  re-runs `[i,j)` weight-shared, load-time, all archs (patch 0122). An
+  in-process probe just calls this in a loop with different plans.
+- A **perplexity** path in the vendored `llama.cpp` (used throughout the TCQ/KV
+  campaigns) — a forward pass over a fixed corpus returning mean NLL.
+- **KLD-vs-reference** tooling (logit-equiv harness).
+- The **boundary-layer advisory** logic that already computes the safe mid-stack
+  band `[max(3,n/8) .. n-max(3,n/8))`.
+
+So the only missing piece is an *outer loop + a scorer + a ranked report*.
+
+## 3. Proposed design — a PPL-scan shortlist, not a Pareto search
+
+### 3a. Score: perplexity on a small fixed corpus (NOT KLD-to-base)
+
+- **Use ΔPPL vs the base model on a small held-out corpus.** RYS improves the
+  model by *changing* it, so a candidate that lowers PPL below base on a
+  quality-correlated corpus is the cheap analogue of "Math/EQ went up". One
+  forward pass over a few-k-token corpus per candidate — seconds to a couple of
+  minutes on a served GPU, no generation/grading.
+- **KLD-to-base is the wrong optimization target here** and we must not use it as
+  the score: RYS *intentionally* diverges from base, so minimizing KLD rewards
+  doing nothing (this is the same trap as "logit-equiv vs base" for RYS —
+  [rys_layer_duplication.md](rys_layer_duplication.md)). KLD is still useful as a
+  **guardrail**: a candidate whose KLD explodes (or whose PPL blows up) is the
+  boundary-fragility / broken-merge regime — flag and drop it, don't rank it.
+- Caveat, stated honestly: **PPL is a weak proxy for reasoning/EQ gains.** dnhkng
+  used task accuracy precisely because self-merge gains are small and PPL-noisy.
+  So the probe's job is to *shortlist*, and the last mile stays a real eval.
+
+### 3b. Space: prune to contiguous mid-stack blocks (no surrogate needed)
+
+Because winners are always contiguous mid-stack, enumerate only:
+
+- contiguous blocks `[i,j)` fully inside the safe band `[max(3,n/8) .. n-max(3,n/8))`,
+- with width `1 .. w_max` (default `w_max ≈ ceil(n/8)` — the +12.5% ceiling),
+- optionally strided starts to cap the count.
+
+For a 64-layer model that is a few hundred candidates, not 2 M — so a **direct
+scan needs no XGBoost surrogate at all.** (An optional depth-2 greedy stack, à la
+their beam search, can be a follow-up once single-block ranking is trusted.)
+
+### 3c. Surface: `--rys-probe`
+
+```
+llamafile --rys-probe \
+  -m model.gguf \
+  [--rys-probe-corpus calib.txt]   # default: bundled ~4k-token slice
+  [--rys-probe-widths 1,2,4,8]     # block widths to try
+  [--rys-probe-band auto|i:j]      # candidate band (default = advisory band)
+  [--rys-probe-topk 10]            # how many to print / emit
+```
+
+Load the model once; for each candidate plan set `layer_plan`, run PPL over the
+corpus (base = empty plan, run first as the reference), record
+`(block, overhead_pct, ppl, dppl_vs_base, kld_guardrail)`. Emit a table sorted by
+ΔPPL and a **Pareto view** (ΔPPL vs overhead%) — the same "better *and* smaller"
+lens dnhkng used, so the user picks a point on the frontier. Output the winner as
+a ready-to-paste `--repeat-layers` spec.
+
+Reuses the 0122 engine + the existing perplexity path; **host-only, no CUDA, no
+new kernels.**
+
+### 3d. Cost
+
+`N_candidates × one small-corpus PPL forward`. A few hundred candidates × a
+few-k-token PPL ≈ **minutes to ~1 h on one GPU** for a mid-size model — vs days.
+The saving is entirely from (1) pruning 2 M → hundreds (structural prior) and
+(2) PPL instead of graded generation.
+
+## 4. Honest limitations
+
+- **Shortlist, not verdict.** PPL can rank a candidate above one that would win
+  on reasoning/EQ. Ship it as "top-K to validate", and keep the real
+  RULER/benchmark run as the decider — exactly how dnhkng re-measured finalists.
+- **PPL validity on franken-merges is model-dependent.** For some Gemma-4
+  omnimerge GGUFs raw PPL is meaningless
+  ([project note: PPL/KLD invalid on Gemma merges]); the probe must sanity-check
+  the base PPL is finite/reasonable and warn (or fall back to a tiny task-probe)
+  when it isn't.
+- **Single-block first.** Stacked (beam) templates are a phase-2 extension; the
+  single-contiguous-block scan already covers every dnhkng Pareto winner.
+
+## 5. Recommendation
+
+Feasible and cheap. Build it as an **opt-in `--rys-probe` subcommand** that
+(1) enumerates contiguous mid-stack blocks, (2) scores each by ΔPPL on a small
+fixed corpus with a KLD/PPL-blowup guardrail, (3) prints a ΔPPL-vs-overhead
+Pareto shortlist and the best `--repeat-layers` spec. It is a thin wrapper over
+the already-shipped 0122 engine + the existing perplexity path — no kernels, no
+CUDA. It does **not** reproduce dnhkng's rigor (no 2 M-candidate surrogate, no
+graded Math/EQ); it deliberately trades that for a minutes-scale shortlist the
+user finishes with one real eval pass.
+
+Decision (user, 2026-07-13, #669/#670): shipped **(b) ΔPPL + a graded
+arithmetic/short-reasoning task-probe**, output narrowed to **two templates**
+(MOST EFFICIENT + MAX GAIN), plus a **(c) external-eval hook**
+(`perf/llamafile/rys-probe-eval.sh`) that runs a real downstream eval per template
+so the final pick is a task score, not a proxy. See §0.

docs/features/sparse_attn.md

@@ -0,0 +1,453 @@
+# Sparse Attention — Quest-style block-selector for long-context decode (#551)
+
+> Status: BUILT + CHARACTERIZED on the 3090 (2026-06-27); 1M validation on bs2
+> pending. The real 1M **decode** lever — companion to the DCA + quant-KV
+> **prefill** work (#444/#445/#554). Default-OFF, byte-identical when off. Develops
+> on the 3090 (A4B / Qwen3-4B / 27B-Q4 @ ≤256k); 1M validation on bs2.
+>
+> **Headline result (§S4 results):** the sparse/dense **decode** ratio is
+> *context-driven, not batch-driven*. Single-stream (B1), Qwen3-4B, f16 KV, 25%
+> block coverage on the 3090, the ratio climbs monotonically and the gap to dense
+> closes 9.2% → 0.8% from 16K to the model's 40K ceiling (0.908 → 0.992). Crossover
+> (>1.0, a real decode win) extrapolates just past 40K — the 256k/1M regime on bs2.
+> Batch is *not* our lever (unlike TheTom — see §S4 results for why).
+
+## Why this, and why now
+
+The 1M serving path has two halves. **Prefill** is handled by DCA's analytical
+band + quantized KV (q5_0/q4_0 — f16 KV at 1M is 206 GB, impossible), and #554
+removed the whole-cache-lift prefill cliff. **Decode is still O(n_kv) per token**:
+at 1M every generated token attends to ~1M KV entries, memory-bound and slow. The
+only thing that breaks that is *sparse attention* — attend to a small, query-chosen
+subset of KV blocks instead of all of them.
+
+## Method: Quest-style min/max block bounds (decided 2026-06-25)
+
+Reference scouted: **TheTom/turboquant_plus** `docs/papers/block-selector-sparse-attention.md`.
+Decisive finding for us: that work is **Apple Silicon / MLX**, where TheTom found
+custom sparse kernels with *real* K-skipping LOSE (14× at B=8) to the tuned
+`sdpa_vector_2pass`, so he uses a float mask that still streams the full K/V (saves
+compute, **not bandwidth**). He explicitly notes the literature wins (Quest, NSA,
+SeerAttention, DuoAttention) "all target **CUDA, where FlashAttention's sparse kernel
+is mature**." **We are CUDA.** Our FA-VEC already has the `KV_max`/`KV_min` tile-trim
+(`fattn-vec.cuh:363`) that lets us skip **whole KV blocks before the QK dot** — real
+K-read + QK + V savings — which TheTom's Metal stack could not. So we build the thing
+he couldn't, using the cleaner CUDA ancestor:
+
+**Quest** (Tang 2024). Parameter-free, exact upper-bound:
+- **Per-(KV-head, block) min/max key bounds.** Block size `B_SEL` (start 64; tune).
+  Store `kmin[head_dim]`, `kmax[head_dim]` per block — a tiny side-cache
+  (`n_blocks × head_dim × 2 × f16`). For quant-KV, compute min/max from the f16
+  values **before** quantizing (composes with q5_0/q4_0).
+- **Per-query upper bound.** For query `q`, the max possible `q·k` over a block is,
+  channel-wise, `Σ_d max(q_d·kmin_d, q_d·kmax_d)` (use kmax where q_d>0, kmin where
+  q_d<0). O(n_blocks·head_dim) — negligible vs O(n_kv·head_dim) dense attention.
+- **Top-K block selection.** Keep the top-`K_SEL` blocks by upper bound **PLUS** the
+  recent window + attention-sink block (streaming-LLM safety so recency/retrieval is
+  never silently dropped). `K_SEL` and window are knobs; tune on RULER-niah.
+- **Kernel skip.** Extend the FA-VEC tile loop to `continue` past unselected blocks —
+  the real K+QK+V skip.
+
+Budget policy: **fixed top-K + recent/sink** (decided). TheTom found adaptive top-K
+"never beat union top-K" and converged loosely at long ctx; fixed-K is predictable in
+VRAM/compute and easy to gate.
+
+## Compose (three sparsity levers, three pipeline stages)
+
+1. **Block-selector (this work)** — skip K blocks *before* QK (saves K-read + QK + V).
+2. **DCA analytical band** — the selector narrows *within* each INTRA/SUCC/INTER band.
+3. **sparse-V (#546)** — within selected blocks, skip V dequant for negligible-weight
+   positions (the residual, post-softmax).
+Plus **quant-KV** (min/max from dequant K at write). The three are independent gates at
+the block / band / position granularities.
+
+## Where it lives (insertion points, from the Explore map)
+
+- **Write path (S1):** per-block min/max side-cache, filled as K is written
+  (KV-cache write in `llama-graph.cpp` / the cpy path). Additive tensor; off ⇒ not allocated.
+- **Selector (S2):** host/graph pre-pass computing the per-query block mask from the
+  bounds; carried to the kernel by **extending the existing `KV_max`/`KV_min`
+  mechanism** (`fattn-common.cuh:27-28`, `fattn-vec.cuh:352-353`) with a block-select
+  bitmask/priority.
+- **Kernel (S3):** block-skip `continue` in the FA-VEC tile loop at `fattn-vec.cuh:363`
+  (the long-ctx **decode** hot path is FA-VEC, not MMA).
+
+## S1 integration anchors (mapped 2026-06-25, read-only)
+
+All file:line in `vendors/sources/llamafile/llama.cpp/`:
+- **K write (scatter):** `src/llama-graph.cpp:2771` — `mctx_cur->cpy_k(ctx0, k_cur, k_idxs, il)` (impl in `llama-kv-cache.cpp`). The per-block min/max must be derived from `k_cur` here (pre-quant) or right after, per write.
+- **KV tensor alloc (add the side-tensor here):** `src/llama-kv-cache.cpp:637` (GPU `ggml_new_tensor_3d(ctx_s, type_k, n_embd_k_gpu, wc, 1)`) + `:686` (host); stored in the `layers` vector (`llama_kv_cache_layer`, pushed `:738`). Add a parallel per-(KV-head, block) `kmin/kmax` tensor alongside `k_l`.
+- **Host-fill model:** `set_input_dca` `src/llama-kv-cache.cpp:3899-4040` (dispatch `:5250`) — the idiom for filling a per-forward input tensor (direct `tensor->data` writes / `ggml_backend_tensor_set`); mirror for the selector's per-block mask in S2.
+- **Flag plumbing (mirror `--dca` for `--sparse-attn*`):** `common/arg.cpp:1516,1528` (add_opt) · `src/llama-cparams.h:56-57` (cparams fields) · `common/common.cpp:1649-1650` (params→cparams) · `tools/server/server-context.cpp:1102` (server gate).
+- **Sizing accessors:** `llama-hparams.h:268` `n_head_kv(il)` · `:283` `n_embd_head_k(il)` · `cparams.n_ctx` (graph ctor `llama-graph.cpp:1009`).
+- **S1 sequence:** vendor-backup WHOLE tree FIRST → add `--sparse-attn{,-topk,-block-size}` flags (default off) → alloc the kmin/kmax side-tensor (only when enabled) → fill it at/after the `cpy_k` write → build host + verify default-off path is byte-identical (no new tensor allocated, no graph change). Selector + kernel skip are S2/S3.
+
+## Correctness gate (NEVER greedy needle — bug-263/270/354)
+
+- **RULER-niah @256k** on the 3090 (A4B + Qwen3-4B): the needle's block MUST survive
+  selection — niah recall is the primary signal that block-selection didn't drop the
+  answer.
+- **Logit-equivalence vs dense** (`real_frac`, top-k agreement) as the tolerance check.
+- Perf: decode + prefill tps vs dense, sparsity-ratio sweep, at 32k/256k (3090) then
+  512k/1M (bs2).
+
+## Knobs (default OFF, byte-identical)
+
+- `--sparse-attn {off,on}` (or auto by ctx threshold)
+- `--sparse-attn-block-size` (B_SEL, default 64)
+- `--sparse-attn-topk` (K_SEL blocks) and `--sparse-attn-recent` (always-keep window)
+- env mirrors for dev (e.g. `OPENCOTI_SPARSE_ATTN_*`).
+
+## Staging
+
+- **S1** — per-block min/max side-cache + write-path plumbing (host). Off ⇒ byte-identical.
+- **S2** — Quest selector (query·bounds → top-K + recent/sink mask) + KV_max/KV_min carry.
+- **S3** — FA-VEC kernel block-skip in the tile loop.
+- **S4** — gate: RULER-niah @256k + logit-equiv vs dense + decode/prefill tps (3090).
+- **S5** — compose with sparse-V #546 and DCA bands; re-gate.
+- **S6** — capture patch (vendor-backup + snapshot-diff, byte-identical re-apply) +
+  this doc's measured table + UPSTREAM_SYNC markers + STATE_SUMMARY + .wolf.
+
+## S2/S3 engineering design (mapped + decided 2026-06-25)
+
+Integration reuses the proven `KV_max`/`KV_min` shape (pool-alloc + pre-pass fill +
+kernel param), but with one decisive difference: the per-step fill **writes** kmin/kmax
+and the selector **reads** them *in the same graph*, so ordering is enforced with real
+ggml **src edges** (two new graph ops) — **not** the DCA thread-local side-channel
+(`opencoti_fattn_dca_pos_q`, `ggml-cuda/fattn.cu:27-29`), which is safe only for
+analytical inputs with no shared-buffer write-then-read.
+
+**Three pieces:**
+
+1. **Fill op** `sparse_attn_fill` (new GGML custom op; pattern = the `ggml_streaming_*`
+   / `ggml_turbo_wht` ops #345/#391).
+   - Inserted in `build_attn` right after each `cpy_k` (`llama-graph.cpp:2771`; iSWA
+     `:2985`, `:3067`), gated on `cparams.sparse_attn_enabled`.
+   - srcs: `k_cur` (new tokens' K, **pre-quant f16**) + `k_idxs` (scatter positions).
+     dst-update: `kmin_l`/`kmax_l` cache tensors (`build_forward_expand`, like `cpy_k`;
+     the write into the persistent cache tensor + the selector's read edge needs the
+     `ggml_view_1d` dep-tie idiom — bug-225 precedent — so the scheduler sees fill→select).
+   - CUDA kernel: per (KV-head, channel), reduce min/max over each affected block's
+     positions; **min/max-merge with the existing** kmin/kmax (blocks fill incrementally
+     across decode). Affected blocks = `[n_past/B, (n_past+n_new)/B]`.
+   - Off ⇒ op not added ⇒ byte-identical.
+
+2. **Selector op** `sparse_attn_select` (new GGML custom op; pre-pass shape mirrors
+   `flash_attn_dca_to_KV_max`, `fattn-common.cuh:891`).
+   - Inserted after `get_k` (`:2813`) before the FA op.
+   - srcs: `Q` (rotated FA query) + `kmin_l` + `kmax_l` (the read edge orders it after the
+     fill). dst: `block_sel` bitmask (`n_blocks` bits per `(seq, KV-head)`, I32-packed).
+   - CUDA kernel: per block b, UB = `Σ_d max(q_d·kmin_b[d], q_d·kmax_b[d])`; **GQA** → max
+     UB over the query group per KV-head (one block set per KV-head, NSA co-load); top-`K_SEL`
+     by UB (smem partial-sort / threshold) **OR'd with** the recent-window + sink blocks;
+     write bits.
+
+3. **FA-VEC block-skip** (S3).
+   - `block_sel` rides as an **extra src on the FA op** — set `result->src[5] = block_sel`
+     after `ggml_flash_attn_ext` (mirror of `sinks`=src[4]); the scheduler builds the
+     selector→FA edge from the src slot regardless of the core op. `launch_fattn` passes
+     `block_sel->data` as a new kernel param `const int * __restrict__ block_sel` alongside
+     `KV_max`/`KV_min` (`fattn-vec.cuh:83-84`, `fattn-common.cuh:27-28`).
+   - Tile loop (`fattn-vec.cuh:363`, immediately after the existing `k_VKQ_min` skip):
+     `if (block_sel && !(block_sel[base + ((k_VKQ_0/B)>>5)] & (1u<<((k_VKQ_0/B)&31)))) continue;`.
+     B (= `B_SEL`) aligned to the FA tile stride (`nthreads`) and the quant blck (32) ⇒
+     `B_SEL` a multiple of both for clean skipping.
+   - Threaded through every FA-VEC instance signature (exactly as `KV_max` was). The MMA
+     prefill path is a **separate later stage** (S1–S4 target FA-VEC decode).
+
+**Gate ladder (NEVER greedy needle — bug-263/270/354):**
+- **Plumbing anchor:** `K_SEL = n_blocks` (select-all) ⇒ sparse must equal dense ⇒
+  logit-equiv `real_frac=0`. Proves the carry + skip are exact before any real sparsity.
+- Then reduce `K_SEL`: RULER-niah@256k (needle's block must survive) + logit-equiv
+  tolerance + decode/prefill tps vs dense, sparsity-ratio sweep (3090 @ 32k/256k → bs2 1M).
+
+**Build reality:** fill+selector+skip ⇒ CUDA DSO rebuild (~25 min/iter, restamp BOTH DSO
+paths). Min 2 builds (B1: fill — verify kmin/kmax vs a CPU min/max reference via a debug
+dump; B2: selector+skip — select-all anchor then sparsity sweep). Plan 2–4 iterations.
+
+## Open questions (resolve during impl)
+
+- Block size vs the FA-VEC 128-thread tile stride and the quant block size (q5_0/q4_0
+  blck=32) — align B_SEL to a multiple of both for clean skipping.
+- GQA: bounds are per-KV-head; the selector reduces the GQA query group to a per-KV-head
+  representative (max over the group's per-block upper bounds) so all grouped Q-heads
+  share one block set (NSA-style co-load) — avoids per-Q-head mask divergence.
+- Interaction with the DCA fused MMA path (prefill) — S1–S4 target FA-VEC decode; the
+  MMA/prefill block-selector (TheTom's 2.9×@128K prefill analogue) is a later stage.
+- Where to compute the selector: a small dedicated CUDA op vs folding into the FA launch.
+
+## Comparison vs TheTom (S4 — user-required, "test the same as theTom to compare")
+
+Reproduce TheTom's exact block-selector benchmark so our numbers sit directly beside
+theirs. Their config (from `turboquant_plus/docs/papers/block-selector-sparse-attention.md`,
+fetched 2026-06-25):
+
+- **Model:** `Qwen2.5-14B-Instruct-1M-4bit` — **we have this exact model on bs2** (the #550
+  DCA vehicle; native-1M, no-YaRN — see `project_qwen25_1m_native_no_yarn.md`). Same model ⇒
+  direct apples-to-apples.
+- **Grid:** batch `B ∈ {1,2,4,8}` × ctx `{16K … 131072}` (their 19-cell grid). B>1 uses our
+  PolyKV `--parallel`. Metric = **dense-vs-sparse decode + prefill tps ratio**.
+- **Block size:** 64 — matches our locked `B_SEL=64`.
+- **Their headline (M5 Max / MLX):** decode **1.19× (B4/16K) → 1.73× (B8/28K) → 1.48× (B4/48K)**;
+  prefill **2.49× (64K) → 2.90× (128K)**. These are the numbers to match/beat.
+- **Their quality validation is INCOMPLETE** — *no* clean end-to-end retrieval benchmark;
+  only cosine-vs-dense (0.999 prefill / 0.99995 long-ctx adaptive-topK). **We add a real
+  RULER-niah@{256K,1M} retrieval gate they lack** — a strict improvement on their eval.
+
+**Why ours should win the decode column:** TheTom's selector is a **JL-projection
+float-additive mask** (`contentDim=32`, `-∞` for non-selected) — *compute-only masking, no
+bandwidth save* — because MLX/Metal couldn't do a real K-skip (cerebrum 2026-06-21). Ours is
+**Quest min/max block-skip on CUDA FA-VEC** that `continue`s past unselected blocks BEFORE
+the K-read + QK + V — a real bandwidth save — so our decode ratio should *exceed* their
+1.19–1.73× at the same sparsity, not just match it. Prefill (MMA path) is a later stage
+(their 2.49–2.90× is the prefill-selector analogue, S5+).
+
+> **CORRECTION (2026-06-27), see §S4 results:** the "should *exceed* their 1.19–1.73×"
+> claim was over-stated *for the batched column*. TheTom's wins are at **batch 4–8**;
+> ours are at **context length** — different mechanisms (his fixed mask-cost is
+> amortized across streams; our real K-skip is a *fixed per-stream* cost that batching
+> can't amortize). So at his batch points we sit near break-even, not above. Where the
+> prediction holds is the axis he does *not* measure — single-stream long-context — and
+> the 3090 ladder (0.908→0.992, climbing) is the first hard evidence for it.
+
+**S4 deliverable:** a side-by-side table — same model, same B×ctx grid, dense-vs-sparse
+decode/prefill ratio (ours, on the 3090 ≤128K then bs2 to 1M) next to TheTom's MLX column,
+PLUS our RULER-niah column. Run on bs2 (the 14B-1M lives there) to mirror their long-ctx grid.
+
+## S4 results — 3090 single-stream ctx-ladder + batch grid (2026-06-27)
+
+The TheTom comparison ran in two parts on the 3090 (bs2/14B-1M busy; the same-model
+1M grid is still pending). Model: **Qwen3-4B-Q4_K_M**, f16 KV, 25% block coverage,
+block size 64, Quest selector (`--sparse-attn on`). Metric = dense-vs-sparse **decode**
+throughput ratio. Decode tps read from the server log's authoritative
+`eval time … / 128 tokens … tokens per second` line — *not* the HTTP JSON, which is
+unreliable under the bug-743 parse-race (see Gotchas).
+
+### Single-stream ctx-ladder (B1) — the decisive axis
+
+| ctx (B1) | dense tps | sparse tps | ratio | gap to dense |
+|---------:|----------:|-----------:|------:|-------------:|
+| 16384 | 74.50 | 67.65 | 0.908 | −9.2% |
+| 24576 | 61.54 | 59.37 | 0.965 | −3.5% |
+| 32768 | 53.41 | 52.06 | 0.975 | −2.5% |
+| 40000 | 47.46 | 47.06 | **0.992** | −0.8% |
+
+Monotonic climb; the gap halves roughly every ~8–16K. At Qwen3-4B's native 40960
+ceiling we are within measurement noise of dense, and the trend extrapolates to
+crossover (>1.0) just beyond — i.e. the 256k/1M serving regime. This is the expected
+signature of a *real bandwidth-saving K-skip*: saved K-reads grow with cache size
+while the selector's fill + QK-estimate + `block_sel`-gather cost is ~fixed, so the
+ratio rises with context. (Script: `.opencoti/sparse-ctx-ladder.sh`, gitignored.)
+
+### Batch grid (B1/B2/B4) — confirms batch is NOT our lever
+
+| ctx/seq | B | dense_agg | sparse_agg | ratio | note |
+|--------:|--:|----------:|-----------:|------:|------|
+| 16384 | 1 | 74.5 | 67.3 | 0.903 | clean |
+| 16384 | 2 | 62.6 | 12.0 | 0.192 | bug-743 parse-race (one req died, n<B) |
+| 16384 | 4 | 42.4 | 11.3 | 0.267 | bug-743 parse-race (n=3) |
+| 24576 | 1 | 60.9 | 58.1 | 0.954 | clean |
+| 24576 | 2 | 44.9 | 43.2 | 0.962 | clean — sparse ~3% slower on **both** slots |
+| 24576 | 4 | 27.7 | 27.7 | 1.000 | clean — uniform ~3% penalty washed out by scheduler asymmetry |
+
+Per-slot eval times on the clean 24K cells show sparse ~3% slower per stream (e.g.
+B2 dense 39.6/5.3 tps vs sparse 38.0/5.2). The 24K "1.000" is *not* a batched win — it
+is the uniform per-stream penalty hidden by one-fast-slot/three-starved scheduler
+asymmetry. So batching gives our selector **no** multiplier. (Script:
+`.opencoti/sparse-batch-grid.sh`.)
+
+### Verdict — we win a different game than TheTom
+
+| | TheTom (turboquant_plus) | ours (#551 Quest) |
+|---|---|---|
+| Selector | JL-projection float-additive mask (compute-only, no bandwidth save) | Quest min/max block-skip on CUDA FA-VEC (real K/V-read skip) |
+| Winning axis | **batch** (1.19× B4/16K → 1.73× B8/28K → 1.48× B4/48K) | **context** (0.908@16K → 0.992@40K, B1) |
+| B1 behaviour | structurally ≤1.0 (mask adds work, saves nothing at B1) | climbs to ~1.0 and past it with ctx |
+| Quality eval | cosine-vs-dense only (0.999) | RULER-niah retrieval gate (a real gate he lacks) |
+| Hardware/model | M5 Max / MLX / 14B-1M-4bit | 3090 / CUDA / Qwen3-4B f16 |
+
+His mask saves no bandwidth, so his win **needs** batch to amortize a fixed selector
+cost across streams; remove the batch and his advantage disappears. Ours saves real
+bandwidth per stream, so it scales with **context** and batching can't amortize a
+per-stream cost. Consequences:
+
+- **At B1 (single-stream serving) we should beat him** — we climb to ~1.0 with ctx via
+  real K-skip; his compute-only mask is structurally ≤1.0 at B1 (he doesn't report B1).
+- **At B8/28K (his sweet spot) he wins on his hardware and we get no batch multiplier** —
+  we sit near break-even there.
+- **On validation we are strictly ahead** — RULER-niah vs cosine-0.999.
+
+A true number-against-his-column comparison is still **not** apples-to-apples (different
+HW/model/quant). The only direct side-by-side is the still-pending S4 deliverable: the
+same 14B-1M-4bit, the same B×ctx grid, on bs2 to 1M, **plus** our RULER-niah column.
+
+### Gotchas surfaced
+
+- **bug-743 (parse-race):** under load the server can feed a request's *generated*
+  filler continuation back into the JSON parser → `500 "Failed to parse input at pos 0"`,
+  which `curl` receives as the response (0 tps). It fires intermittently even at
+  `--parallel 1`. **Always read decode tps from the server log, not the HTTP JSON.** A
+  reliability bug to fix before batched sparse can be trusted in production.
+
+## DeepSeek Sparse Attention (DSA) — investigated, NOT portable (2026-06-27)
+
+Investigated per user request (Reddit thread on a llama.cpp DSA effort). DSA
+(arXiv:2512.02556) is **not transferable** to our Qwen/Gemma targets:
+
+- It needs a **trained "lightning indexer" head** (FP8, ReLU-gated, KL-distilled from
+  dense attention) plus ~**946B tokens** of base-weight sparsity adaptation, and is
+  **MLA/MQA-coupled** — our models are GQA with no indexer weights.
+- The llama.cpp implementation (fairydreaming/sszymczyk, fork branch `deepseek-dsa`,
+  merged upstream as **PR #23346**, 2026-05-29) is **correctness-only** — the author's
+  own note says it "doesn't improve long-context performance yet" (oversized compute
+  buffers).
+- The one transferable idea — a small trained indexer head — is a **training project**,
+  not a port. Out of scope for the vendored-fork decode lever.
+
+## All-KV-types composition (bug-747, 2026-06-27)
+
+Sparse attention originally composed **only with f16 KV**: the B1 fill op
+(`GGML_OP_SPARSE_ATTN_FILL`) read the *quantized cache* K view, and its CUDA
+`supports_op` + launcher required `src[1]` to be `GGML_TYPE_F16`. With any quantized KV
+cache (`q8_0`/`q4_0`/`q6_0`/turbo*/tcq*) the cache K is non-f16 → `supports_op=0` on
+CUDA0 → the scheduler could not place the op on the buffer holding the pre-allocated
+`kbounds` side-cache → **abort at graph-reserve** (`ggml-backend.cpp:919`,
+`cannot run the operation`). (Sibling of bug-737, which was the I64 `k_idxs` variant of
+the same abort.)
+
+**Fix — k_cur incremental-merge fill.** The fill now reads **`k_cur`** (the *new* token K
+projection, always f16/f32 *before* cache quantization) instead of the quantized cache
+view. The kernel is templated on K type (`half`/`float` via `sparse_ld<>`), derives
+`n_new = k_cur->ne[2]` and the column stride `nb1 = k_cur->nb[2]`, maps cache positions
+via `p0 = k_idxs[0]`, and for each block either **MERGEs** (min/max) into the prior
+`kbounds` (the straddling last block at decode / unaligned prefill chunk) or
+**OVERWRITEs** (a fully-new block). Because min/max is exact, this reproduces the old
+dense recompute **byte-identically on f16**, and because the cache is quantized *downstream*
+(in `cpy_k`), the Quest bound is the exact pre-quant envelope — so sparse now composes
+with **every** cache KV type for free. `supports_op` + launcher relaxed to `f16||f32`;
+the graph passes `k_fill = ggml_is_contiguous(k_cur) ? k_cur : ggml_cont(k_cur)`.
+
+### Gate (Qwen2.5-14B-Instruct-1M-Q6_K, RTX 3090, RULER niah_single_1, 12 samples)
+
+| ctx | KV | mode | coverage | niah | notes |
+|----:|----|------|---------:|-----:|-------|
+| 24k | f16 | dense  | 100% | 100.0 | baseline |
+| 24k | f16 | sparse |  25% | 91.67 | lossy regime (below f16 lossless threshold) |
+| 24k | f16 | sparse |  65% | **100.0** | f16 lossless reference |
+| 24k | q8_0 | dense | 100% | 100.0 | q8 alone is clean |
+| 24k | q8_0 | sparse | 25% | 75.0  | lossy regime |
+| 24k | q8_0 | sparse | 65% | **100.0** | **composition lossless** |
+| 24k | q8_0 | sparse | 80% | 100.0 | composition lossless |
+| 48k | q8_0 | dense | 100% | 100.0 | |
+| 48k | q8_0 | sparse | 25% | 83.33 | lossy regime (improves vs 24k: more absolute blocks) |
+
+**Composition is lossless.** q8_0⊕sparse holds niah=100 at the same coverage (≥65%) where
+f16⊕sparse is lossless. The 25%-coverage gap (q8 75 vs f16 91.67) is *not* a composition
+defect — the block *selection* is identical (both fill `kbounds` from the same f16 `k_cur`),
+so the gap is q8 quantization noise on the attended needle block plus 12-sample variance in
+the sub-threshold lossy regime. At shippable coverage the penalty vanishes.
+
+### The decode win lives on quantized KV
+
+Steady-state decode (max over niah gen samples), niah=100 configs:
+
+| KV | dense | sparse@65% | ratio |
+|----|------:|-----------:|------:|
+| f16 | 27.4 tps | 20.2 tps | 0.74× (loss) |
+| q8_0 | 15.2 tps | ~17.4 tps | ~1.15× (win) — cross-gate hint |
+
+**Same-boot A/B confirmation (q8_0, Qwen2.5-14B-1M-Q6_K, 24k, `sparse-q8-win-confirm.sh`).**
+To remove cross-run variance the dense baseline and every sparse coverage were run
+back-to-back in one harness boot. The win is real and grows monotonically as coverage drops,
+while niah stays 100 all the way down to 50% coverage:
+
+| coverage | niah | steady tps | vs dense |
+|---------:|-----:|-----------:|---------:|
+| 100% (dense) | 100 | 13.71 | 1.00× |
+| 80% | 100 | 15.15 | 1.10× |
+| 65% | 100 | 17.63 | 1.29× |
+| 50% | 100 | 18.33 | **1.34×** |
+
+Best lossless operating point in this sweep: **50% coverage → 1.34× decode at full niah.**
+(25% coverage is lossy — q8 75 — so the lossless band bottoms out between 25% and 50%.)
+The same-boot dense baseline (13.71) ran ~10% below the cross-gate extraction (15.2),
+which is exactly the run-to-run variance the same-boot A/B exists to eliminate — the
+sparse/dense *ratio* is the trustworthy number, not the absolute tps.
+
+On **f16**, sparse@lossless coverage is *slower* than dense: a 14B model at 24k is not
+KV-bandwidth-bound enough for a 35%-block skip to overcome the per-stream selector cost.
+On **q8_0** — the actual long-context serving config — sparse is a **lossless decode win**,
+because the dense path must **dequantize every block** (`q8→f16` lift) before FA, and the
+block-skip skips that dequant work too. That extra saved compute is what tips q8
+positive where f16 stays negative. So bug-747 didn't just make q8⊕sparse *work* — quantized
+KV is *where the win is*, even on a model that loses on f16.
+
+### Stacking with sparse-V (#546) — orthogonal on speed, tau must be retuned per config (2026-06-27)
+
+sparse-attn (#551, block-skip) and sparse-V (#546, attention-gated V-skip, `TURBO_SPARSE_V_TAU`)
+both live in `fattn-vec.cuh`: the block-selector chooses which blocks to visit; the V-skip drops
+negligible-weight positions *inside* visited blocks. Same-boot 4-cell A/B, q8_0 KV,
+Qwen2.5-14B-1M-Q6_K, 24k, block coverage 50%, sparse-V tau 0.1 (`sparse-stack-ab.sh`):
+
+| cell | niah | tps | vs dense |
+|------|-----:|----:|---------:|
+| dense | 100 | 17.91 | 1.00× |
+| block-skip @50% | **100** | 19.87 | 1.11× (lossless) |
+| sparse-V τ=0.1 | **0** | 20.11 | 1.12× (lossy) |
+| both | **0** | 23.49 | **1.31×** |
+
+Two findings: (1) the levers **compound on throughput** — `both` (1.31×) is faster than either alone,
+confirming they cut on orthogonal axes (which blocks vs which positions-within-block). (2) sparse-V's
+**τ=0.1 does not transfer**: it was tuned lossless on turbo/turbo_tcq KV on 27B and Gemma-A4B (#546,
+niah 100), but on q8_0 KV / Qwen2.5-14B-1M it destroys the needle (niah 0) — τ=0.1 is far too aggressive
+for this model's q8 attention distribution. The lossless compound win is recoverable via a per-config
+τ-sweep (0.1 → 0.01 → 0.001) to find where V-skip rejoins niah=100; block-skip alone remains the
+proven lossless lever here (1.11× @ 50%).
+
+## Sparse-V auto-policy (#565, patch 0088) — distinct from the block-selector above
+
+NB: this is the **sparse-V** decode lever (#546, quantized-V attention-gated skip in the FA-VEC
+kernel), NOT the Quest/vslash block-selector that is the subject of the rest of this doc. They are
+independent. Sparse-V is shipped + validated; this section documents only its **auto-policy**.
+
+The win is **arch-dependent**: lossless on interleaved-SWA (Gemma-4 — small concentrated global
+V-cache), no-win on full-attention (Qwen — diffuse weights near the mean). Rather than make every
+standalone `llamafile --server` user discover `TURBO_SPARSE_V_TAU`, the **binary self-configures**:
+
+- **Where:** `src/llama-context.cpp`, `llama_context` ctor (`// opencoti-hook: sparse-v auto-policy (#565)`).
+- **Rule:** if `getenv("TURBO_SPARSE_V_TAU")==NULL` AND `hparams.swa_type != LLAMA_SWA_TYPE_NONE`
+  (iSWA) AND `params.type_v` is quantized (not f16/bf16/f32) → `setenv("TURBO_SPARSE_V_TAU","0.05",0)`.
+- **Why host-side env (not a DSO setter):** ggml-cuda is a runtime-loaded DSO; the kernel already
+  reads the threshold from `TURBO_SPARSE_V_TAU`. Pre-populating that env is host-only (no DSO ABI
+  change, DSO byte-identical), and is the only channel that also reaches a standalone CLI run.
+- **Override:** `overwrite=0` → an explicit `TURBO_SPARSE_V_TAU` (or `_EPS`/`_SINK`/`_RECENT`) wins.
+- **Safety:** full-attn models leave the env unset → kernel τ=0 → dense, byte-identical. f16/bf16 V is
+  inert via the kernel's own `type_V != F16/BF16` guard even if the env were set.
+- **Visibility:** announced once at `LLAMA_LOG_WARN` (llamafile suppresses the model-load INFO block;
+  a silent INFO-only auto-change is invisible — bug-2090).
+- **Sampling:** non-gating. Sparse-V is a KV-bandwidth decode lever; correctness is proven at greedy
+  (temp 0, the strictest case), so the policy is sampling-agnostic.
+
+Verified (3090): decision-table 4/4 + A/B @ 24k lossless (niah 100==100) and non-slower; the
+measurable decode win lands at 64k+ (production 50-sample: A4B/31B niah 100, +2.7/3.8% @ 64k →
++1.3/3.1% @ 256k). See docs/evaluations/context.md (2026-06-29).
+
+### Adaptive τ (the `eps` mode) — built and characterized, NOT the shipped default (#565 verdict)
+
+#565's literal goal was a *runtime weight-distribution-driven* threshold (not a fixed peak floor). It
+**exists in the binary** as the `eps` mode: `fattn-vec.cuh:515/566` computes the per-query threshold
+`thr = eps · d_inv_nproc · KQ_sum[jj]` — recomputed from the running softmax denominator so it adapts
+to each token's actual attention mass (skip iff the dropped *normalized mass* < `eps`). Selectable via
+`TURBO_SPARSE_V_EPS`; `eps=0` ⇒ falls back to the static `d_sparse_v_tau`.
+
+**It is NOT what the auto-policy regulates** — the policy above only sets the *static* `TURBO_SPARSE_V_TAU`;
+it never sets `TURBO_SPARSE_V_EPS`. That is deliberate, not a gap: the adaptive `eps` mode was measured
+**uniformly lossless but ~zero speedup** (Qwen-14B-1M full-attn: `eps` 0.005–0.10 all niah=100, tps flat
+7.8–7.9 vs dense 7.83). The reason is structural — a mass-relative threshold can only harvest the
+*sub-mean* tail, and on diffuse full-attention weights cluster near the mean, so almost nothing is
+skippable regardless of `eps`; on iSWA the static τ=0.05 already captures the concentrated-global-cache
+win more aggressively. So adaptive τ ships as a **safe opt-in knob** (never slower/lossy, `eps=0`
+byte-identical) while the static τ via auto-policy is the production win. **Verdict:** the question "does a
+runtime-adaptive threshold beat static τ on the architectures we have?" is answered — *no*. Reopen only for
+a fundamentally different full-attention adaptive scheme (which is the content-selection problem #551, itself
+a closed dead-end). #565 closed as **built + characterized**, 2026-06-30.

docs/features/tiered_inference.md

@@ -0,0 +1,805 @@
+# F1 — Tiered Inference Engine
+
+> Parent: [../MASTER_PLAN.md](../MASTER_PLAN.md)
+> Status: **planning**
+> Owner: TBD
+
+## Problem
+
+A single provider is never the right answer for a real coding agent:
+
+- A small local model is cheap and private but stalls on hard tasks.
+- A subscription provider (Ollama Cloud Gemma 4 31B, Claude Max,
+  ChatGPT Plus, etc.) has caps; you don't want every token to count
+  against them.
+- A per-token provider (Anthropic API, OpenAI API, etc.) works but
+  charges for everything it sees.
+
+We want opencoti to start work on the cheapest provider that can
+plausibly handle it, and **escalate only when needed**.
+
+## Tier 0 vision — primary local execution, two paths
+
+Tier 0 is the role-label for "the cheap-and-fast tier we start every
+turn on." It is *not* hard-wired to a specific runtime. opencoti
+supports two first-class Tier-0 realizations and the user picks one
+explicitly at first-run setup (no silent auto-fallback):
+
+- **GPU path — the advanced one.**
+  Patched llamafile + `ManniX-ITA/gemma-4-A4B-98e-v5-coder-it`
+  (Gemma 4 26B-A4B base, 98-expert prune of the original 128, ~4 B
+  active per token, 256 K context max, IQ3_M ≈ 9.82 GB, runs end-to-end
+  on a 24 GB GPU with full offload). The patched llamafile is what
+  opens up the advanced features below.
+- **No-GPU path — the simple one.**
+  Ollama Cloud as an OpenAI-compatible Tier-0 provider with the
+  Gemma 4 31B dense model (free BF16 on Ollama Cloud at time of
+  writing). Plain HTTP request flow, no local compute, no advanced
+  features. Equally first-class for users without a GPU.
+
+Anything else the OpenAI-compatible provider interface accepts works
+too (a remote vLLM, a different Ollama endpoint, etc.). Tier 0 is the
+abstraction; the realizations are config-driven.
+
+Smaller Gemma 4 variants (E2B, E4B) and the stock Gemma 3 lineage are
+explicitly *not* primary targets — evaluated much later if at all.
+
+### Advanced features tied to the GPU path
+
+The patched-llamafile path unlocks features the no-GPU path cannot
+offer. opencoti **does not** expose this distinction in user-facing
+configuration: features that need the patched llamafile are present
+automatically when the GPU path is selected, and silently absent on
+the no-GPU path. From the user's perspective Tier 0 is Tier 0;
+internally the engine queries provider capabilities and degrades
+gracefully.
+
+Concretely (see [local_llamafile.md](local_llamafile.md) for detail):
+
+- **KV-cache reuse across agentic turns** — turn N+1 reuses turn N's
+  prefix, dramatic latency drop on long contexts.
+- **Big-context throughput patches** — keep tokens/sec usable at
+  64K+ contexts.
+- **Single-download Ollama-blob reuse** — if Ollama is installed
+  locally, the patched llamafile can mount the existing
+  `~/.ollama/models/blobs/sha256-…` GGUF directly. No duplicate
+  download.
+
+## Goals
+
+- **G1.** Configure an ordered set of tiers (Tier 0, Tier 1, ...),
+  each with one or more providers.
+- **G2.** Run a request on the lowest-numbered tier first; escalate
+  on failure-to-progress, hard error, or explicit escalation signal.
+- **G3.** Support two tier topologies:
+  - **Linear** — try a single provider in this tier, fall through on
+    failure.
+  - **Fan-out** — try multiple providers in this tier in parallel
+    (or as a quorum), pick the best response.
+- **G4.** Within a tier, prefer subscription providers; fall back to
+  metered providers when subs are exhausted or rate-limited.
+- **G5.** Make the escalation policy a **user-visible config**, not
+  a hidden heuristic.
+
+## Non-goals (for now)
+
+- Reordering tiers automatically based on observed quality.
+- Cross-session learning of provider quality.
+- Coordinating tiers across multiple concurrent sessions.
+
+## Design sketch
+
+### Where it lives
+
+- `packages/opencoti-tiers/` — new package. Holds the tier schema,
+  the router, the escalation policy, the provider-pool primitives.
+- **Surgical hook** in opencode's model-call site: when an opencoti
+  tier config is present and enabled, the call is delegated to the
+  router from `opencoti-tiers`. Otherwise opencode behaves as
+  upstream.
+
+### Concepts
+
+- **Tier** = an ordered slot in the escalation chain. Tier 0 is the
+  cheapest, tried first.
+- **Provider Pool** inside a tier = one of:
+  - `linear: [Provider, Provider, ...]` — tried in order
+  - `fanout: { mode: "first-good" | "best-of-n" | "quorum", members: [...] }`
+- **Billing class** per provider:
+  - `subscription` — has a cap; depleted state is observable.
+  - `metered` — pay per token, no cap.
+- **Escalation triggers** (configurable, ORed):
+  - hard-fail (provider error, timeout, refused)
+  - stuck-progress (no tool calls / no diff after N turns)
+  - explicit (user keypress, or model emits an "escalate" tool call)
+  - structured-quality (output fails a structured validator — e.g.,
+    code did not parse, tests still failing after N attempts)
+
+### Config shape (illustrative, not final)
+
+GPU-path example:
+
+```yaml
+opencoti:
+  tiers:
+    enabled: true
+    effort: balanced     # low | balanced | high | max
+    models:
+      # Small local registry — names → entries. Same idea as
+      # claude-hooks' provider registry. Entries are matched by name
+      # from the `chain` below.
+      gemma4-a4b-coder:
+        kind: llamafile
+        gguf: ~/.opencoti/models/gemma-4-A4B-98e-v5-coder-it-IQ3_M.gguf
+        contextLength: 131072    # capped; model supports up to 256K
+      gemma4-31b-cloud:
+        kind: openai-compat
+        baseURL: https://ollama.com/api
+        apiKeyEnv: OLLAMA_CLOUD_API_KEY
+        modelId: gemma4:31b
+        billing: subscription
+    chain:
+      - id: tier0
+        topology: linear
+        providers:
+          - gemma4-a4b-coder       # patched llamafile, advanced features
+      - id: tier1
+        topology: linear
+        providers:
+          - gemma4-31b-cloud       # subscription (Ollama Cloud)
+          - groq/llama-3.3-70b     # metered, fallback
+      - id: tier2
+        topology: fanout
+        mode: first-good
+        providers:
+          - anthropic/claude-sonnet-4-6      # subscription via Max
+          - anthropic-api/claude-sonnet-4-6  # metered, fallback
+    escalation:
+      on: [hard-fail, stuck-progress, structured-quality]
+      stuck-progress:
+        no-progress-turns: 3
+      structured-quality:
+        validators: [code-parses, tests-pass-or-improve]
+```
+
+No-GPU example (Tier 0 is the cloud provider; no llamafile present):
+
+```yaml
+opencoti:
+  tiers:
+    enabled: true
+    effort: balanced
+    models:
+      gemma4-31b-cloud:
+        kind: openai-compat
+        baseURL: https://ollama.com/api
+        apiKeyEnv: OLLAMA_CLOUD_API_KEY
+        modelId: gemma4:31b
+        billing: subscription
+    chain:
+      - id: tier0
+        topology: linear
+        providers:
+          - gemma4-31b-cloud
+      - id: tier1
+        # ... user's preferred escalation provider ...
+```
+
+### Tool surface
+
+opencode already injects a strong default tool set into every
+`streamText` call (15 built-in: `shell`, `read`, `write`, `edit`,
+`glob`, `grep`, `task`, `fetch`, `search`, `todo`, `skill`, `patch`,
+`question`, `lsp`, `invalid`), plus full MCP server integration, plus
+plugin / `.opencode/tool/*` user-defined tools, **all gated by the
+same permission flow**. That surface already meets-or-exceeds what
+claude-hooks' caliber-grounding-proxy + consultants skill expose
+(`survey_project`, `list_files`, `read_file`, `glob`, `grep`,
+`recall_memory`, and the sandboxed `write_file` for the coder role).
+
+What opencode does NOT have today, and that opencoti will port from
+claude-hooks in a later phase (target: F4 or F5 — out of scope for
+F1):
+
+- **`recall_memory` as a project-injected tool.** Opencode requires
+  the user to wire pgvector/sqlite_vec via MCP. claude-hooks injects
+  recall_memory directly into the tool list so the model can query
+  memory without an MCP roundtrip. The opencoti tiers package already
+  injects synthetic `__tier_*` tools (M5); the same mechanism extends
+  to `__opencoti_recall_memory` cleanly.
+- **`survey_project` — hierarchical project map.** Composable from
+  `glob` + `read`, but having one shot of "tell me the project shape"
+  saves the model several turns.
+- **Grounding system-prompt prepend.** A short paragraph that frames
+  the model's task in terms of "you're working in this project at
+  cwd …". The M5 prompt-augmentation primitive (`augmentSystem`) is
+  the right seam.
+- **Multi-role consultant orchestration.** opencode has `task`
+  (launch one subagent), claude-hooks consultants has a 6-role
+  LangGraph council (planner / researcher / critic / synthesizer /
+  tool_executor / coder). The structured orchestration is the part
+  worth porting; opencode's `task` is the underlying primitive it
+  would build on. Likely a `opencoti-consultants` package consuming
+  opencode's existing subagent + permission infrastructure.
+
+These ports are documented here only as the **direction** for later
+phases. F1 M5.7 (below) covers the configuration plumbing; F4/F5
+plans, when written, will own the actual ports.
+
+### "Effort" presets
+
+The `effort` knob is a one-line way for the user to pick how willing
+opencoti is to spend tokens. Conceptual mapping:
+
+| Effort | Behavior |
+| --- | --- |
+| `low`      | Stay on Tier 0 unless hard-fail. Never fan-out. |
+| `balanced` | Default. Escalate on stuck-progress and structured-quality. Fan-out only on tier-config that explicitly opts in. |
+| `high`     | More aggressive escalation, allows fan-out on Tier 1+. |
+| `max`      | Fan-out on multiple tiers when configured; never silently degrade. |
+
+### Subscription accounting
+
+Subscription depletion is observed per-provider:
+
+- Provider adapters expose a `subscriptionState()` returning
+  `{ available: bool, resetsAt?: Date, reason?: string }`.
+- The router consults it before picking a sub-class provider in a
+  pool. On `available: false`, it falls through to the next member
+  (typically metered) until reset.
+- Adapters are responsible for parsing rate-limit / quota signals
+  from their provider's responses.
+
+## Milestones
+
+### M1 — Schema + dry-run router *(done — 2026-05-21)*
+
+- [x] Tier config Zod schema in
+  `packages/opencoti-tiers/src/config.ts` (`Effort`, `Billing`,
+  `FanoutMode`, `EscalationTrigger`, `Provider`, `Tier`,
+  `Escalation`, `TierConfig`). Strings normalize to providers via
+  `z.preprocess`; defaults: `enabled=false`, `effort=balanced`,
+  `escalation.on=[hard-fail]`, fanout `mode=first-good`.
+- [x] Provider-state model in `src/provider-state.ts`
+  (`ProviderState`, `AVAILABLE`, `exhausted()`).
+- [x] Effort presets in `src/effort.ts` mapping `low|balanced|high|max`
+  to `{ maxTiersToTry, allowFanout, defaultEscalationTriggers }`.
+- [x] Pure `plan()` function in `src/router.ts` that produces a
+  `Plan` (ordered `PlanStep`s with `attempts`, `skipped`,
+  `blockedByEffort`, `notes`). No real provider calls.
+- [x] 17 unit tests (`bun test`) cover: schema parsing + rejects
+  (empty chain, unknown effort, unknown fanout mode), provider
+  normalization (string → object, billing preserved), linear
+  escalation order with subscription-first reordering,
+  subscription-exhausted fallback (single and all-exhausted),
+  fanout modes (`best-of-n`, `quorum`) preserving config order,
+  effort presets (`low` caps to tier 0; `balanced` downgrades fanout
+  to linear; `high` preserves fanout; explicit effort overrides
+  config).
+- Typecheck (`tsc --noEmit`) clean.
+
+### M2 — Surgical hook into opencode model call *(done — 2026-05-21)*
+
+- [x] Mapped the call site to
+  `packages/opencode/src/session/llm.ts` (`run` Effect.fn around the
+  `streamText({...})` call). The file already has a sister
+  experimental-native runtime seam — opencoti's hook is its
+  immediate neighbor, mirroring the same shape (one Effect.fn yield,
+  one conditional early return).
+- [x] Added `packages/opencoti-tiers/src/runtime.ts` exposing
+  `TierRuntime.maybeRoute(input): Effect<TierRunResult | undefined>`
+  and `TierRuntime.isEnabledIn(cfg)`. M2 implementation is a
+  passthrough (`Effect.succeed(undefined)`); M3 swaps it for the
+  real router → adapter flow without touching upstream again.
+- [x] `TierRunResult` is typed to match opencode's `run` return
+  shape (`{type: "ai-sdk"; result: ReturnType<typeof streamText>}`
+  | `{type: "native"; stream: never}`). The `never` on the native
+  stream keeps the inferred return-type union at the call site
+  identical to upstream's — `never` is the absorbed element in
+  discriminated-union narrowing.
+- [x] Surgical hook in `packages/opencode/src/session/llm.ts`:
+  - import block (line ~29): `import * as TierRuntime from "@opencoti/tiers/runtime"`
+    tagged with the `// opencoti-hook: tiered-inference — see docs/features/tiered_inference.md` marker.
+  - call site (just before the `flags.experimentalNativeLlm` seam):
+    `const tier = yield* TierRuntime.maybeRoute({ input, prepared, cfg }); if (tier) return tier`
+    tagged with the same marker.
+- [x] `packages/opencode/package.json` gains
+  `"@opencoti/tiers": "workspace:*"`. No comment marker is possible
+  in JSON; the dep is registered in `docs/protocols/UPSTREAM_SYNC.md`.
+- [x] All three hook entries registered in the UPSTREAM_SYNC.md
+  hook registry.
+- [x] `bun typecheck` from repo root: 17/17 packages green (including
+  `opencode` with the new hook).
+- [x] `bun test` in `packages/opencoti-tiers`: 23/23 green, including
+  6 new tests for `TierRuntime.maybeRoute` + `TierRuntime.isEnabledIn`.
+
+Net hook footprint in upstream source: **one import line + a
+two-line conditional + one JSON dep entry** — the minimum that lets
+M3 wire real routing without ever editing `llm.ts` again.
+
+### M3 — Tier 0 = llamafile end-to-end *(done — 2026-05-21)*
+
+- [x] `TierProvider` interface defined in
+  `packages/opencoti-tiers/src/provider.ts` (`TierProvider`,
+  `OpenedTierProvider`, `TierStreamRequest`). Types-only; no
+  dependency on any concrete provider package — the dep direction is
+  llamafile → tiers, not the other way around.
+- [x] Llamafile implementation in
+  `packages/opencoti-llamafile/src/tier-provider.ts`
+  (`tierProvider()` factory). Singleton process per host (lazy
+  promise cache); uses `@ai-sdk/openai-compatible` to build a
+  `LanguageModel` against the running llamafile's
+  `/v1/chat/completions` endpoint.
+- [x] `TierRuntime.maybeRoute` replaced with real routing logic in
+  `packages/opencoti-tiers/src/runtime.ts`:
+  - Env gate (`OPENCOTI_TIERS_ENABLED=true`) — opt-in by design at
+    M3.
+  - Dynamic-imports `@opencoti/llamafile/tier-provider` via a string
+    variable, so `@opencoti/tiers` has no static dep on llamafile
+    (would be circular otherwise — llamafile already depends on
+    tiers for the interface).
+  - All failure modes (provider not installed, launch fails, open
+    fails) fall through to upstream by returning `undefined`. The
+    surgical hook never throws — it's forgiving by design.
+  - Constructs `streamText` with the opened `LanguageModel` and the
+    upstream `prepared` request (messages, system, params, abort).
+- [x] **No further edits to `packages/opencode/src/session/llm.ts`.**
+  The M2 surgical hook is the last upstream touch needed for this
+  feature. M3 lands entirely inside opencoti packages.
+- [x] Tests: 24 unit tests passing in `@opencoti/tiers`
+  (env-gate-off, env-gate-on-but-no-llamafile, `isEnabledIn`
+  variants); 2 unit tests in `@opencoti/llamafile` covering
+  `tierProvider()` factory shape. Live `open()` covered by the
+  smoke script (manual).
+
+**M3 known limitations** (intentional — addressed by later
+milestones):
+- Tool calling not wired through. M3 calls `streamText({model,
+  messages, system, params})` without forwarding `prepared.tools`.
+  Many local llamafile models can't tool-call usefully anyway; tool
+  routing comes when M5 brings escalation triggers and tools
+  matter.
+- Config is env-var-driven, not opencoti-config-driven. The
+  user-facing tier config (effort, chain, escalation) defined in
+  `@opencoti/tiers/config` is **not yet read** by M3's
+  `maybeRoute` — M4 wires it in (and removes the env gate in favor
+  of `opencoti.tiers.enabled` in the config file).
+- Single hardcoded llamafile tier. No escalation, no fan-out — those
+  are M4, M5, M6.
+- Singleton llamafile process for host lifetime. If the user
+  reconfigures the model mid-session, M3 ignores the new model. M3+
+  refines lifecycle.
+
+### M4 — Tier 1 cloud fallback (linear) **[DONE 2026-05-21]**
+
+- One subscription provider + one metered fallback work end-to-end
+  with subscription exhaustion observed correctly.
+- Implementation:
+  - `cloud-provider.ts` — generic `TierProvider` over any
+    OpenAI-compatible HTTP endpoint. Preflight GET `${baseURL}/models`
+    classifies the response: 200 → AVAILABLE; 401/403 → exhausted
+    (unauthorized); 402 → exhausted (payment-required); 429 →
+    exhausted with `resetsAt` parsed from `Retry-After` (both numeric
+    seconds and HTTP-date forms). State lives in the provider's
+    closure and is exposed via `getState()`.
+  - `registry.ts` — pure resolver mapping a config `Provider` entry
+    to a concrete `TierProvider`. `kind: "llamafile"` (or string
+    shorthand `"llamafile"` / `"llamafile/…"`) dynamic-imports
+    `@opencoti/llamafile/tier-provider`; `kind: "openai-compat"` with
+    `baseURL` + `modelId` builds a `cloudProvider`.
+  - `executor.ts` — walks the `plan()` output across tiers and
+    providers. On each iteration it collects state from each
+    provider's `getState()` plus a module-level overlay (for
+    `open()` failures that don't self-mark) and calls `plan()` again.
+    A bounded loop (`totalProviders + 1`) prevents runaway replanning.
+  - `runtime.ts` — env-var gate **removed**; replaced by
+    `isEnabledIn(cfg)` reading `cfg.opencoti.tiers.enabled === true`,
+    plus `safeParseConfig` on `cfg.opencoti.tiers`. The hook then
+    delegates to `execute()` and wraps the returned `languageModel`
+    in `streamText`. `maybeRouteWith` is a test seam letting tests
+    inject an in-memory resolver.
+- Schema additions (in `config.ts`): provider entries now accept
+  `kind: "llamafile" | "openai-compat"`, `baseURL`, `apiKey`,
+  `apiKeyEnv` (preferred — read at runtime), `modelId`. Backwards
+  compatible: existing string-shorthand entries (`"llamafile/…"`)
+  still parse.
+- Test coverage: 47 tests across `config.test.ts`, `router.test.ts`,
+  `runtime.test.ts`, `cloud-provider.test.ts`, `executor.test.ts`.
+  Cloud preflight tests use `fetchImpl` injection — no live HTTP.
+- Zero upstream edits — M4 is entirely additive.
+- Out of scope for M4 (deferred to M5+):
+  - **Cross-tier** escalation triggers (hard-fail, stuck-progress,
+    structured-quality). M4 only covers within-tier sub→metered.
+  - Forwarding `prepared.tools` through `streamText`.
+
+### M5 — Escalation triggers v1 **[DONE 2026-05-21]**
+
+In/post-stream escalation observer wraps the M4 pre-stream router so
+the engine can react to bad turns and route the *next* turn to a
+higher tier. All four configured triggers (`hard-fail`,
+`stuck-progress`, `structured-quality`, `explicit`) work end-to-end.
+
+- Architecture (locked from planning conversation):
+  - **Model-as-judge structured-quality**: the model's in-stream
+    behavior IS the quality verdict — tool errors, no progress,
+    explicit bail-out — NOT a parallel validator registry. The user's
+    tools (test runners, language servers) are already the validators.
+  - **Per-turn escalation unit**: one `streamText` invocation = one
+    chance. Per-session counters survive across turns. When a
+    threshold is met, the provider is overlay-marked; the next turn
+    picks the next tier via the existing executor machinery.
+  - **Distinct executor + escalator primitives** sharing the
+    `stateOverlay`. Executor stays pre-stream (M4). Escalator wraps
+    `streamText` and is in/post-stream.
+  - **No mid-stream stream-swap** — mid-stream errors mark the
+    provider exhausted and let the next turn re-route. Only
+    pre-first-chunk hard-fail triggers a same-turn retry via a
+    `restartStream` callback the runtime supplies.
+- Implementation:
+  - `escalator.ts` — `wrapWithEscalation({result, opened, cfg,
+    sessionID, telemetry, restartStream?}) → StreamTextResult`.
+    Returns a `Proxy<StreamTextResult>` that overrides only
+    `fullStream`; the wrapped `fullStream` is an async generator that
+    tees events into a `TurnObserver` and re-yields them. The
+    `TurnObserver` counts tool-call / tool-result / tool-error / error
+    events, evaluates triggers at stream finish, calls
+    `_markEscalated(providerName, "trigger:reason")` from `executor.ts`
+    when a threshold is hit, and emits telemetry.
+  - `session-state.ts` — `Map<sessionID, SessionTierState>` with 30-min
+    TTL eviction on read. Fields: `stuckTurns`, `qualityFails`,
+    `lastDifficulty`, `lastTier`, `lastAccess`,
+    `explicitEscalationPending`.
+  - `synthetic-tools.ts` — `syntheticTools(sessionID, telemetry?)`
+    returns `{ __tier_report_difficulty, __tier_request_escalation }`
+    as AI-SDK `Tool` objects with `inputSchema` (zod) and `execute`
+    handlers that record into session state and emit telemetry. Both
+    names start with `__tier_` so any consumer can filter them.
+  - `prompt-augmentation.ts` — `augmentSystem(systemLines, enabled)`
+    appends one narrow paragraph telling the model about the two
+    synthetic tools when `enabled` is true.
+  - `telemetry.ts` — `Telemetry` interface with 7 methods
+    (`attemptStart`, `attemptSuccess`, `exhaustionObserved`,
+    `escalationTriggered`, `difficultyReported`,
+    `requestEscalationCalled`, `sessionStateUpdated`) + a
+    `noopTelemetry` default. F3 (`opencoti-server`) will eventually
+    implement this interface; M5 ships the schema only.
+  - `executor.ts` extensions: `ExecuteResult.tierId` (so the escalator
+    knows which tier opened the stream); `telemetry?` in
+    `ExecutorOptions`; `_markEscalated(name, reason)` exported
+    package-internal; telemetry calls at `attemptStart` /
+    `exhaustionObserved` branch points.
+  - `runtime.ts` extensions: merges `prepared.tools` with
+    `syntheticTools(sessionID, telemetry)` (the M3/M4 known limitation
+    is gone); replaces `system: ...join("\n")` with
+    `augmentSystem(...).join("\n")`; provides a `restartStream`
+    callback to the escalator that re-calls `execute()` (the just-marked
+    provider is skipped automatically).
+  - Schema additions in `config.ts`:
+    `escalation.qualityFails.threshold` (defaults to 2).
+- Trigger semantics:
+  - `hard-fail`: pre-first-chunk error → silently restart THIS turn
+    once via `restartStream`. Mid-stream errors → mark exhausted,
+    escalate NEXT turn. User-AbortSignal abort is excluded.
+  - `stuck-progress`: turn finishes with zero tool calls AND
+    `lastDifficulty != "low"` → increments `stuckTurns`. Escalates
+    when `stuckTurns >= cfg.escalation.stuckProgress.noProgressTurns`.
+    A productive turn (≥1 tool call) resets the counter.
+  - `structured-quality`: any `tool-error` event → increments
+    `qualityFails`. Escalates when `qualityFails >=
+    cfg.escalation.qualityFails.threshold`. A clean turn with ≥1
+    successful tool result resets the counter.
+  - `explicit`: any call to `__tier_request_escalation` (model bailout)
+    OR an `explicitEscalationPending` flag from a prior turn →
+    immediate escalation regardless of threshold.
+- Test coverage: 33 new M5 tests (`escalator.test.ts`,
+  `session-state.test.ts`, `synthetic-tools.test.ts`,
+  `prompt-augmentation.test.ts`, `telemetry.test.ts`, + 3 extensions
+  to `runtime.test.ts`). Total `@opencoti/tiers`: 80 tests, all green.
+  Workspace typecheck clean.
+- Zero upstream edits — M5 is entirely additive in
+  `packages/opencoti-tiers/`. Hook footprint unchanged: still the 2
+  M2 markers in `packages/opencode/src/session/llm.ts`.
+
+### M5.5 — Effort-level + tiermode schema expansion **[DONE 2026-05-21]**
+
+Pure-schema milestone, no runtime change beyond what the new config
+shape brings.
+
+- **Effort enum**: replaced `low|balanced|high|max` with a 5-base set
+  `low|normal|medium|high|max` plus 5 x-prefixed fanout opt-ins
+  `xlow|xnormal|xmedium|xhigh|xmax`. The `x` prefix marks "fan-out at
+  this tier"; aggression is the base level, fanout is the orthogonal
+  bit. No `balanced` legacy alias — pre-1.0, follow CLAUDE.md's
+  "don't keep backwards-compatibility shims" rule.
+- **`effort.ts` constraints table** rebuilt as `5 × 2` grid:
+  - `low` / `xlow`: cap at one tier; `hard-fail` trigger only.
+  - `normal` / `xnormal`: walk all tiers; `+ stuck-progress`.
+  - `medium` / `xmedium`: walk all tiers; `+ structured-quality`.
+  - `high` / `xhigh`: same trigger set as medium; reserved for later
+    threshold-tuning differentiation.
+  - `max` / `xmax`: same trigger set, most-aggressive thresholds.
+  - `allowFanout` is `false` for the bare base levels, `true` for the
+    `x*` opt-ins. (Behavior change: old `high`/`max` enabled fanout;
+    that capability moved to `xhigh`/`xmax`.)
+- **Tiermode**: new top-level config field. `tiermodes:` is a map of
+  named profiles; each profile may override `effort`, per-tier
+  `topologyOverrides`, or a partial `escalation` overlay. The
+  top-level `tiermode: string` field points into the map and is
+  validated at parse time (refinement: name must exist).
+- **`tiermode.ts` resolver**: `resolveTiermode(cfg) → ResolvedTierConfig`
+  pure function. Returns the effective `{ effort, topologyOverrides,
+  escalation, config }` after profile merge. Router and runtime do
+  NOT consume it yet (M5.5 is pure schema); the resolver is package-
+  exported so M6/M7 wiring is trivial.
+- **Tests**: total `@opencoti/tiers` test count up from 80 to 107
+  (+27 net: 10 new + parameterized parsing across all 10 effort
+  values + 5 tiermode parsing + 6 resolver tests; 2 existing fanout-
+  tests migrated from `high`/`max` to `xhigh`/`xmax`). Workspace
+  typecheck clean. Zero new upstream surgical hooks.
+
+### M5.7 — Tier-0 provider configuration + model registry **[DONE 2026-05-21]**
+
+User-facing first-run setup and a small project-local model registry.
+Lands after M5.5 (schema is final by then) and before M6 (fan-out
+needs a working multi-provider Tier 0 to test against). No runtime
+change beyond what config-read needs.
+
+- **Model registry.** A `models:` map in the config (see Config-shape
+  example above) maps short names (`gemma4-a4b-coder`,
+  `gemma4-31b-cloud`, …) to provider entries. The `chain[].providers`
+  field references those names. Same idea as claude-hooks' provider
+  registry. Schema lives in `packages/opencoti-tiers/src/config.ts`.
+- **First-run setup CLI** (`opencoti tier setup` or equivalent —
+  exact subcommand TBD, runs from the standard opencode CLI surface):
+  1. GPU detection. Reads `nvidia-smi`, `rocm-smi`, or an equivalent
+     probe. Reports detected VRAM.
+  2. **GPU path branch.** Prompts:
+     - Confirm Gemma 4 A4B-98e-v5-coder as the Tier-0 model (yes by
+       default, with the option to point to a different GGUF).
+     - Quant pick — IQ3_M default for ≥10 GB VRAM, IQ4_XS or CD-IQ4_K_M
+       for ~11 GB, Q4_K_S for ~12 GB, Q5_K_L for ~15 GB, Q8_0 for 24 GB.
+       Recommended pick is computed from detected VRAM and shown as
+       the default.
+     - GGUF source — three options:
+       - **Download from HuggingFace.** Downloads to
+         `$XDG_DATA_HOME/opencoti/models/` (`~/.local/share/opencoti/models/`
+         on Linux default) using the user's HF auth if any.
+       - **Use existing file.** User points at a GGUF path. Validated
+         (magic bytes + metadata read) before being recorded.
+       - **Import from local Ollama.** If `~/.ollama/models/` is
+         detected, parse manifests under
+         `~/.ollama/models/manifests/registry.ollama.ai/library/<model>/<tag>`
+         to find the blob digest; the patched llamafile mounts
+         `~/.ollama/models/blobs/sha256-<digest>` directly. **Single
+         download, zero duplication.**
+  3. **No-GPU path branch.** Prompts:
+     - Confirm Ollama Cloud as the Tier-0 provider.
+     - Recommend `gemma4:31b` (free BF16 on Ollama Cloud at time of
+       writing); allow override.
+     - Capture `OLLAMA_CLOUD_API_KEY` via env-var reference (never
+       stored in config — `apiKeyEnv` indirection).
+  4. **No silent auto-fallback.** If GPU detection fails and the user
+     hasn't picked a path, setup exits with a clear next-step prompt.
+     No defaulting to cloud.
+  5. Writes the chosen entries into the project's opencoti config
+     (`opencoti.tiers.models[...]` and `chain[0].providers[0]`).
+- **Ollama local registry detection.** Helper module reads the
+  manifest JSON, follows the `digest` field to the blobs/ entry,
+  returns `{path, sha256, sizeBytes}`. Pure read-only — never modifies
+  the Ollama store. Tests use a fixture `~/.ollama/` tree.
+- **Reuse pattern in llamafile launcher.** When a registry entry
+  points at an Ollama blob (kind: `llamafile`, gguf path inside
+  `~/.ollama/`), the launcher passes the blob path directly to the
+  patched llamafile — no copy, no symlink, just read. Verified by a
+  smoke test.
+- **Test coverage:** new test files in `packages/opencoti-tiers/test/`
+  (`model-registry.test.ts`, `ollama-detect.test.ts`) and
+  `packages/opencoti-llamafile/test/` (`ollama-blob-reuse.test.ts`).
+  GPU detection itself is shimmed in tests (the real probe lives
+  outside `@opencoti/tiers` to keep the package testable in CI).
+
+Zero new upstream surgical hooks. M5.7 is entirely additive in
+opencoti packages + a new CLI subcommand exposed via opencode's
+existing command surface.
+
+**M5.7 implementation notes (what actually shipped):**
+
+- Schema: `ModelEntry` + `models: Record<string, ModelEntry>` on
+  `TierConfig`. Refinements enforce "llamafile requires exactly one
+  of `gguf`/`ollamaBlob`" and "openai-compat requires `baseURL`".
+- `src/registry/ollama-detect.ts` — read-only manifest walker
+  (`registry.ollama.ai/<ns>/<model>/<tag>`), follows the GGUF layer
+  digest to `~/.ollama/models/blobs/sha256-…`. Tolerant of malformed
+  manifests.
+- `src/registry/gpu-detect.ts` — async `detectGPU()` probes
+  `nvidia-smi --query-gpu=...` first, falls back to
+  `rocm-smi --showmeminfo vram --json`. 5 s default timeout.
+  `runImpl` test seam.
+- `src/registry/quant-recommender.ts` — pure table-driven picker
+  encoding the Gemma 4 A4B-98e quant table from
+  `local_llamafile.md`. Reserves ~3 GB KV per 32 K context + 1.5 GB
+  safety; picks highest-quality (HE+) quant that fits.
+- `src/registry/hf-download.ts` — minimal HF Hub HTTP client. Resume
+  via `${destPath}.partial` + `Range: bytes=N-`, optional
+  `$HF_TOKEN` auth, SHA256 verify (deletes the partial on mismatch),
+  per-chunk progress callback.
+- `src/cli/setup-flow.ts` — pure-logic flow with injected deps
+  (`SetupDeps`). `runSetupFlow(deps)` returns `cancelled` or
+  `configured` with the `ConfigPatch` to write. Testable via
+  scripted-answers stub deps.
+- `src/cli/setup.ts` — live wiring: @clack/prompts for input,
+  `jsonc-parser` `modify`/`applyEdits` for comment-preserving
+  config write to `~/.config/opencode/opencode.jsonc`. Exports
+  `tierCommand` (yargs parent) and `tierSetupCommand` (subcommand).
+- Surgical hook in `packages/opencode/src/index.ts`: 1 import +
+  1 `.command(tierCommand)` call, both marker-tagged. Both
+  registered in `docs/protocols/UPSTREAM_SYNC.md`.
+- `package.json`: adds `@clack/prompts`, `jsonc-parser`, `yargs`
+  as deps (same pinned versions opencode uses) and
+  `@types/yargs` as devDep. New `./cli` export.
+- Test coverage: 35 net new tests across 5 new files
+  (`ollama-detect.test.ts` 6, `gpu-detect.test.ts` 6,
+  `quant-recommender.test.ts` 10, `hf-download.test.ts` 7,
+  `cli/setup-flow.test.ts` 8). Total `@opencoti/tiers`: 144 tests.
+  Workspace typecheck clean. Hook conformance: exactly 4 markers
+  (2 M2 + 2 M5.7).
+
+### M6 — Fan-out tier (first-good) **[DONE 2026-05-21]**
+
+Goal: when the plan picks a fanout tier (xhigh / xmax effort, or
+explicit `tiermode` override), open every provider in that tier in
+parallel and let the fastest useful response through. Losers are
+aborted; losers that fail before being aborted are overlay-marked
+the same way the M5 linear path marks them.
+
+Behavior knobs the user locked this turn:
+
+- **`winnerTrigger` (per tier)**: `"content"` (default) or `"finish"`.
+  `content` races on the first real-content event (same predicate
+  the M5 escalator uses — `text-delta`, `tool-call`, `tool-result`,
+  `file`, etc.). `finish` races on the first racer to complete a
+  clean turn.
+- **Pre-trigger loser failures** → overlay-marked via
+  `_markEscalated(name, "fanout-prefail")`. Aborts-by-us are NOT
+  marked (the racer was cut off through no fault of its own —
+  tracked via a `abortedByUs` flag).
+- **`maxConcurrency` (per tier, optional)**: caps how many providers
+  actually race. The rest are recorded in `skippedByConcurrencyCap`
+  on the executor result and are NOT overlay-marked — they just sit
+  out this turn.
+
+The fanout `mode` enum was expanded to include `"synthesizer"` and
+`"synthesizer-critic"` for forward-compatibility with the consultants-
+council pattern (Tier 0 as synthesizer + separate critic model).
+M6 ships `"first-good"` only; `"best-of-n"`, `"quorum"`,
+`"synthesizer"`, and `"synthesizer-critic"` throw a typed
+`FanoutModeNotImplemented` error that the runtime catches as a
+forgiving fall-through.
+
+Implementation surface (all additive in `packages/opencoti-tiers/`):
+
+- `src/config.ts` — `FanoutTier` gets `winnerTrigger` + `maxConcurrency`;
+  `FanoutMode` enum expanded.
+- `src/executor.ts` — `executeFanout()` opens every router-emitted
+  attempt for the first fanout step in parallel via `Promise.all`,
+  applying `maxConcurrency` and overlay-marking open-failures the
+  same way `execute()` does.
+- `src/fanout.ts` (new) — `raceFanout()` runs N per-racer drain
+  goroutines, exposes `{proxy, outcome, settled}`. The proxy is a
+  `Proxy<StreamTextResult>` whose `fullStream` getter replays the
+  winner's buffered pre-trigger events then continues from the
+  underlying iterator. Per-racer buffer cap (default 4 MB) catches
+  pathological non-content streams.
+- `src/runtime.ts` — `peekTopology()` decides linear vs fanout
+  branch from `plan()`. The fanout branch builds per-racer
+  streamText invocations with child `AbortController`s, calls
+  `raceFanout`, then wraps the proxy with `wrapWithEscalation` so
+  M5's escalation triggers apply on the winner's stream. The
+  `restartStream` callback handles the `all-failed` outcome the
+  same way M5's hard-fail-pre-stream path does.
+- `src/escalator.ts` — `isRealContent` was exported so `fanout.ts`
+  shares the exact same predicate.
+
+Test coverage: 20 net new tests across
+`test/{config,executor,runtime,fanout}.test.ts`. Total
+`@opencoti/tiers` count: **168 tests**.
+
+Zero new surgical hooks. Hook count stays at 4 (2 M2 + 2 M5.7).
+
+### M7 — TUI effort + tiermode pickers **[DONE 2026-05-21]**
+
+Goal: in-TUI dials for the M5.5 `effort` (10 values) and `tiermode`
+(named profiles) so users can change them mid-session without
+re-editing `~/.config/opencode/opencode.jsonc`. Plus a runtime-side
+override seam so a TUI pick actually takes effect on the very next
+request, plus auto-wire of the new picker plugin into the M5.7
+setup CLI.
+
+Three additive surfaces:
+
+- **`@opencoti/tiers/src/active-config.ts`** — module-level overlay
+  (`setActiveEffort` / `setActiveTiermode` / `getActiveOverride`)
+  matching the existing `stateOverlay` pattern. Both the TUI plugin
+  and the M2 hook run in the same Bun process, so module-level
+  singleton state is genuinely shared.
+- **`runtime.ts` overlay + tiermode wiring** — `routeImpl` now
+  merges the overlay onto the parsed config, calls `resolveTiermode`
+  (which M5.5 left unwired), patches each tier's topology per the
+  resolver output, and forwards the effective effort to the
+  executor via `ExecutorOptions.effort`. Precedence:
+  **TUI overlay > tiermode profile > base `cfg.effort`**.
+- **`@opencoti/tui-tiers`** — new sibling package. Ships an opencode
+  TUI plugin (`TuiPluginModule`) that registers two commands
+  (`opencoti.effort`, `opencoti.tiermode`), opens
+  `DialogSelect`s, and persists picks via `api.kv` under a
+  per-project key (`opencoti.<basename-hash12>.{effort|tiermode}`)
+  so each project remembers its own selection. On plugin start, the
+  persisted values replay into the runtime overlay — TUI restart
+  preserves the last pick.
+
+Setup CLI extension (M5.7 → M7):
+- `setup-flow.ts` got a `tuiPickerInstalled` sentinel on `SetupDeps`
+  and an `installTuiPicker` boolean on `ConfigPatch`. After the
+  GPU/no-GPU branch builds its patch, a `finalizePatch` helper asks
+  "Install the TUI picker plugin?" (skipped when the sentinel is
+  true — re-runs don't re-prompt).
+- The live `setup.ts` reads the sentinel from the existing config
+  (best-effort `parseTree` + `findNodeAtLocation`) and pre-populates
+  `tuiPickerInstalled` on the live deps. When `installTuiPicker` is
+  true on the patch, the writer appends `"@opencoti/tui-tiers"` to
+  the user's `plugin: []` (deduped against the existing array) and
+  flips `opencoti.tiers.tuiPickerInstalled = true`.
+
+Schema: `TierConfig` got one new optional sentinel field
+`tuiPickerInstalled: z.boolean().default(false)`. Pure schema —
+defining it in Zod makes a typo a clear error instead of silent.
+
+Test coverage: **16 net new tests** across `@opencoti/tiers`
+(active-config, runtime, executor, setup-flow) + **12 tests** in the
+new `@opencoti/tui-tiers` package. Totals: tiers **184 tests**,
+tui-tiers **12 tests**, workspace 18/18 typecheck clean.
+
+Zero new surgical hooks. The TUI plugin loads via opencode's
+existing `plugin: []` discovery path (the setup-flow auto-wire
+writes the entry). Hook footprint stays at 4 markers.
+
+**Behavior change worth noting**: `resolveTiermode` was unwired
+since M5.5. Wiring it in M7 means users who set `tiermode` in their
+config — silently ignored pre-M7 — now actually see the profile
+apply. This is a fix, not a regression, but anyone debugging
+tiermode behavior should know the activation date is 2026-05-21.
+
+### M8 — Structured-quality validators v1
+
+- Pluggable validators (code-parses, tests-pass-or-improve). Trigger
+  escalation when validator fails N times on the current tier.
+
+## Open questions
+
+- ~~Where exactly is the upstream call site?~~ Resolved at M2:
+  `packages/opencode/src/session/llm.ts`, just before the existing
+  experimental-native-llm seam.
+- **Streaming under fan-out.** First-good is fine; quorum and
+  best-of-n are not naturally streaming. Defer their UX to M6+.
+- **Cost accounting.** Should opencoti show per-tier token spend?
+  Probably yes, but it's not on the critical path. The M5 telemetry
+  interface already carries the data; F3's web dashboard is the
+  obvious place to surface it.
+- **Capability negotiation between escalator and provider.** The
+  patched llamafile exposes capabilities the no-GPU path lacks (KV
+  reuse, big-context throughput). Today the executor doesn't ask
+  providers what they can do — it just opens them. M5.7+ may want to
+  add a `capabilities()` method to `TierProvider` so the escalator
+  can know whether KV-prefix-reuse is on the table for the next turn.
+  Not blocking; revisit when the second patched-llamafile feature
+  lands.
+
+## Risks
+
+- The opencode provider abstraction may not factor cleanly enough for
+  a single hook. If so, we widen the hook to a small adapter shim in
+  the upstream call site, and keep the adapter implementation in
+  `opencoti-tiers`. The hook stays small; the implementation stays
+  ours.
+- Provider rate-limit signal parsing is fragile. We isolate it per
+  provider adapter so a broken adapter never poisons the router.

docs/features/tstar_auto_spill.md

@@ -0,0 +1,189 @@
+# T\*-aware auto rolling-KV spill (HAL #582 P1+, #621 follow-on)
+
+Status: **T1–T5 VALIDATED on both hosts, 2026-07-11. T5 quality/tps gate PASS.** Opt-in under
+`OPENCOTI_KV_AUTO_TIER=1` (same switch as the 0116 auto-tier). Builds on patch 0116
+(`opencoti_auto_select_kv_tier`) and the rolling-KV window (0070/POSITION_WINDOW). **Host-only
+— the FA microbench dispatches to the existing CUDA DSO, so no DSO rebuild.** Byte-identical
+when the env is unset (opt-in gate returns before any measurement). Patch 0118 capture +
+final commit are the only remaining step (held for explicit go) — see "T5 verdict" below.
+
+## T5 verdict (2026-07-11, bs2 GPU0, Qwen2.5-14B-1M Q8_0, ctx 65536, DCA-off)
+
+The physics reframes the gate: T\*(0.20) is only ~14–17 MiB at this config (small `t_resident`
+at 65k, `c_engage ~1 ms` eats most of `0.20·t_resident`) while f16 spills ~2 GB, so the SHIP
+question is not "does auto KEEP f16" (a razor-thin band) but **"is T\*'s quantize-vs-keep
+verdict CORRECT?"**. Four uniform RULER-niah cells (`--vram-target 27000` forces the spill):
+
+| cell | niah | decode tps |
+|---|---|---|
+| f16_resident (vram 0) | 100.0 | 50.2 |
+| f16_spill (~2 GB, window) | 100.0 | **10.81** |
+| q8_resident (fits) | 100.0 | 35.26 |
+| auto (`OPENCOTI_KV_AUTO_TIER=1`) | 100.0 | 35.28 |
+
+**PASS — all four checks:** (1) **auto → q8_0/q8_0**: boot WARN shows *"top tier f16 SPILLS by
+1630 MiB … T\* tolerable spill 14 MiB (bw_eff=10.3 host-RAM-bound, target_drop=0.20) → auto KV
+tier = q8_0/q8_0 (fits)"* — the drop-model rejects f16 because 1630 ≫ 14 MiB. (2) **f16-spill tps
+drop = 78.5%** (50.2→10.81) ≫ 20% target — keeping f16 would cost far more than `target_drop`,
+so rejecting it was justified (host-RAM-bound 10.3 GB/s link craters even a 2 GB spill). (3) **q8
+niah = f16 niah = 100.0** — quantizing lossless-enough; f16_spill niah 100.0 also confirms the
+streaming window is itself lossless. (4) **q8 tps 35.26 ≥ f16_spill 10.81** — the fallback
+strictly dominates (3.3× faster AND lossless). Consistency: **auto 35.28 ≈ q8_resident 35.26**
+(auto's end-to-end matches the tier it selected). Correctness via RULER niah, never greedy needle.
+
+## Validation (2026-07-11, Qwen2.5-14B-Instruct-1M, both hosts)
+
+Measured terms confirm T2 (bandwidth probes) + T3 (c_engage microbench) fire and T4 (drop model)
+responds monotonically to `target_drop`:
+
+| host | pcie | host_ram (1-thread memcpy) | **bw_eff = min** | binding | fa_ms/cell | c_engage |
+|---|---|---|---|---|---|---|
+| solidPC (3090, PCIe4, DDR4) | 6.7 | 11.5 | **6.7** | PCIe-bound | 6.2e-4 | 0.80 ms |
+| bs2 (RTX 6000, PCIe5, DDR5) | 51.2 | 9.3 | **9.3** | **host-RAM-bound** | 2.6e-4 | 0.82 ms |
+
+**req #3 demonstrated:** the binding bandwidth is host-dependent — PCIe on solidPC, host-RAM on
+bs2 (fast PCIe5 makes DDR5 single-stream read the bottleneck). `min()` picks correctly on each;
+without the host-RAM probe bw_eff would be 5× too optimistic on bs2. T\* drop-response (solidPC,
+ctx 106 k, f16 spilling): tolerable spill **0 → 62 → 157 → 235 MiB** as drop 0.02 → 0.30 → 0.70
+→ 0.99 (jitter-immune, same-boot). `T*=0` at drop 0.02 (c_engage > 0.02·t_resident) confirms the
+safe degradation to the T1 "fully resident" gate. Byte-identical when env unset: 0 auto-tier WARN
+lines. **Caveat:** on the 3090's 6.7 GB/s link T\* is a few tens–hundreds of MiB, so a tier *flip*
+via T\* is only reachable at long ctx on high-bandwidth hosts (bs2); on the 3090 the model
+correctly almost always quantizes. `host_ram` is a single-thread `memcpy` proxy (conservative
+under-estimate of DMA-engine host read) — bug-2151 fixed a dead-store-elimination defect that had
+made it return garbage (~2e7 GB/s). Diagnostics are WARN-level (`budget…top tier`, `T* drop-model
+— …tolerable spill`) mirroring the existing `auto KV tier` WARN, for operator auditability.
+
+## Problem
+
+0116 auto-tier picks the least-compressing scalar KV pair that keeps the cache **fully
+resident**. But a *higher* KV tier (better quality) with a *small* host-tail spill can beat a
+harder-compressed fully-resident tier — **when** the spill tail is within a "negligible
+tps-drop" band. This spec adds that decision: prefer `f16-window + small spill` over
+quantizing, but only when the spill is provably cheap on *this* host + model + context.
+
+## The decision (fit-condition reformulation)
+
+The 0116 ladder walk changes from "first tier that fully fits" to "first (least-compressing)
+tier whose **spill** ≤ T\*":
+
+```
+choose first tier i (least→most compression) s.t.  kv_bytes(i) ≤ budget + T*_bytes
+```
+
+`budget` is the existing resident budget (free − already-used − compute-reserve, capped by
+`--vram-target`), identical to `opencoti_compute_resident_window_cells`. When even the floor
+tier spills past T\*, pick the floor (rolling-KV spills the rest, as today). If f16's spill ≤
+T\*, f16 is chosen and the window spills it — **Policy C: prefer f16-window+spill over
+quantizing** (user, 2026-07-11).
+
+## T\* ��� the negligible-drop tail band
+
+Target drop = **20%** (user raised from 5% — at 5% it essentially never engaged because the
+fixed streaming-engagement step already costs ~15%). T\* solved from a per-token drop model:
+
+```
+drop(tail) = ( c_engage + tail_bytes / bw_eff ) / t_resident
+T*_bytes   = ( 0.20 · t_resident − c_engage ) · bw_eff          // ≤ 0  ⇒ never spill → quantize
+```
+
+Terms and how each user requirement maps in:
+
+- **`t_resident`** = `(W_active + KV_resident_bytes) / vram_bw` — per-token time with the cache
+  resident. `KV_resident` grows with context, so short-ctx ⇒ tiny `t_resident` ⇒ `c_engage`
+  dominates ⇒ `T*≤0` ⇒ quantize (never spill a short-ctx cache); long-ctx ⇒ spill viable.
+  **This is the ctx-value gate (req #2) — it falls out of the formula, no separate threshold.**
+  `W_active`/`KV_resident` are summed from the model's real per-layer KV-head structure
+  (`n_embd_k_gqa`/`n_embd_v_gqa`, global vs local heads) — **req #4 (attention structure)**.
+- **`bw_eff`** = `min(pcie_bw_gbps, host_ram_bw_gbps)` — **req #3: RAM→GPU bandwidth matters,
+  not just PCIe.** On solidPC (PCIe4 ~6.5 + DDR4) PCIe binds; on bs2 (PCIe5 ~50 + DDR5) they
+  are comparable so `min` bites. `pcie_bw_gbps` exists (`common/pcie-profile.cpp`);
+  `host_ram_bw_gbps` is NEW (boot memcpy probe or dmidecode DDR type/speed).
+- **`vram_bw`** = GPU memory bandwidth (**req E**), host-side from `nvidia-smi`/device props.
+- **`c_engage`** = fixed streaming-path engagement cost (the −15% first-spill step, §5c ≈
+  ~1.4 ms). **Boot micro-measured (user chose option B).** It is model- and DCA-dependent, so a
+  static constant is unreliable (§5c 256k shows −15% first-spill; the 12k clean p639 run showed
+  <1% — same host). See "c_engage probe" below.
+- **DCA (req #5)**: DCA-on vs DCA-off have different knees (§5c: DCA-off linear
+  bandwidth-bound; DCA-on flat-then-sharp-cliff, post bug-2144). The probe runs in the actual
+  `cparams.dca_enabled` state, so `c_engage` and the post-knee ceiling are captured for the
+  live DCA setting automatically. A DCA-on validity ceiling clamps T\* to the pre-cliff band
+  (~800 MiB observed on bs2).
+
+### Anchors (sanity)
+- 3090 (§4/#588): `t ≈ 23.6 ms + tail/6.5 GB/s` ⇒ T\*(20%) ≈ a few tens of MiB — essentially
+  "quantize" on the slow link. ✓
+- bs2 256k (§5c): −15% plateau holds to ~680 MiB then cliffs at ~1080 ⇒ T\*(20%) ≈ up to the
+  ~680 MiB plateau edge once `t_resident` is large. ✓
+
+## c_engage boot probe (req B — boot micro-measure)
+
+The tier is chosen in the kv-cache ctor *before* allocation, but timing `c_engage` needs a
+decode-capable path. Two viable mechanisms (pick at implementation):
+1. **Standalone synthetic streaming-FA microbench** — build a tiny ggml graph on the CUDA
+   backend: resident FA over a small K vs windowed FA (window + one-stride tail) over the same
+   shapes; time both; `c_engage = t_windowed − t_resident`. Self-contained, no model decode,
+   runs at ctor time. Preferred.
+2. **Post-context two-phase** — build resident, run a few real decode steps resident vs
+   forced-tiny-tail, measure, then (only if the tier would change) rebuild. Most faithful but
+   double-allocates.
+Cache the measured `c_engage` per `(device, dca_state, head_dim class)` in `.opencoti/` so
+only the first boot pays the probe.
+
+## Requirement #4 — honor an explicit `-ctk` as a block
+
+**Simplification (2026-07-11): no new cparams fields / no cross-struct plumbing.** Auto-tier
+runs in the ctor BEFORE it mutates `type_k`/`type_v`, and nothing else makes them non-f16 at
+that point — so the *incoming* type already encodes explicitness. In
+`opencoti_auto_select_kv_tier` (replacing the blanket `type_k != F16 || type_v != F16 →
+return`):
+- incoming `type_v != F16` → user set `-ctv` → **disable auto-tier entirely** (only rolling-KV
+  window stays auto). User.
+- incoming `type_k != F16` (and `type_v == F16`) → user set `-ctk` → **hold K fixed**, walk
+  only the V axis (V-ladder `{f16, q8_0, q4_0}` under the pinned K).
+- both F16 → full behavior above.
+
+## Bandwidth acquisition — also no plumbing
+
+- `pcie_bw_gbps` is ALREADY a `llama_kv_cache` ctor param (`cparams.pcie_bw_gbps`) — pass it to
+  the helper (one-arg change).
+- `vram_bw_gbps` and `host_ram_bw_gbps` are measured **inside the helper**, host-side
+  (`nvidia-smi --query-gpu=…` bus/clock for VRAM bw; a one-shot `memcpy` bench for host DDR),
+  cached in `.opencoti/` — no new cparams fields, no arg.cpp / llama_context_params threading.
+
+## File surface (all additive, opt-in, byte-identical when env unset)
+
+**Contained to ONE file** (`src/llama-kv-cache.cpp`), plus one ctor-arg addition (pass the
+existing `pcie_bw_gbps` to the helper). No `cparams.h` / `arg.cpp` /
+`llama_context_params` changes — see the two simplifications above. `common/pcie-profile.*` may
+optionally host the host-RAM memcpy probe for reuse, but it can also live in the helper.
+
+- `src/llama-kv-cache.cpp` — `opencoti_auto_select_kv_tier`: explicit-axis handling (T1), a
+  host-side `vram_bw`/`host_ram_bw` probe (T2), the `c_engage` synthetic microbench (T3), the
+  T\* drop-model fit-condition + DCA branch (T4). One extra ctor arg: `pcie_bw_gbps`.
+
+## Increment plan (all in `llama-kv-cache.cpp` unless noted)
+
+1. **T1 ✅** — explicit-axis handling in the helper (req #4): replace the blanket override guard
+   with the V-disables / K-holds / both-walks logic. No plumbing (incoming types encode
+   explicitness). Verified both hosts (bs2 + solidPC).
+2. **T2 ✅** — `host_ram_bw` host-side probe (req #3), static-cached (env
+   `OPENCOTI_KV_HOST_RAM_GBPS` override). `pcie_bw_gbps` passed from the ctor. Logs both +
+   `bw_eff = min(pcie, host_ram)`. **VRAM bw dropped — subsumed by the FA microbench** (which
+   reads KV from VRAM), so no separate device-memcpy probe. bug-2151: dead-store elision fixed.
+3. **T3 ✅** — `c_engage` boot probe (req B) — reused the existing FA compute microbench
+   (`opencoti_fa_compute_probe_ms`, #351) at two cell counts (8192 win, 256 floor) →
+   `fa_ms_cell` + `c_engage`. **HOST-ONLY** (dispatches to the existing DSO — no CUDA rebuild).
+4. **T4 ✅** — T\* drop-model fit-condition in the helper (20% default target, env
+   `OPENCOTI_KV_TSTAR_DROP`; cliff `OPENCOTI_KV_TSTAR_MAX_SPILL_MIB` default 800). Fast-path
+   guard keeps the common "f16 fits" boot byte-identical. `dca_enabled` plumbing deliberately
+   avoided — universal 800 MiB cliff instead (conservative DCA-on pre-cliff band).
+5. **T5 (PENDING)** — RULER-niah quality gate: verify auto picks f16-window+spill at long-ctx
+   where it beats quant (niah + decode-tps ≥ the quant alternative). Needs a bs2 long-ctx cell
+   where T\* is genuinely large enough to KEEP f16 (on the 3090 T\* is too small to flip). Then
+   capture patch 0118 + bookkeeping. Build on both hosts already done.
+
+## Standing gates
+Correctness via RULER niah / logit-equiv, never greedy needle. Byte-identical no-op when
+`OPENCOTI_KV_AUTO_TIER` unset. bs2 build (host-mostly; the synthetic probe may touch CUDA — if
+so, DSO rebuild + dual-restamp + `nm -D`). Commit only when the user asks.