# F5 M6 — PolyKV (shared compressed KV pool) Child plan for the PolyKV milestone. Parent: [advanced_kv.md](advanced_kv.md) §M6. Composed **atop M7 Rolling KV** ([rolling_kv.md](rolling_kv.md)). Status: **M6 SHIPPED (2026-06-06)** — S0 asymmetric q8/q4 (#369, banked correct at 4k; large-ctx GPU-fast asymmetric folds into the S2 kernel), S1 SharedKVPool (#370, gate PASS on Gemma-4 iSWA + 6.9× fan-out memory win), and S2 TurboQuant tier family + InnerQ + Level-A materialize + Level-B fused FA-VEC (#371/#374–396, 4/4 tiers logit-equiv PASS, decode ceiling broken) are all done, composed on the full F5 stack (S3 glue gate), and **captured as patches `0072`/`0073`** (byte-identical proof). The remaining milestone is **M6-S4** speculative MTP draft head (`gemma4_assistant`, #373) — a separate follow-on. ## What PolyKV is (reference) Reference: arXiv 2604.24971, MIT — [github.com/ishan1410/PolyKV](https://github.com/ishan1410/PolyKV). A document is prefilled **once**; its KV is extracted, **asymmetrically compressed**, and held **once** in a host-resident `SharedKVPool`. N concurrent agents share that single compressed blob (no per-agent dense copy of the shared prefix) → **O(1) memory in agent count** for the shared context. Reference numbers: **2.91× compression**, 15 agents 19.8 GB → 0.45 GB, +1.59% perplexity @2k (→ +0.57% @4k). **Asymmetric scheme** (the core idea — keys and values quantized differently because key errors amplify exponentially through softmax while value errors scale linearly): - **K → int8 (q8_0)** — higher precision. - **V → 3-bit TurboQuant MSE** — FWHT (fast Walsh-Hadamard) rotation to spread outlier energy across dims, then Lloyd-Max (MSE-optimal) quantization. The reference is **HuggingFace Transformers / PyTorch**, framework-agnostic w.r.t. serving stacks (no vLLM/SGLang dep). Crucially it does **decompress-then-attend** — each agent decompresses the pool into a dense `DynamicCache` before attention, so its savings are at the **storage** layer, not **in-attention**. No custom attention kernel. ## The opencoti angle — why M7 makes this better than the reference M7 already ships **dequant-on-lift** (Stage 3d, `to_fp16_nc`): the streaming/window forward reads host-resident KV and dequantizes it to a device f16 slot **per tile, inside the attention read path**. That is exactly the "dequant-on-read" the PyTorch reference lacks. So opencoti can run PolyKV's compression **in-attention** — the compressed pool never has to be densified, getting the memory win at the attention layer, not just storage. The composition is already designed (rolling_kv.md): *"M7 streams the compressed tiles, decompressing in-slot before FA; the auto-adapt loop becomes `tile_bytes_max × compression_ratio`."* Building blocks already in place: host-resident KV (M7-A pinned pool), quant-aware dequant-on-lift (S3d), independent K/V cache types (`-ctk`/`-ctv`), session-keyed KV-reuse prefix (M0), the iSWA two-cache split (shared-prefix scaffolding). ## Scope decision (from advanced_kv §M6) **Single-tenant correctness first; multi-tenant memory bench follows when Tier-0 fan-out lands.** PolyKV degenerates cleanly to a 1-tenant pool (= aggressive asymmetric KV-compression of one sequence), so single-tenant is a real, shippable first milestone. The user directive is to **validate shared-KV + rolling-KV functionality + performance together**. ## Staging — fast-first (bootstrap with stock types, then the hard kernel) ### S0 — Bootstrap: asymmetric KV compression via existing ggml types (ZERO new kernel) Use llama.cpp's independent `-ctk q8_0 -ctv q4_0` (both existing KV types, both FA-supported, both dequant-on-lift-able). This **is** the PolyKV K/V asymmetry (K precise, V aggressive) with stock types, routed through M7's existing host-residency + dequant-on-lift. Proves the asymmetric-compressed-KV + M7 composition architecture end-to-end with no new code. - Compression: q8_0 K (0.53×) + q4_0 V (0.28× vs f16) ≈ **~2.4× KV** overall. - **Feasibility: PROVEN 2026-06-04.** `-ctk q8_0 -ctv q4_0` boots, attends, and recalls the needle on Gemma head_dim 512 (q4_0 V dequants to f16 in-path; coherent output). KV 186.9 MiB (q4_0 V) vs 244.4 (q8_0 V) — asymmetric compression confirmed. The head_dim-512 q4_0-V risk is retired. - **Correctness gate (RUNNING 2026-06-05, dir `baselines/20260604-2346`):** RULER vt+niah, asymmetric ours (`KVTYPE_OVERRIDE=q8_0 CTV_OVERRIDE=q4_0 FLASH_ATTN=on BINARIES=ours`) vs cached controls. `FLASH_ATTN=on` is **mandatory** — at head_dim 512 the auto-resolver disables FA for q4_0 V (no native kernel), producing empty cells; forcing it on keeps FA + dequant-on-lift engaged. - **The control matrix isolates V-quant exactly** (q8/q8 → q8/q4 is a V-only change): | task | ctx | vanilla f16 | vanilla q8/q8 | ours q8/q8 | **ours q8/q4 (S0)** | |------|-----|-----|-----|-----|-----| | vt | 4096 | 100 | 100 | 100 | **80** | | vt | 32768 | 100 | 80* | 96 | 0† | | niah_single_1 | 4096 | 100 | 100 | 100 | **100** | | niah_single_1 | 32768 | 100 | 100 | 100 | n/a† | \*vanilla q8/q8 vt@32k=80 is a known vanilla-specific prompt fragility (ours q8/q8 there = 96). †**ours q8/q4 at 32k is a CPU-FA timeout, not a quality result** (re-confirmed 2026-06-18 on the current post-upstream-align binary, sha `5e1b3a53`): q4_0-V has no GPU FA kernel at head_dim 512, so resident-mode prefill runs on CPU-FA (~22 tok/s, ~1440s for 32k) and exceeds the harness's 1200s/sample budget — vt 5/5 ReadTimeout, niah the 2 naturally-completing samples both ReadTimeout @1200s (the other 3 were ConnectionError from the reap, not data). This is the **S0 WALL below**, not a retrieval failure. niah@4k=100 (0/5 empty) is the decisive correctness signal and it holds. - **vt@4096 = 80**: both f16 and q8/q8 controls hold 100, so the 20pp drop isolates to V f16→q4_0 — **one flipped sample at NS=5** ("direction real, magnitude not robust" per the baseline's own NS-sizing note). This is the textbook PolyKV signature: aggressive V-quant nicks hard multi-hop (vt) while single-needle (niah) should stay robust — and the exact motivation for TBQ3_0 (S2, MSE-optimal) over naive q4_0. **niah@4k = 100 (re-confirmed 2026-06-18) is the decisive correctness signal — q4_0-V keeps the single needle, and the upstream-align rename did not regress plain gemma4 decode**; niah@32k could not be measured on naive q4_0-V (CPU-FA timeout — see † and the S0 WALL). - **Perf note (resident regime):** at 4k/32k the KV is FULLY RESIDENT (no host tail, POSITION_WINDOW=0) so M7's dequant-on-lift does **not** engage; q8/q4 pays stock llama.cpp's head_dim-512 dequant tax (MMA kernel is f16-only → full-KV dequant before FA, every batch) → ~3–4× slower prefill than f16. **The performance win lives in the OVERFLOW regime** (256k cell), where M7 streams compressed tiles and amortizes dequant per-tile in-attention. **Attempted 2026-06-18** (`m6-s0-overflow-tps.sh`, current binary `5e1b3a53`): the 256k composition cell is NOT obtainable as milestone evidence on the current binary, for two independent reasons — (1) the harness's raw `/completion` prompt at 256k greedily emits an immediate end-token (RES_q8q8: GPU-fast prefill 222k tok @1122 tok/s, but `eval = 1 token`, needle=NO), so decode-tps is unmeasurable without a generation-capable 256k prompt; and (2) the q8/q4 *overflow* win is gated on the **deferred q4_0-V GPU overflow kernel** (S2 TBQ3_0) — the same head_dim-512 gap as the resident wall. The q8/q8 overflow path itself is already proven GPU-fast at 256k (M7 ship, bug-226: vt=96/niah=100). So the rolling-KV + *asymmetric* milestone evidence **awaits the S2 kernel** (and a generation-capable 256k harness), consistent with the finalized S0 verdict below. (The perf script was hardened this session: `--max-time 900` on the completion curl + the resident-q4_0-V@256k config skipped, since it is the CPU-FA multi-hour path — see [[bug-564]].) - Pass = retrieval holds (≥ vanilla − 2pp on the robust niah signal; vt characterized at NS=5). #### S0 WALL (2026-06-05): stock q4_0-V has NO fast GPU FA path at head_dim 512 The correctness gate surfaced a hard performance wall, not a quality result: - **vt@4096** = 80, **316s/5 = 63s/sample, 0/5 empty** — fast, valid (real q4_0-V quality at 4k). - **vt@32768** = 0 is a **TIMEOUT ARTIFACT, not a collapse**: `elapsed_secs=6001 = exactly 5×1200s` (the ruler_runner `--http-timeout 1200`); server log shows `srv stop: cancel task` + `stop processing: n_tokens=…, truncated=0` for every sample — **cancelled mid-prefill, never decoded** (no error/NaN/abort). The 32k q8/q4 prefill does **not finish in 20 min**. - **GPU utilization ~5% during prefill** (14 GB used = weights resident) ⇒ the quant-V attention runs on **CPU** (no GPU FA kernel for q4_0 V at head_dim 512). ~50× slower than f16. niah@32k (same slow prefill) is equally doomed → killed to stop wasting GPU. - **Consequence:** at RESIDENT (4k/32k/256k all fit compressed), stock FA handles quant-V on CPU → prohibitively slow at large ctx; the perf premise (smaller tail → higher tps) is swamped by the CPU-FA tax, and a 256k RULER/needle cell is infeasible (hours/timeout). - **Salvage hypothesis (to test):** M7's dequant-on-lift (S3d, `to_fp16_nc`) dequants q4_0→f16 **on GPU per-tile**, but ONLY engages when the **window** is active (overflow). Forcing `--kv-residency-mode window` / a tight `--vram-target` may route q4_0-V through the **GPU** dequant-on-lift — potentially *faster* than the resident CPU-FA path, which would make the composition both correct AND the perf win. Diagnostic: 32k forced-window q8/q4, compare prefill wall-time + GPU-util vs resident. If GPU-fast → composition viable; if still CPU/slow → naive q4_0-V is a dead end on Gemma-4 head_dim 512, and S0 needs V=q8_0 (memory ~1.9× not 2.4×) or S2's TBQ3_0 must ship a fused GPU dequant-FA kernel (raising S2 scope/risk). #### S0 SALVAGE RESULT (2026-06-05): forced-window q4_0-V CRASHES → both paths blocked Ran `.opencoti/m6-s0-window-gpupath-probe.sh` (32k, `--kv-residency-mode window --vram-target 1700`, q8/q4). The window **engaged** correctly (POSITION_WINDOW=5, window 20480 / host tail 20480 cells) but then **aborted at `fattn.cu:509: GGML_ABORT("fatal error")`** — the `BEST_FATTN_KERNEL_NONE` branch: `ggml_cuda_get_best_fattn_kernel` returns NONE for q4_0-V at head_dim 512 (MMA is f16-only; VEC needs head_dim ≤256; WMMA excluded at 512; TILE didn't apply to this tile shape). **S3d's dequant-on-lift was validated only with q8_0** (bug-226 RULER vt@262144); q4_0-V tiles reach the kernel selector still-quantized → NONE → abort. So: - **Resident q8/q4:** stock TILE-kernel fallback, ~50× slower than f16, RULER-timeout at ≥32k. - **Window q8/q4 (M7 GPU path):** crashes at `fattn.cu:509` (q4_0-V not handled by the streaming kernel selector / dequant-on-lift). **Verdict:** the "zero-new-kernel" naive-q4_0-V bootstrap is **perf-blocked on Gemma-4 head_dim 512** — correct at 4k (niah=100, vt=80) but cannot be made GPU-fast at scale without kernel work. Note: rolling-KV itself is already proven GPU-fast at 256k with **q8_0** (M7 ship, bug-226: vt=96/niah=100). The blocked piece is specifically the **asymmetric q4_0/3-bit V**. FORK (user decision pending): (A) fix the streaming kernel/dequant-on-lift for q4_0-V (DSO debug + rebuild) → proves asymmetric composition GPU-fast; (B) descope S0 to V=q8_0 (validated, GPU-fast; ~1.9× not 2.4×; loses asymmetry); (C) skip to S2 TBQ3_0 with a proper fused GPU dequant-FA kernel (the real PolyKV V-quant); (D) bank S0 correctness (4k proven) + proceed to **S1 SharedKVPool** (the "shared KV" core, host-side, orthogonal to the V-quant kernel gap; delivers an O(1)-in-agents memory-perf story independent of it). **DECISION (2026-06-05, user):** q4_0-V GPU kernel **DEFERRED** ("we'll see later about the q4_0 kernel") — it folds into S2 TBQ3_0, which needs a custom fused dequant-FA kernel regardless. **Proceeding to S1 SharedKVPool now** (path D). S0 banked as: asymmetric architecture correct at 4k (niah=100, vt=80); large-ctx GPU-fast asymmetric compression awaits the S2 kernel. S1 uses the validated q8_0 KV (no kernel gap) and demonstrates the headline O(1)-in-agents sharing win. ### S1 — SharedKVPool: shared read-only prefix across sequences (the multi-tenant core) ✅ DONE (2026-06-05) N slots/seq_ids attend ONE physical copy of a common document prefix + a per-slot private suffix → O(1) prefix memory in agent count. Builds on M0 (prefix reuse), M7 (host residency), iSWA (two-cache). Single agent degenerates to S0. **Design (minimal-additive, rides native unified-mode sharing).** llama.cpp already lets one KV cell belong to many `seq_id`s (`cells.seq[i]` is a `bitset`). In **unified mode** (`--kv-unified`, `n_stream=1`) all slots live on stream 0, so `llama_memory_seq_cp(pool, slot, 0, P)` takes the same-stream **zero-copy** branch (`cells.seq_add` — just OR the destination bit into the existing prefix cells; no buffer copy, no new allocation). Attention isolation is automatic: `set_input_kq_mask_impl` masks a cell for a seq unless `cells.seq_has(j, seq_id)`, so shared-prefix cells (both bits) are attended by both agents while private-suffix cells (one bit) stay isolated. S1 is therefore a tagged, **default-off** server hook (`shared_pool_slot`/`shared_prefix_n_tokens` request params; gated prefill branch that `seq_cp`'s the pool's `[0,P)` prefix into the sharer's seq and sets `n_past=P` to skip re-processing it) — **no new storage**. Counted-once is *structural* (one boot-fixed KV buffer, prefix cells bit-shared), proven by cell occupancy, not a MiB delta. **RUNTIME PREREQUISITE — `--no-clear-idle` (MANDATORY).** Upstream `clear_idle` defaults **on** (with default `cache_ram_mib=8192`), so `--kv-unified` alone auto-enables the idle-slot prompt-cache offload (PR #16391): when the pool slot goes idle, the next task launch serializes its KV to a host blob **and frees the device cells**, evicting the prefix before any sharer's `seq_cp` runs → silent prefix miss (suffix/isolation still work; only the shared prefix is lost). Booting `--no-clear-idle` keeps the pool slot device-resident. The hook also self-guards: if `seq_pos_max(pool) < P-1` it warns and falls through to a normal full reprocess (correct, just slower) instead of sharing nothing. Root cause: [[bug-367]] — it masqueraded as a cross-seq attention regression; the seq_cp/mask machinery was never at fault. **Gate (`.opencoti/m6-s1-shared-pool-gate.sh`) — PASS on Gemma-4 A4B iSWA, q8/q8, `--parallel 3 --kv-unified --no-clear-idle`:** pool prefills an 874-token doc with needle `QUASAR-4417` on slot 0; agents on slots 1/2 POST `shared_pool_slot:0` + a distinct private officer (Alice/Bob). Results: both agents recall the needle (shared); each recalls only its own officer (isolation, R1 crux); `cache_n=874≈P` & `prompt_n=39≈suffix` per agent; Σ prompt_n `{pool,A,B}` = 953 ≈ `P+sufA+sufB` (950), **not** `3·P` (2622) → prefix enters KV exactly once; pool slot intact after both sharers; `shared_pool_slot:-1` default-off = full reprocess (`cache_n=0`) and still correct. iSWA two-cache `seq_cp` delegation (the R2 risk) confirmed across the 5-layer SWA + 25-layer global caches. Also verified on the non-iSWA path (Qwen2.5-1.5B-instruct). **Fan-out MEMORY win — measured (`.opencoti/m6-s1-fanout-mem.sh`, Qwen, N=8, fixed 8192-cell unified buffer).** Physical KV-cell occupancy (read from the `find_slot ... used =` debug counter) for the SAME 8 agents over an 874-token shared doc: **naive (sharing off) = 7183 cells** (≈ 8×P, one full copy per seq) vs **shared (S1) = 1041 cells** (874 prefix counted ONCE + 167 cells for the 7 private suffixes ≈ 24 cells/agent). That's a **6.9× reduction at N=8**, and projected onto the fixed buffer: the naive path fits **~9 agents** before exhaustion while S1 fits **~306** — the O(1)-prefix headline (prefix amortized once; per-agent cost is just the private suffix). This is the *memory* counterpart to the gate's *compute* counted-once, so S1 is validated on both axes. ### S2 — TurboQuant tier family (TBQ8_0 / TBQ4_0 / TBQ3_0): the MSE-optimal KV-quant ladder (HIGH RISK) TurboQuant is **one** scheme — FWHT (fast Walsh-Hadamard) rotation to spread outlier energy across dims + Lloyd-Max (MSE-optimal) codebook — *parameterized by bit-width*. S2 ships the **full tier ladder**, not a one-off 3-bit kernel: new ggml types **TBQ8_0, TBQ4_0, TBQ3_0** that share the same rotation + codebook machinery, each with its own dequant-on-lift entry. This lets the asymmetric scheme mix tiers freely (e.g. K→TBQ8_0, V→TBQ3_0) and supplies MSE-optimal replacements for the naive q8_0/q4_0 KV used in S0/S1. **TBQ3_0 is the headline** (replaces V=q4_0 → 3-bit at reference quality, +1.59% ppl @2k); **TBQ8_0/TBQ4_0** are the precision/mid rungs *and* the lower-risk bring-up path — validate the shared rotation+codebook at 8-bit (near-lossless, easy cosine gate) first, then descend to 4-bit and 3-bit. Highest-risk milestone (new quant kernels + dequant-on-lift entries + ggml type registration + `GGML_TYPE_COUNT` bump + CUDA dequant). **Adoption source (2026-06-05): the AtomicBot-ai TurboQuant fork, not a from-scratch kernel.** [`AtomicBot-ai/atomic-llama-cpp-turboquant`](https://github.com/AtomicBot-ai/atomic-llama-cpp-turboquant) (MIT, fork of `TheTom/llama-cpp-turboquant`) already ships a working, multi-platform (CUDA + Metal + CPU) TurboQuant family: runtime KV types **`turbo2`/`turbo3`/`turbo4`** → `TURBO2_0`/`TURBO3_0`/ `TURBO4_0` (their wording: **"WHT + PolarQuant"**; `turbo3` = 3-bit ~4.3×, the recommended default, with a Metal **"TurboFlash"** flash-attention decode kernel) selected via `-ctk/-ctv`; plus quantize-time **weight** types `TQ3_1S`/`TQ4_1S` ("WHT-rotated Lloyd-Max", `block_size=32`) chosen as `llama-quantize --type` with masks under `scripts/quantize-masks/`. **S2 becomes a study-and-adapt of that fork** rather than a clean-room build: port its WHT rotation + low-bit codebook + the CUDA/Metal dequant/`TurboFlash` decode path into our vendored `llama.cpp`, mapped onto our dequant-on-lift (S3d) machinery and ggml-type registration. Reconcile naming on the way in (our `TBQ3_0` ≡ their `TURBO3_0`/`turbo3`); decide whether to keep their `turbo*` spelling for CLI compatibility with the fork's `-ctk turbo3` (and the draft `-ctkd/-ctvd turbo3`) so the published Gemma-4 GGUFs run unmodified. **Caveat:** the fork's base differs from our vendored tree + we build under **cosmocc** (no CMake/CUDA-nvcc in the host path; CUDA lives in the DSO) — this is a *port*, not a cherry-pick, and pairs with S4 (both adopted from the same fork, both use `turbo3` KV). - **Gate:** cosine/RULER holds at **each** tier (TBQ8_0 near-lossless, TBQ4_0 ≈ q4_0-or-better, TBQ3_0 at reference); TBQ3_0 V compression → ~2.9× (matches reference 2.91×); the fork's published `-ctk turbo3` Gemma-4 path reproduces "no measurable quality regression". DSO rebuild. #### S2 InnerQ — per-channel equalization, PER-LAYER head_dim ✅ DONE + ACCEPTED (2026-06-05) turbo3 as a **K**-quant (not just V) needs the missing correctness ingredient: **InnerQ** per-channel K-variance equalization *before* the WHT (`x[ch] *= scale[ch]` at K-write; dequant `×scale_inv[ch]` after the inverse WHT). Without it turbo3-K is ~0.983 cosine — too lossy for softmax (K≫V asymmetry). opencoti uses **calibrate-then-activate-from-token-0**; `scale_inv` crosses the host↔DSO boundary as a **file artifact** (the turbo3 forward is GPU-only in the DSO; our active dequant is the CPU `to_float` on the lift path — both load the file at boot). **The mixed-head_dim problem (Gemma-4).** `ne00` of the K set-rows view is `n_embd_gqa = head_dim × n_head_kv`, **not** head_dim — and head_dim is NOT derivable from `ne00` alone. Gemma-4 mixes head_dim **per layer**: SWA layers `head_dim 256` (`n_head_kv 8` → `n_embd_gqa 2048`) and global layers `head_dim 512` (`n_head_kv 2` → `1024`), ~5:1. A single head_dim env mis-folds the minority layer-type and contaminates the shared `scale_inv`. **Fix = per-WIDTH slots** keyed by `n_embd_gqa`: the host supplies `TURBO_INNERQ_HEADDIM_MAP="2048:256,1024:512"`; the DSO carries per-slot `scale/scale_inv/sq_accum[MAX_SLOTS=4][512]` + `slot_hd[]`, the dispatch maps `ne00→slot` and passes it as a **kernel arg** (no per-launch H2D), the kernel folds `ch=(i_grp*128+j) % head_dim[slot]` into that slot's accumulator (no cross-width contamination). Finalize writes a **v2 file** (`TIQ2`: `nslots`, per slot `{n_embd_gqa, head_dim, float[head_dim]}`); v1 (`TIQ1`) still loads as one wildcard slot. The CPU dequant selects its slot by `k == n_embd_gqa`. Empty map ⇒ one wildcard slot (uniform models, e.g. Qwen head_dim 128 → identity fold `ch==j`, byte-compatible). Files: `ggml-cuda/set-rows.cu`, `ggml-cuda/turbo-innerq.cuh`, `ggml-turbo-quant.c`; gate `.opencoti/s2-innerq-gate.sh` (+`HEADDIM_MAP`). **Fused-path invariant (S2-next).** InnerQ's forward-equalize (`×scale` at K-write) is **path-shared**; the un-equalize is the only path-specific half and lives **only** in the lift's `to_float` (`×scale_inv`). The fused kernel keeps K equalized and moves the correction to a **Q pre-scale** (`Q[c]*=scale_inv[c]`, `Q_pre·K_eq = Q·K`), reusing the **same per-width table** — so it must NOT route through `to_float` (no double-correction). Per-layer InnerQ is therefore a prerequisite the fused path consumes wholesale. **Acceptance (format-robust, NEVER greedy).** A greedy short-coherence probe is invalid here — the `...-it` coder model is thinking-tuned (`<|channel>thought`) and degenerates on raw short prompts *even at f16*. The valid gate is **logit-equivalence vs f16** (`.opencoti/s2-logit-equiv-gate.sh` over the #355 harness). **VERDICT PASS** on Gemma-4 A4B, turbo3/turbo3 + per-layer InnerQ vs f16/f16, 12 needle prompts: `real_frac=0.0` (≤0.15), answer-scoped `recall_mismatch=0` (12/12 both recall), all 10 divergences `post_answer`-benign, `mean_tv=0.0102`/`p95=0.0255` (catastrophe 0.30). turbo3-K + per-layer InnerQ is distributionally faithful to f16. Validated non-regressive on Qwen3-4B (uniform head_dim 128, identity fold). (#379) #### S2 tier-family overview — quality × speed × memory, all 7 KV modes ✅ (2026-06-05) Full sweep over `{f16, q8_0, q4_0, turbo8, turbo4, turbo3, turbo2}` on one rig (Gemma-4 A4B Q4_K_M, RTX 3090, 6025-tok prompt, 256-tok forced decode `ignore_eos`, ctx 8192, single stream). Harness: `.opencoti/s2-tier-overview.sh` + `s2-overview-assemble.py`. Quality = logit-equiv vs f16 (#355, n=12 needle); turbo verdicts from the tier-family gate, q8/q4 captured in the same pass. ``` mode | verdict real rec meanTV p95TV | pre tps dec tps util% | KV MiB bpv vsF16 -------------------------------------------------------------------------------------------- f16 | REF 0.000 0 0.0000 0.0000 | 2751 73.2 97 | 460.0 2.000 1.00x q8_0 | PASS 0.000 0 0.0122 0.0785 | 2610 65.7 94 | 244.4 1.062 0.53x q4_0 | PASS 0.000 0 0.0094 0.0007 | 2654 64.9 93 | 129.4 0.562 0.28x turbo8 | PASS 0.000 0 0.0098 0.0157 | 994 2.7 54 | 233.6 1.016 0.51x turbo4 | PASS 0.000 0 0.0152 0.0768 | 1069 3.2 58 | 122.2 0.531 0.27x turbo3 | PASS 0.000 0 0.0075 0.0090 | 1003 3.1 53 | 89.8 0.391 0.20x turbo2 | PASS 0.000 0 0.0228 0.1304 | 1129 3.3 58 | 61.1 0.266 0.13x ``` - **Quality:** all 6 quant modes PASS (real_frac 0, recall_mismatch 0). turbo2's p95 TV 0.13 is the loosest but far under the 0.30 catastrophe line. Quality is **not** the differentiator — the whole ladder is usable. - **Memory (the win):** 460→61 MiB, a 7.5× span. turbo8 (234) and turbo4 (122) **Pareto-beat** q8_0 (244)/q4_0 (129) at matched bit-width — turbo carries one `norm`/128 vals (1.6% overhead) vs stock's scale/32 (6.25%). turbo3 (90) and turbo2 (61) reach footprints stock has no q3/q2 KV type for. - **Speed (the catch — RESOLVED 2026-06-06 by the perf-path below):** in this validation-path snapshot, native quants decoded 65–73 tps / 93–97% util while **turbo decoded 2.7–3.3 tps / 53–58% util (~25× slower).** The fingerprint: prefill dropped ~2.7× but decode collapsed ~25× — the CPU dequant-on-lift reconstructed the WHOLE KV cache on the host every step (prefill amortizes over the query batch, decode pays per single token). Root cause: the `ggml_cast(turboN→f16)` lift emitted a `GGML_OP_CPY` that CUDA's CPY `supports_op` rejected (no turbo src), so the scheduler spilled it to the CPU backend → whole compressed cache D2H + host dequant + H2D every token. **Fixed by the GPU-resident dequant-on-lift (Level A, next subsection).** (#371) #### S2 perf-path Level A — GPU-resident InnerQ-correct turbo dequant-on-lift ✅ DONE + GATED (2026-06-06) > **⚠ SUPERSEDED — historical.** The ~41 tps plateau here is the whole-cache `ggml_cast(turboN→F16)` > materialize path; it is broken open by **Level B** (in-register fused FA-VEC, next subsection) and > again by the **WS1/WS2 in-register turbo-V** work (context.md §5x). The current turbo decode story > lives there — treat these Level-A numbers as a milestone record, not the shipping figures. The "Speed (the catch)" spill above is fixed. The lift stays a graph `ggml_cast(turboN→F16)` but now executes **entirely on CUDA**: a new block-cooperative kernel does the inverse WHT **once** per 128-element group in shared memory (vs the per-output-pair recompute in `turbo-dequant.cuh`), applies InnerQ `×scale_inv[ch]` in true-channel space, and writes f16 in VRAM — stock flash-attention then runs unchanged. No host round-trip. Materialize-f16-then-stock-FA is preserved by design, so the ceiling is the f16 path (Level B = fused FA-VEC reading turbo in-register, a later milestone). **Edit surface (4 hook sites + 3 new files, all `// opencoti-hook: turboquant perf-lift`; 10 markers):** - `ggml/src/ggml-cuda/turbo-cpy.{cu,cuh}` (NEW) — `cpy_turbo{2,3,4,8}_f16` kernels + `ggml_cpy_turbo_f16_cuda` launcher (WHT-once cooperative inverse + InnerQ un-equalize + strided f16 write). - `ggml/src/ggml-cuda/turbo-innerq-dev.cuh` (NEW) — cross-TU bridge declaring `turbo_innerq_active_device_scale(ne00, &d_scale_inv, &head_dim)`. - `ggml-cuda/set-rows.cu` — implements that accessor (owns the InnerQ device symbols; selects slot via the existing `innerq_slot_for((int)ne00)`, returns a `cudaGetSymbolAddress` pointer + `head_dim`). - `ggml-cuda/ggml-cuda.cu` — CPY `supports_op` now accepts `turbo{2,3,4,8}_0 → F16`. - `ggml-cuda/cpy.cu` — `ggml_cuda_cpy` dispatches turbo→F16 to the launcher (InnerQ pointer from the bridge, or `nullptr` ⇒ identity). - `src/llama-graph.cpp` — collapses the old two-step `turbo→F32→F16` cast to a single `turbo→F16` (removes the f32 intermediate whose `turbo→F32` leg was the CPU-spilling op). **Correctness model (R2, resolved empirically):** the GPU lift is a faithful numeric replica of the CPU dequant — same graph tensor, same `ne00`, same `innerq_slot_for`/`head_dim` table, same inverse-WHT order (signs2 → butterfly → 1/√128·signs1) — so it makes the *identical* InnerQ decision as the CPU lift it replaces. The gate passes iff the CPU gate passed (it does). No K/V flag needed. **Build:** `build:llamafile:cuda` (DSO, ~24min; restamp `/root/.llamafile/v/0.10.1/ggml-cuda.so` **and** `/root/ggml-cuda.so` byte-identical, bug-224) + `build:llamafile:make` (host, llama-graph.cpp). Scheduler proof: `GGML_SCHED_DEBUG=1` shows `graph splits = 2` (no CPU-spill fragmentation; the per-node dump is compiled out under `-DNDEBUG`). **Gates (both GREEN, RTX 3090, Gemma-4 A4B Q4_K_M, 6025-tok prompt, 256-tok forced decode `ignore_eos`):** ``` mode | verdict real rec | dec tps was speedup util% was | KV MiB (unchanged) ------------------------------------------------------------------------------------- f16 | REF 0.00 0 | 73.8 73.2 1.0x 95 97 | 460.0 q8_0 | PASS 0.00 0 | 65.9 65.7 1.0x 95 94 | 244.4 q4_0 | PASS 0.00 0 | 65.9 64.9 1.0x 96 93 | 129.4 turbo8 | PASS 0.00 0 | 42.5 2.7 15.7x 96 54 | 233.6 turbo4 | PASS 0.00 0 | 40.6 3.2 12.7x 98 58 | 122.2 turbo3 | PASS 0.00 0 | 41.6 3.1 13.4x 96 53 | 89.8 turbo2 | PASS 0.00 0 | 42.2 3.3 12.8x 95 58 | 61.1 ``` - **Decode:** all turbo tiers **12.7–15.7×** faster (3→~41 tps), every tier clears the ≥10× bar. Prefill also recovered (turbo8 994→2513 tps — the spill taxed prefill too); util 53–58% → 95–98%. - **Quality:** all four tiers stay **logit-equiv PASS** (real_frac 0, recall_mismatch 0) vs the SAME f16 ref — the GPU dequant reproduces the CPU dequant's InnerQ output distributionally (R1 validated). - **Memory + regression:** KV MiB identical to the validation-path table; f16/q8_0/q4_0 byte-unchanged. - **Residual (root-caused 2026-06-06, #389):** turbo plateaus at a uniform ~41 tps **regardless of tier** (turbo8/4/3/2 all ~41 despite 234→61 MiB KV) — the tell-tale of a **tier-independent** bottleneck. It is the Level-A lift itself: `build_attn_mha` casts the **whole** turbo KV view to a full **460 MiB f16 buffer every decode step** (`ggml_cast(turboN→F16)` over all n_kv, redundantly re-dequantizing unchanged positions), then runs stock f16 FA on it. The f16 output is the same size for every tier ⇒ flat ~41 tps; the turbo read (61–234 MiB) is not the bottleneck. This **throws away** turbo's bandwidth advantage — turbo decodes *below* q8/q4 (which the FA kernel reads in-register, no materialize). **The AtomicBot fork proves the real ceiling: its CUDA path is the fused FA-VEC (`fattn-vec.cuh` reads `TURBO{2,3,4}_0` in-register as "unquantized" K/V + sparse-V skip; full template-instance set).** Fork benchmark (M4 Max Metal, fused): turbo3 = **88–92 % of f16** (61.84 vs 70.09 tps MoE; 19.71 vs 21.34 dense), 78.5 tps tg128 on M5 — i.e. ≈ q8/q4-class, **not** halved. Ratio-extrapolated to our rig (f16=74) the fused target is **~63–66 tps turbo3**. Closing the gap = **Level B = port the fork's fused FA-VEC turbo path** (#390), existing vendored code, the next milestone. (A cheaper "incremental materialize into a persistent f16 shadow" would remove the per-step redundancy but costs a full f16 shadow buffer — negates the runtime memory win and is strictly dominated by the fused path; go straight to Level B.) - Patch capture: the 6-file Level-A edit surface is captured in **`0073-turboquant-kv`** at S3 (#372), registered via the patch README row (same snapshot-diff mechanism as `0070`). (S3pA / #383–388) #### S2 perf-path Level B — fused FA-VEC turbo (in-register K/V, no f16 materialize) ✅ DONE + GATED (2026-06-06) Level A's flat ~41 tps ceiling is **broken**. The whole-cache `ggml_cast(turboN→F16)` per decode step is gone for the value tiers: flash-attention now reads turbo2/3/4 K/V **in-register** (centroid×norm, in the WHT-rotated domain), and the expensive Walsh–Hadamard rotation runs only on the n_q-sized **Q** (forward) and the small FA **output** (inverse) — never on the KV cache. Acceptance = logit-equivalence, never greedy byte-equality. **New op `GGML_OP_TURBO_WHT`** (`ggml/include/ggml.h`, `ggml/src/ggml.c`, `ggml-cpu/ggml-cpu.c` abort-stub, `ggml-cuda/turbo-wht.{cu,cuh}` + `turbo-wht-core.cuh`): direction 0 forward-rotates Q (`×scale_inv → signs₁ → WHT → 1/√G·signs₂`); direction 1 inverse-rotates the output. InnerQ folds into **both** WHT kernels via the device symbol `turbo_innerq_active_device_scale(ne[0]=head_dim)` (B-adapted — no graph-tensor scale src, unlike the fork's mctx tensor). **Parseval:** `WHT(Q⊘s)·WHT(K⊙s) = Q·K`, so the InnerQ scale cancels and the turbo K/V reads are InnerQ-**free** (pure centroid×norm). **Edit surface (all `// opencoti-hook: turboquant perf-lift (… Level B)`):** `fattn-turbo.cuh` (NEW — in-register `vec_dot_fattn_vec_KQ_turbo{2,3,4}` + `dequantize_V_turbo{2,3,4}`), `fattn-common.cuh` (`get_vec_dot_KQ`/`get_dequantize_V` turbo cases), `fattn-vec.cuh` (`is_turbo_K` config; turbo rides the **float** Q path), `fattn.cu` (turbo→VEC gating, D∈{128,256}), `ggml-cuda.cu` (TURBO_WHT dispatch + supports_op), `template-instances/fattn-vec-instance-turbo{2,3,4}_0` (+ generator), `src/llama-graph.cpp` (`build_attn_mha` fused gate). turbo8 + head_dim-512 global layers keep the Level-A materialize fallback (no vec instance / no fork reference). **Graph gate (`build_attn_mha`):** fuse iff `flash_attn && turbo2/3/4 && K==V && head_dim∈{128,256}` **and** (L4.5 prefill-hybrid) per-stream `n_q ≤ TBV_FUSE_MAX_NQ` (default 8) — DECODE fuses, large-n_q PREFILL falls to the Level-A materialize→MMA path (turbo prefill on the vec kernel is ~2–3× slower than f16-MMA). **Gate (GREEN, RTX 3090, Gemma-4 A4B Q4_K_M, 6025-tok prompt, 256-tok forced decode):** ``` mode | verdict mean_tv | dec@6k LevelA low-ctx | prefill | KV MiB | path ---------------------------------------------------------------------------------- f16 | REF — | 72.4 73.8 77.6 | 2788 | 460.0 | native q8_0 | PASS 0.0122 | 65.4 65.9 — | 2617 | 244.4 | native q4_0 | PASS 0.0094 | 64.6 65.9 70.2 | 2656 | 129.4 | native turbo8 | PASS 0.0181 | 42.6 42.5 — | 2502 | 233.6 | materialize (fallback) turbo4 | PASS 0.0136 | 48.5 40.6 — | 817* | 122.2 | FUSED turbo3 | PASS 0.0067 | 51.4 41.6 64.1 | 1113* | 89.8 | FUSED turbo2 | PASS 0.0179 | 56.0 42.2 — | 1537* | 61.1 | FUSED ``` `*` always-fuse prefill; the prefill-hybrid (default) restores turbo prefill to ~2474 tps (= materialize) while keeping 51.8 decode (A/B turbo3: T=0 2544/41.4, T=8 2474/51.8, T=huge 1099/52.2; hybrid logit-equiv PASS, mean_tv 0.0063). - **Decode ceiling broken + scales with compression:** turbo2 56 > turbo3 51 > turbo4 48 > turbo8-materialize 43 tps — the more compressed the tier, the less KV bandwidth/step, the faster decode (the in-register payoff). turbo2 = **86 % of q8_0**, 1.31× the Level-A ceiling. GPU util 96–98 % (Level-A ~53 %). - **Quality:** 4/4 logit-equiv PASS (real_frac 0, recall_mismatch 0) vs the same f16 ref; **turbo3 mean_tv 0.0067 is the tightest of any tier** — a 3-bit cache beating 4-bit q4 (0.0094) on quality. - **Decode-tps is ctx-dependent** (turbo3 64.1 low-ctx → 51.4 @6k). Report the ctx. **Tier positioning (decided 2026-06-06, user):** at matched bits, native **q8_0/q4_0 dominate turbo8/turbo4** on speed AND quality at ~equal VRAM — TurboQuant's MSE centroids buy nothing at 4–8 bit (scalar quant is already near-lossless). **Use q8_0/q4_0 for 8/4-bit KV.** TurboQuant earns its place at **3-bit and below**, where native has no KV type: **turbo3** Pareto-beats q4 (90 vs 129 MiB, TV 0.0067 vs 0.0094) at ~80 % of q4 long-ctx decode; **turbo2** is the smallest logit-equiv KV that exists (61 MiB, 56 tps) — the 256k play. turbo8/turbo4 stay as ladder-rung / materialize fallbacks; turbo8 is **not** fused. **Reference (`animehacker/llama-turboquant`, investigated #395):** its headline "tb3 ≈ q4_0" is a **regime artifact** (Qwen-0.8B, tg128, Strix-Halo UMA APU → KV cost negligible); reproduced on our rig at low ctx (turbo3 64.1 vs q4 70.2 = 91 %). Its kernel does the **same float centroid-LUT dot** we do; the gap at long ctx is **intrinsic centroid-decode compute** (turbo3 reads *less* KV than q4 yet decodes slower). No kernel unlocks q4-parity at long ctx. **L4.5 optimization round:** **prefill-hybrid = WINNER, shipped** (default `TBV_FUSE_MAX_NQ=8`). **int8-dp4a** (restructure the KQ dot into integer `__dp4a` + int8 centroid codebook) — correctness **de-risked viable** (`.opencoti/ab-dp4a-unit2.cu`: softmax TV 0.0005), but gather-bound (dp4a speeds the MAC, not the centroid LUT lookups, nor the float V path) and needs re-enabling the q8_1 Q path through the L4-green kernel — deferred to **#397** (flag-gated follow-on, correctness already proven). AB1 (fold-WHT-into-FA) costed at ~1–2 % (60 tiny launches/step) — subsumed by #397's q8_1 path; AB2/AB4/AB5 dropped (centroid-LUT is the right primitive per the reference). ⚠️ **Correction (user, 2026-06-07):** the original AB4 write-off — "turbo8/4 demoted so D=512 turbo-vec moot" — was a **non-sequitur**. Head_dim-512 coverage is orthogonal to *which tiers* survive: turbo8/4 being demoted says nothing about whether the 512-dim global layers need a fused turbo path. They do — for turbo2/turbo3 (see below). AB4 should never have been dropped on that basis. #### S2 perf-path D=512 fused turbo-vec for turbo2/turbo3 ✅ DONE + GATED (2026-06-07, #411) — ships as `0075-d512-turbo-vec` **Done.** The "D=512 turbo-vec moot" verdict above was scoped to turbo8/4 (demoted in favour of native q8/q4). It did **not** hold for the sub-4-bit tiers: **turbo3/turbo2 are the 256k play**, and at head_dim 512 (Gemma-4 global layers, `n_head_kv=2`, ~1 in 6 layers) they fell back to the **Level-A materialize→MMA** path — the fused gate was `head_dim∈{128,256}` only. At long ctx that whole-cache `ggml_cast(turboN→F16)` per step is the decode tax that scales linearly with `n_kv`, biting hardest at 256k (our primary case). **The assumption that a flat D=512 would blow the VEC register budget (the original reason upstream caps `can_use_vector_kernel` at ≤256) was WRONG for the turbo path** — no split / tiling was needed. The VEC kernel is already **D-generic** (`nthreads`/`nthreads_V` derive from D arithmetically; at D=512, ncols=1: `nthreads_V=min(D/4,32)=32`→`V_cols_per_iter=1`, KQ smem `max(ne_KQ, nwarps·1·512)` halfs ≈ 4–8 KB on the Ampere half2 path — well under the 48 KB static-smem cap — and VKQ ≈ 8 half2 ≈ 16 regs). A plain `DECL_FATTN_VEC_CASE(512, turbo2/3)` instance compiles + runs clean. Ships as **`0075-d512-turbo-vec`**: `fattn-vec.cuh` extern decls + the two turbo template instances (+ `generate_cu_files.py` autogen line), `fattn.cu` kernel-selection relax (`Q->ne[0]==512`→VEC) + dispatch, `llama-graph.cpp` `fused_turbo_512` fuse-gate extension. turbo4/turbo8 @ 512 keep Level-A (no D=512 instance). **Gate (`.opencoti/p411-d512-gate.sh` + `p411-perf.sh`):** correctness = **Parseval-EXACT** (the WHT is block-diagonal over 128-blocks → D=512 = 4 independent Parseval-exact 128-blocks): fused-vs-materialize logit-equivalence **`real_frac=0.0` for BOTH turbo3 and turbo2** (bit-equivalent, not merely close). Perf: turbo3 @ ctx 27.7k, **fused 41.0 vs materialize 29.7 tok/s = +38%**, scaling with `n_kv` (larger at 256k). No new ABI / flag (auto-engages via the existing `TBV_FUSE_MAX_NQ`=8 decode gate). This unblocks the long RULER re-run. - Patch capture: the Level-B edit surface joins **`0073-turboquant-kv`** at S3 (#372) — captured + byte-identical-proven (vendored source stays uncommitted per policy; DSO sha `19168f6a`, both `/root` paths restamped byte-identical). ### S4 — Speculative MTP draft head (`gemma4_assistant`) for Gemma-4 A4B — adopt AtomicBot fork (HIGH RISK, NEW 2026-06-05) Pairs with S2 (same fork, same `turbo3` KV). Gives **ours** llamafile a working **Multi-Token Prediction** speculative-decode path for Gemma-4 A4B: a ~0.4B **assistant draft head** predicts `B-1` future tokens from the target's last hidden state, the 27B-MoE target verifies them in parallel → **same output distribution** (lossless), ~85-88% accept / claimed +30-50% short-prompt throughput on dense targets. The user frames the head as **"EAGLE3 weight layout with low/mid/high feature taps"**; the fork ships it as the custom **`gemma4_assistant`** architecture (EAGLE-family; `eagle3` is a sibling `--spec-type`). **Draft artifact (published, ready to consume):** [`AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF`](https://huggingface.co/AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF) — F16 (816 MB) / Q8_0 (440) / Q5_K_M / **Q4_K_M (310, recommended draft)** / Q4_K_S. Target = any Gemma-4 26B-A4B-it GGUF (our `gemma-4-A4B-98e-v6-coder-it-Q4_K_M` or unsloth's). These GGUFs use the `gemma4_assistant` arch and **will NOT load in stock llama.cpp** — that's exactly the gap S4 closes. **Fork usage we must reproduce (CUDA/Linux):** ``` llama-server -m TARGET.gguf --mtp-head ASSISTANT.gguf --spec-type mtp \ --draft-block-size 3 --draft-max 8 --draft-min 0 -ngl 99 -ngld 99 \ -ctk turbo3 -ctv turbo3 -ctkd turbo3 -ctvd turbo3 -fa on -c 16384 ``` **What S4 must adopt/port into our vendored cosmocc-built `llama.cpp` (study-and-adapt, not cherry-pick — fork base ≠ our base, and CUDA lives in our DSO):** 1. **`gemma4_assistant` architecture** — arch enum + hparams + `models/gemma4-assistant.cpp` graph builder (the EAGLE3 low/mid/high feature-tap head + centroid LM head for E2B/E4B; dense tied head for 26B-A4B). Loader for the assistant GGUF's tensors. 2. **In-context MTP scheduler** — the fork loads the draft **into the target context** (no second `llama_context`, **no second KV cache**), on a dedicated scheduler; the head consumes the target's last hidden state, drafts a block, the target batch-verifies. This composes with M7 residency and S1 unified KV — design the seam carefully (one cache, one slot, draft tokens appended/verified). 3. **Runtime flags** (`common/arg.cpp`): `--mtp-head` (+ `-md` alias), `--spec-type mtp`, `--draft-block-size`, `--draft-max`, `--draft-min`, `-ngld`, and draft KV `-ctkd/-ctvd` (depends on S2 for `turbo3`; until S2 lands, draft KV can ride q8_0/f16). 4. **TS adapter / inference-engine surface** — expose the draft-head path through the opencoti tiered-inference config (later; not S4 core). - **Gate:** with the assistant draft loaded, decode is **distribution-identical** to no-draft greedy (teacher-forced KL ≈ 0 / top-1 agreement — reuse the #355 logit-equivalence harness), measured accept-rate in the published 70-88% band, and net **throughput up** vs no-draft on Gemma-4 A4B at RTX-3090. Draft-off byte-identical. NEVER greedy-needle alone. - **Risks:** (R-S4a) fork base divergence — the arch/scheduler may touch files heavily rewritten by our 0030-0042/0070 patches; budget a real port. (R-S4b) "no second KV cache" claim must be reconciled with our M7/S1 single-unified-cache model — verify the draft's appended tokens don't corrupt the shared/rolling cache. (R-S4c) cosmocc has no nvcc in the host path; any new CUDA kernel (TurboFlash for the draft) goes in the DSO build, not `:make`. ### S3 — Compose-with-glue + bench + patches + docs ✅ DONE + SHIPPED (2026-06-06, #372) The ship: prove PolyKV composes on the M5 advanced-KV stack, capture the vendored delta into reproducible patches, wire the user-facing TS knob, and land bookkeeping. **S4 (MTP draft) is split out as its own future milestone — its `0062` patch is NOT part of this ship.** **Adapter knob (P1).** The PolyKV user-facing KV-type selector is `-ctk`/`-ctv`, surfaced on the `opencoti-llamafile` adapter as `config.ctypeK`/`ctypeV` (env `OPENCOTI_LLAMAFILE_CTYPE_K/_V`, validated against `KNOWN_KV_TYPES` incl. `turbo{2,3,4,8}`), emitted in `buildServerArgs` after `--kv-residency-mode` / before `--neo-pipeline`. Default-unset ⇒ no flag emitted (byte-identical to prior arg vector). S1's pool selection is a **per-request** JSON param (`shared_pool_slot` / `shared_prefix_n_tokens`), server-side, not a CLI flag. 98 adapter tests PASS + typecheck clean. **Compose bench (P1, `perf/llamafile/advanced-kv-stack.bench.ts`).** Added config **A4** = baseline + `-ctk q8_0 -ctv q4_0` (S0 native asymmetric, greedy-needle-safe) and **S′** = the full F5 stack + asymmetric KV. Live (Qwen 0.5B) result **11/12 hard PASS**: A4 boots + recovers needle, A4 CUDA KV 4.89 vs baseline 12.00 MiB (0.41× ≤ 0.6× gate), S′ `cached_n>0` ∧ needle_recovered, S′ per-cell mem < S per-cell (soft W3 PASS). The sole hard FAIL (C9, S 51.92 vs B 336.05 tps) is the **pre-existing M2 head-split perf floor**, not PolyKV — A4-alone decodes 128 tps, proving the asymmetric KV path itself is fast; C3/C7 SKIPPED (workload forced no shift this run). **Live turbo3 compose gate (P1b, `.opencoti/m6-s3-glue-gate.sh`) — the PolyKV ↔ HeadInfer-moot evidence.** turbo3 **and** turbo2 K/V under the live F5 stack flags (`--kv-unified --cache-reuse 256 --rest-kv-eviction --rest-kv-recent 256 --rest-kv-layer -1`, `--parallel 2`, Gemma-4 A4B, InnerQ-calibrated) vs the **same-stack f16 reference** (isolates the KV-type effect, not the stack effect). Both PASS: turbo3 `real_frac=0.0` mean_tv 0.0164, turbo2 `real_frac=0.0` mean_tv 0.01784, `recall_mismatch=0`. **PolyKV ↔ HeadInfer conflict is MOOT:** M7 retired the head-split default (`kvResidencyMode auto` = *window*, not the M2 head axis), so TurboQuant's Level-B fused FA-VEC never collides with the M2 head-axis split — they cannot both be active. The gate is the proof: turbo fused decode composes cleanly with window residency + multi-slot reuse + retention eviction. **Patches (P2/P3) — captured + byte-identical-proven.** The vendored PolyKV delta over the `0001-0042 + 0070 + 0071` baseline (snapshot-diff of a faithful `reset → bootstrap → apply` baseline vs the live tree) splits into two patches: | Patch | Stage | Touches | Acceptance | |-------|-------|---------|------------| | `0072-poly-kv-pool` | S1 SharedKVPool (#370) | 3 server files (`tools/server/server-{task.h,task.cpp,context.cpp}`) | default-off byte-identical; gate `m6-s1-shared-pool-gate.sh` | | `0073-turboquant-kv` | S2 TurboQuant (tiers+InnerQ+Level-A+Level-B, #371/#374–396) | 32 files (13 NEW); `GGML_OP_TURBO_WHT` op; carries the outer `llamafile/build-functions.sh` host-source add | logit-equiv vs f16 (#355); gates `s2-tier-family-gate.sh` + `m6-s3-glue-gate.sh` | Both numbered **`0072`/`0073` (NOT the plan's stale `0060`/`0061`)** because the build pipeline applies patches in **pure lexical sort** order (`build-pipeline.ts`, `.sort()`), and the TurboQuant FA-VEC edits sit **on top of** `0070`'s `fattn.cu`/`llama-graph.cpp`/`ggml-cuda.cu` rewrite — `0060/0061` would sort *before* 0070 and break the chain (M7 shipped before M6, so the original labels predate it). S0 asymmetric q8/q4 needs **no patch** (rides M7 dequant-on-lift natively). **Proof:** both reverse-apply clean against live (`git apply --check -R`), and a forward-apply onto a fresh baseline copy **reproduces the live tree byte-for-byte** (`diff -r` = 0 differing / 0 only-in-live / 0 only-in-fwd, outer `build-functions.sh` identical). The full `0001-0042+0070+0071+0072+0073` chain is strict-apply proven by composition (#366 + the baseline `:apply` + the `0072/0073` `--check`). Vendored source stays uncommitted per policy — only the `.patch` files commit. (DSO sha `19168f6a`; both `/root` paths restamped byte-identical.) #### S2 perf-path — turbo-MMA verify kernel + scalar-K/turbo-V fused decode ✅ SHIPPED OPT-IN default-OFF (2026-07-09, #628/#630, WS1, `0105-0107`) WorkStream-1 = the **asymmetric strong scalar-K (q8_0/q6_0/q5_0/q4_0) + cheap turbo/TCQ-V** decode family — K read as scalar (true-space), V read in-register as rotated turbo/TCQ (no f16 materialize); the graph `build_attn_mha` `wht_q`/`wht_o` split applies the OUTPUT inverse-WHT only (K is true-space, V is FWHT-rotated). Two decode regimes: - **Fused-VEC decode (n_q=1)** — the foundation, unconditional. 16 new FA-VEC instances (`fattn-vec-instance-q{4,5,6,8}_0-turbo{2,3}_{0,tcq}.cu`) + `FATTN_VEC_CASES_ALL_D` dispatch + K≠V selector admission + `build-functions.sh` 3e default-build inclusion. Invariant K bit-width > V; reverse (turbo-K/scalar-V) not built. Ships in `0107`. - **turbo-MMA verify (n_q>1)** — a turbo-aware tensor-core kernel that dequants q8_0-K + turbo2-V into the MMA smem tile (rotated V; graph `wht_o` keeps it exact) for MTP verify batches at head_dim **128** (Qwen full-attn), **256** (Gemma SWA-local), **512** (Gemma global). Three per-head-dim knobs `WS1_TURBO_MMA_D128/D256/D512`, **all default-OFF** — off ⇒ verify keeps the fused-VEC routing (the pre-#628 default + logit-equiv reference). `WS1_DEROTATE` (inverse-FWHT V in-load ⇒ graph skips `wht_o`) is a separate default-OFF experiment. bug-2130: `!is_quant_kv` for `need_f16_K/V` so `launch_fattn` reads raw quant blocks (f16 keeps true,true = byte-identical). Ships in `0107`. **VERDICT (#630 — 3-model crossover map on bs2, A4B / omni27 / qw35a3, decode-tps + teacher-forced TV + niah, ctx 32k/128k/256k):** turbo-MMA is **mid-ctx-specialized and earns NO default-on anywhere.** | model (attn) | 32k | 128k | 256k | fidelity | |---|---|---|---|---| | **A4B** Gemma iSWA, D256-local + D512-global | +4.2% (241.4 vs 231.6) | **+6.4%** (249.1 vs 234.1) | **−14.3%** (190.3 vs 222.0) | niah 100 | | **omni27** Qwen3.6-27B dense, D128 full | −0.1% (99.1 vs 99.2) | +0.3% (92.2 vs 91.9) | +0.1% (87.2 vs 87.1) | niah 100, real_frac 0 | | **qw35a3** Qwen3.6-35B-A3B MoE, D128 full | +1.2% (204.8 vs 202.4) | +0.5% (187.7 vs 186.7) | +0.4% (168.0 vs 167.3) | niah 100, real_frac 0 | **Named mechanism (A4B, nsys `--cuda-graph-trace=node`):** *windowed-D256 cast-saving swamped by the ctx-growing D512 global-cast tail.* turbo-MMA eliminates the `cpy_turbo2_f16` materialize cast on the D256 **SWA-windowed** V-cache, paying instead ~2× slower D256 attention (in-kernel re-dequant) — both ctx-INDEPENDENT constants (windowed). The saving it must beat is the **D512 GLOBAL** materialize (`cpy_turbo2_f16`, still paid — D512 knob off), whose per-cast cost doubles with ctx (123µs@128k → 242µs@256k). So the eliminated (windowed) cast stays fixed while the total step balloons with the D512 tail: net-win @128k (cast eliminated ≈ 2744ms), net-loss @256k (only ≈ 211ms left to eliminate). And it is **un-gatable** — a per-op `k->ne[1]` context-gate sees the SWA WINDOW length, not session ctx (bug-2133). On **Qwen D128 full-attention** the per-op gate WOULD work (uniform n_kv), but there is nothing to gate for: turbo-MMA is bit-exact yet tps-flat because the dense/MoE **FFN dominates the verify step** — attention acceleration has no leverage. **Ship:** OPT-IN, **default-OFF** (`0107`); the D128/D256/D512 instances are additive/correct/harmless. WS1's shipped decode wins are `0106` (bug-2126 boot correctness) + **`0105` bug-2127 DCA per-tensor defer = the ~2× decode win** — independent of turbo-MMA. Chain 0001..0107 strict-applies from clean 0.10.3 and reproduces the FINAL tree byte-identically (verified 2026-07-09). > **This WS1 null does NOT transfer to DCA-on — see WS2 (#629).** The turbo-MMA "no default-on > anywhere" verdict is a **noDCA, full-attention verify-batch** result: the D512-global > `cpy_turbo2_f16` cast it had to beat was *still paid* (D512 knob off), and on Qwen the FFN > swamps the attention win. **WS2 (#629) is the opposite verdict on the DCA-on decode path:** reading > the rotated turbo/TCQ-V **in-register inside the multichunk DCA fused flash-attn kernel** (n_q≤16 > gate) *eliminates* that context-growing whole-cache cast, and is a **validated decode WIN of > +68–95% @256k** across A4B / 31B (D512) and omni27 / qw35a3 (D256), every turbo/TCQ tier, > retrieval-lossless (niah 100) — now **DEFAULT-ON**. Full matrix + caveats: > [context.md §5x](../evaluations/context.md). WS1 turbo-MMA (verify-batch, noDCA) and WS2 > in-register turbo-V (decode, DCA-on-fused) are distinct kernels with distinct verdicts. ### S4 — Speculative MTP draft head ✅ DONE + SHIPPED (2026-06-07, #373/#404–410, `0074`) The ship: the Gemma-4 A4B `gemma4_assistant` Multi-Token-Prediction draft head, loaded **into** the target (no second `llama_context` / KV cache), proven a **decode-throughput win** AND logit-equivalent, captured as `0074-speculative-mtp.patch`, surfaced on the adapter, bookkept. **The binding requirement was throughput, and it is MET.** MTP-on must BEAT MTP-off in decode tok/s — not just be correct. Final gate on the capture-candidate binary (`.opencoti/p4-fused-gate.sh`, Gemma-4 A4B v6-coder Q4_K_M @ RTX 3090, 200 tok × 2 prompts, temp 0): | mode | decode tok/s | vs off | draft accept | |------|-------------|--------|--------------| | off (plain greedy) | 78.34 | — | — | | seq (per-step MTP) | 113.49 | **1.45×** | 0.769 / 0.606 | | fused (single-graph MTP) | 114.03 | **1.46×** | 0.769 / 0.606 | `seq == fused` byte-identical (the permitted refactor-equivalence use of byte-equality — two implementations of the same MTP computation). `off != seq` text divergence is **NOT** a regression: it is the documented benign **batch-vs-sequential near-tie** numerical artifact (bug-270) — plain greedy is bit-deterministic (verified off-vs-off A==B, 761 B identical), so the divergence is the deterministic float difference between the spec-decode's batched target-verify and sequential decode flipping an argmax within tolerance. The directive-mandated correctness metric is **logit-equivalence**, not greedy byte-equality: `.opencoti/s4-p3-logit-equiv.sh` (8 needle prompts, top-20 teacher-forced) gives `real_frac=0.0`, `frac_full_agree=1.0`, 100% draft acceptance (the `VERDICT:FAIL` string is the known `mean_tv` artifact for spec-accepted empty-`top:[]` tokens — judge on `real_frac`, per [[project_mtp_s4_p3]]). **Root cause of the early "MTP is ~19% SLOWER" result (bug-419) was weight placement, not cuda-graph.** Stage-0 measurement (phase-split timers: submit=22 ms, sync=0.01 ms; `GGML_SCHED_DEBUG=2`) showed the assistant's **dense LM head ran as a CPU island** mid-graph, stalling the GPU ~95% idle. The `gemma4_assistant` `tok_embd` (~277 MiB) is the weight-tied dense output head, but llama.cpp classifies `tok_embd` as an INPUT tensor and `dev_input` is hard-pinned to the CPU buft — so the per-draft-step full-vocab (262144) output mul_mat executed on CPU. **The fix is one loader edit:** route the `gemma4_assistant` `tok_embd` through the OUTPUT buft (`dev_output`, GPU when the output layer is offloaded). cuda-graph replay was MOOT (sync already free); the originally-planned custom fused MTP-head kernel was unnecessary. See [[project_dso_cache_poisoning]] for the parallel DSO gotcha class. **Adapter knob.** `config.mtpHead` / `specType` / `draftNGpuLayers` (env `OPENCOTI_LLAMAFILE_MTP_HEAD` / `_SPEC_TYPE` / `_DRAFT_NGL`), emitted in `buildServerArgs` as `--mtp-head ` / `--spec-type mtp` / `-ngld ` after `--fused-moe-up-gate`, before `extraArgs`. `-ngld` is REQUIRED for the win (a CPU-resident draft head re-seeds the stall). Default-unset ⇒ no draft (plain decode, byte-identical arg vector). 31 adapter tests PASS + typecheck clean. **Patch (`0074-speculative-mtp`) — captured + byte-identical-proven both directions.** Snapshot-diff of the live tree vs a faithful `base-mtp` (= `base-poly` + `apply(0072,0073)` = `0001-0042+0070+0071+0072+ 0073`, no MTP): 24 files, 1 NEW (`src/models/gemma4-assistant.cpp`), 0 outer-vendor delta. Forward-apply onto a fresh base-mtp reproduces the live tree byte-for-byte (0/24 differ); reverse-apply onto live reproduces base-mtp (0/23 differ + new file deleted). Three surgical upstream edits (marker `opencoti F5 M6-S4 mtp`) + additive surface — registered in `docs/protocols/UPSTREAM_SYNC.md`. The reverted sparse-argmax centroid fast path (dead for this dense `use_ordered_embeddings=false` GGUF) is banked at `.opencoti/banked/mtp-sparse-argmax.md`, not shipped (verification discipline: never ship an unexercised path). Vendored source stays uncommitted per policy — only the `.patch` commits. **P5 (#409) — POSITION_WINDOW MTP perf path: characterized, NOT implemented (route inapplicable).** The task hypothesized routing the MTP cross-read through `build_attn_mha_position_window` if the `get_k` streamback under window residency is a bottleneck. Measurement (forced `--vram-target 600`, n_kv~2919) shows Gemma-4 A4B's iSWA cache is **POSITION_WINDOW-ineligible** (`swa_type=1`; eligibility requires `swa_type==NONE`) — overflow relief is **GPU_STREAM**, so the proposed window route never engages. Under GPU_STREAM overflow MTP is **throughput-negative** (MTP-on 30.3 vs off 46.3 tok/s; correct — needle recovered, accept 0.786). Refused to ship a dead position-window MTP route (same discipline as the reverted sparse-argmax). DISPOSITION: the supported long-context MTP path is **turbo-KV-resident** — S2 Level-B + the D=512 fused turbo instances (#411) keep the 256k cache in VRAM → `GPU_RESIDENT` → MTP keeps its 1.46× win; f16-overflow-streaming is a config to avoid (f16 KV doesn't fit 256k anyway). See [[bug-419]], bug-426. ### Phase-0 fork recon (2026-06-05) — findings that shape S2 + S4 Shallow-cloned the fork (`/srv/dev-disk-by-label-opt/dev/_recon/atomic-llama-cpp-turboquant`) and diffed its TurboQuant + MTP surfaces against OURS. **Headline: the fork's base is contemporaneous with ours, not years apart** — both end the ggml type table at `MXFP4=39`, `GGML_TYPE_COUNT=42`; both carry the same recent model set (`gemma4-iswa`, `minimax-m2`, `qwen35*`); OURS already has `LLM_ARCH_GEMMA4` and `COMMON_SPECULATIVE_TYPE_EAGLE3` + a full ngram/draft speculative family. So both milestones are *additive onto a near-identical base*, far cheaper than a cross-era port. **S2 (TurboQuant) edit surface:** - ggml types append cleanly at 42-46 (`TURBO2_0/3_0/4_0`, `TQ3_1S/TQ4_1S`), `COUNT 42→47` — fork `ggml.h:432`, traits `ggml.c:737`. Block structs in `ggml-common.h` (TURBO3_0 = 14B/128-blk: fp16 norm + 2-bit qs[32] + 1-bit signs[16]). - CPU quant/dequant is ONE new file `ggml/src/ggml-turbo-quant.c` (~1026 lines: seeded WHT/Hadamard rotation `TURBO_SEED_ROTATION=42` + PolarQuant/Lloyd-Max codebook) + dispatcher cases in `ggml.c`. **Self-contained, low-conflict.** - CUDA (DSO): `turbo-wht.cu`, `turbo-quant.cuh`, `mmvq-tq.cu`, `vecdotq.cuh`, + **12 `template- instances/fattn-vec-instance-turbo*.cu`**. The fattn hook is the **only CRITICAL conflict** (OURS 0070 heavily rewrote `fattn.cu`). - CLI: `kv_cache_types[]` in `common/arg.cpp` maps `"turbo3"` → enum (keep the spelling for fork-GGUF compat). - **Bring-up order:** Gate-1 = TURBO3_0 **CPU-only** (enum + struct + `ggml-turbo-quant.c` + traits + dispatcher + CLI, NO CUDA, NO fattn) → validates WHT/codebook + type registration with zero fattn.cu risk. Gate-2 = CUDA dequant w/ dequant-then-compute fallback (reuse our S3d lift). Gate-3 = fused TurboFlash templates (the 0070 rebase). - **Gate-2 design (2026-06-05, decided).** The fork's `convert.cu` turbo3→f16 emits **rotated** f16 (it relies on the fused kernel to rotate q on-the-fly). Our S3d "dequant-on-lift → standard f16 FA" path does NOT rotate q, so the correct fallback un-rotates K/V instead: **dequant turbo3 → rotated f16 → `ggml_cuda_turbo_wht` (direction=1, inverse) → true-space f16 K/V → unmodified standard f16 FA.** Minimal additive port set: `turbo-quant.cuh` + `turbo-wht.cu` /`.cuh` (verbatim) → `dequantize.cuh` turbo3 accessor → `convert.cu` to_fp16 cases → graph insertion of the inverse-WHT pass on the lifted K/V → `supports_op`/type registration in `ggml-cuda.cu` → CUDA DSO rebuild → GPU cosine/needle gate vs vanilla f16. - **InnerQ stub-disabled for Gate-2 (decided, user).** `turbo-quant.cuh` couples to InnerQ device globals (`d_innerq_scale*`, per-channel Q/V equalization) defined in `turbo-innerq.cu`. For the Gate-2 fallback we compile with **identity InnerQ** (scales = 1, `d_innerq_active = 0`) — either by porting `turbo-innerq.cu` in its dormant/identity state or by stubbing the globals — so K/V un-rotation is the plain inverse WHT. **InnerQ is re-enabled in Gate-3** (the fused TurboFlash path), where the q-rotation + channel-scale fusion actually pays off. Identity InnerQ is numerically exact for the un-rotation (it only rescales channels pre-quant; with scale=1 the round-trip is the bare WHT codebook path Gate-1 already validated at cosine 0.983). - **Gate-2 is two halves (both mechanism-matched to our tree):** - **READ (dequant → true f16) — source DONE 2026-06-05.** New `ggml-cuda/turbo-dequant.cuh` (InnerQ-free; folds the inverse WHT into the per-block dequant since turbo3 block=128=WHT group) + `convert.cu` `to_fp16_cuda` & `to_fp16_nc_cuda` TURBO3_0 cases (use `dequantize_turbo3_0_inv`) + `QR_TURBO3` define in ggml-common.h. Routes through our **type-agnostic `GGML_OP_STREAMING_FLASH_ATTN`** (supports_op gates head_dim only; the op dequants K/V to f16 on the S3d lift at fattn.cu:748) → **no standard-FA supports_op change, no graph node added.** The gate must force the streaming path (residency-mode / low `--vram-target`) so turbo3 never hits the stock `FLASH_ATTN_EXT` (which lacks a turbo3 vec kernel until Gate-3). - **WRITE (f32 → turbo3) — NEXT.** Our KV store uses `ggml_set_rows` (cpy_k/cpy_v → `ggml_set_rows`, llama-kv-cache.cpp:3091/3188/3204) — the SAME mechanism as the fork's `set-rows.cu`. Port `k_set_rows_turbo3` + `k_set_rows_turbo3_tail` + `set_rows_cuda_turbo3` (~300 lines: group forward-WHT via warp shuffle, `__ballot_sync` sign pack, corrected-norm) into our `set-rows.cu` (which today has 0 turbo refs), + `GGML_OP_SET_ROWS` supports_op / dispatch for a turbo3 dst. Then ONE CUDA DSO rebuild + GPU cosine/needle gate vs vanilla f16. - **Gate-1 — DONE (2026-06-05).** Host build clean (`ggml-turbo-quant.c.o` linked; two splice bugs fixed: bug-333 truncated TQ4_1S `type_traits` entry, bug-334 decls trapped inside `#ifdef __cplusplus`). Registration proven by the boot line `llama_kv_cache: … K (turbo3): 1.37 MiB, V (turbo3): 1.37 MiB` + `attn_rot_k = 1` on Qwen-1.5B CPU (`-ngl 0 -ctk turbo3 -ctv turbo3`) — i.e. `-ctk turbo3` accepted, type name resolves, cache allocates at the compressed block size (~5× vs f16's ~7 MiB). CPU then **segfaults at warmup** — EXPECTED: `dequantize_row_turbo3_0` is a rotated-space stub (no inverse WHT; the GPU FA kernel rotates q so `(Rq)·(Rk)=q·k`), so turbo3 *attention compute* is Gate-2/3 (CUDA) scope, not a Gate-1 fail. Gate-1c math validated out-of-band: vendored kernel primitives (`turbo_cpu_fwht`/`_inverse`/`nearest_centroid_3bit`/`CENTROIDS_3BIT`) are byte-identical (modulo `GGML_API` macro + a comment) to `_recon/turbo3_selftest` (cosine 0.983 round-trip). Gate script `.opencoti/s2-gate1.sh`. **Next: Gate-2 (CUDA dequant-then-FA via S3d lift) — requires DSO rebuild.** **S4 (MTP draft head) edit surface:** - `gemma4_assistant` arch = new file `src/models/gemma4-assistant.cpp` (clean) + enum/hparams in `llama-arch.{h,cpp}` (`n_embd_backbone`, `n_centroids`, `centroid_top_k`, `use_ordered_embeddings`, `requires_target_arch`). EAGLE3 feature taps = target token-emb + prev backbone hidden → `mtp_pre_projection` → transformer → backbone hidden → (centroid head for E2B/E4B | dense head for A4B). Greedy argmax on device. - **"No second KV cache" CONFIRMED in code:** the assistant is loaded INTO the target via `llama_model_load_mtp_from_file()` (tensors prefixed `mtp.`, stored as `target->mtp_assistant`); there is no second `llama_context`. MTP cross-attn reads the target's existing KV at the **last full/SWA target layer** (`build_attn_mtp()` → `gemma4_mtp_kv_layer_last_in_range`). Scheduler = `common_speculative_state_mtp` (`common/speculative.cpp`) + `llama_decode_mtp_async/wait` (`llama-context.cpp`), async depth-2 with lazy-wait overlap; MTP only READS target KV, never writes. - Flags additive in `common/arg.cpp`: `--mtp-head`/`-md`, `--spec-type mtp`, `--draft-block-size` (default 3), `--n-draft-max/min`, `-ngld`, `-ctkd/-ctvd`. - **#1 integration risk (R-S4b, confirmed):** `build_attn_mtp()` must read target KV through OUR M7 rolling-KV (POSITION_WINDOW) + S1 accessor dispatch, not the fork's direct `cache_k/cache_v`; and S1/M7 must not mutate the target prefix mid-MTP-snapshot. Mitigation: route MTP's attn through the same accessor as the regular forward; keep the slot in decode-state (no compress) while an MTP draft is pending. Fallback: dequant-on-lift the window into a temp buffer for the MTP read. - **Bring-up order:** P1 arch+loader (smoke-load the assistant GGUF, no decode) → P2 flags/params → P3 scheduler skeleton (async no-ops) → P4 KV-accessor merge (single-token MTP forward, cosine vs naive) → P5 async + S1 race-guard (turn-2 latency + logit-equivalence) → P6 docs/patch 0062. ## Critical files (anticipated) - `src/llama-kv-cache.{cpp,h}` — pool/shared-prefix storage, type-asymmetric K/V, compression-ratio into the M7 tile-sizing. - `ggml/src/ggml-cuda/fattn.cu` + S3d dequant-on-lift — read compressed tiles (S0 reuses; S2 adds the TBQ8_0 / TBQ4_0 / TBQ3_0 dequant kernels). - `ggml` core — `TBQ8_0` / `TBQ4_0` / `TBQ3_0` type registration (S2). - `common/arg.cpp` — pool/agent + (later) `--poly-kv` knobs; `packages/opencoti-llamafile` adapter. - Patches `0072-poly-kv-pool` (S1) / `0073-turboquant-kv` (S2); gates `.opencoti/m6-s*-*.sh`; `perf/llamafile/advanced-kv-stack.bench.ts`. ## Verification 1. S0: single-tenant cosine/RULER == vanilla at asymmetric q8_0/q4_0; ~2.4× KV; M7 overflow path OK. 2. S1: 2-agent shared-prefix recall; O(1) shared-prefix memory. 3. S2: cosine/RULER holds per TurboQuant tier (TBQ8_0 / TBQ4_0 / TBQ3_0); TBQ3_0 ~2.9× compression. 4. S3: glue bench (A4/S′ + turbo3/turbo2 logit-equiv compose); patches `0072`/`0073` strict-apply in series after `0071`, reverse-apply clean + forward-reproduce byte-identical; hooks registered; reap/DSO hygiene. ✅ ## Standing ops `OPENCOTI_NO_CCACHE=1`; `:cuda` rebuilds DSO (~24min) only when S2 lands its kernels — S0/S1 are host-side; vendor-backup before any destructive vendor op; reap `llamafile.*--server` (never embedder 38092 / SSH); DSO restamp before each ours boot; GPU ≥10 GiB; commit only when asked; soft-fork additive + `opencoti-hook:` markers registered; RULER acceptance via cosine/semantic/RULER, never greedy byte-equality.