Upload folder using huggingface_hub

9cee049 verified 8 days ago

67 kB

F5 M6 — PolyKV (shared compressed KV pool)

Child plan for the PolyKV milestone. Parent: advanced_kv.md §M6. Composed atop M7 Rolling KV (rolling_kv.md). Status: M6 SHIPPED (2026-06-06) — S0 asymmetric q8/q4 (#369, banked correct at 4k; large-ctx GPU-fast asymmetric folds into the S2 kernel), S1 SharedKVPool (#370, gate PASS on Gemma-4 iSWA + 6.9× fan-out memory win), and S2 TurboQuant tier family + InnerQ + Level-A materialize + Level-B fused FA-VEC (#371/#374–396, 4/4 tiers logit-equiv PASS, decode ceiling broken) are all done, composed on the full F5 stack (S3 glue gate), and captured as patches 0072/0073 (byte-identical proof). The remaining milestone is M6-S4 speculative MTP draft head (gemma4_assistant, #373) — a separate follow-on.

What PolyKV is (reference)

Reference: arXiv 2604.24971, MIT — github.com/ishan1410/PolyKV. A document is prefilled once; its KV is extracted, asymmetrically compressed, and held once in a host-resident SharedKVPool. N concurrent agents share that single compressed blob (no per-agent dense copy of the shared prefix) → O(1) memory in agent count for the shared context. Reference numbers: 2.91× compression, 15 agents 19.8 GB → 0.45 GB, +1.59% perplexity @2k (→ +0.57% @4k).

Asymmetric scheme (the core idea — keys and values quantized differently because key errors amplify exponentially through softmax while value errors scale linearly):

K → int8 (q8_0) — higher precision.
V → 3-bit TurboQuant MSE — FWHT (fast Walsh-Hadamard) rotation to spread outlier energy across dims, then Lloyd-Max (MSE-optimal) quantization.

The reference is HuggingFace Transformers / PyTorch, framework-agnostic w.r.t. serving stacks (no vLLM/SGLang dep). Crucially it does decompress-then-attend — each agent decompresses the pool into a dense DynamicCache before attention, so its savings are at the storage layer, not in-attention. No custom attention kernel.

The opencoti angle — why M7 makes this better than the reference

M7 already ships dequant-on-lift (Stage 3d, to_fp16_nc): the streaming/window forward reads host-resident KV and dequantizes it to a device f16 slot per tile, inside the attention read path. That is exactly the "dequant-on-read" the PyTorch reference lacks. So opencoti can run PolyKV's compression in-attention — the compressed pool never has to be densified, getting the memory win at the attention layer, not just storage. The composition is already designed (rolling_kv.md): "M7 streams the compressed tiles, decompressing in-slot before FA; the auto-adapt loop becomes tile_bytes_max × compression_ratio."

Building blocks already in place: host-resident KV (M7-A pinned pool), quant-aware dequant-on-lift (S3d), independent K/V cache types (-ctk/-ctv), session-keyed KV-reuse prefix (M0), the iSWA two-cache split (shared-prefix scaffolding).

Scope decision (from advanced_kv §M6)

Single-tenant correctness first; multi-tenant memory bench follows when Tier-0 fan-out lands. PolyKV degenerates cleanly to a 1-tenant pool (= aggressive asymmetric KV-compression of one sequence), so single-tenant is a real, shippable first milestone. The user directive is to validate shared-KV + rolling-KV functionality + performance together.

Staging — fast-first (bootstrap with stock types, then the hard kernel)

S0 — Bootstrap: asymmetric KV compression via existing ggml types (ZERO new kernel)

Use llama.cpp's independent -ctk q8_0 -ctv q4_0 (both existing KV types, both FA-supported, both dequant-on-lift-able). This is the PolyKV K/V asymmetry (K precise, V aggressive) with stock types, routed through M7's existing host-residency + dequant-on-lift. Proves the asymmetric-compressed-KV + M7 composition architecture end-to-end with no new code.

Compression: q8_0 K (0.53×) + q4_0 V (0.28× vs f16) ≈ ~2.4× KV overall.
Feasibility: PROVEN 2026-06-04. -ctk q8_0 -ctv q4_0 boots, attends, and recalls the needle on Gemma head_dim 512 (q4_0 V dequants to f16 in-path; coherent output). KV 186.9 MiB (q4_0 V) vs 244.4 (q8_0 V) — asymmetric compression confirmed. The head_dim-512 q4_0-V risk is retired.

Correctness gate (RUNNING 2026-06-05, dir baselines/20260604-2346): RULER vt+niah, asymmetric ours (KVTYPE_OVERRIDE=q8_0 CTV_OVERRIDE=q4_0 FLASH_ATTN=on BINARIES=ours) vs cached controls. FLASH_ATTN=on is mandatory — at head_dim 512 the auto-resolver disables FA for q4_0 V (no native kernel), producing empty cells; forcing it on keeps FA + dequant-on-lift engaged.

The control matrix isolates V-quant exactly (q8/q8 → q8/q4 is a V-only change):

task	ctx	vanilla f16	vanilla q8/q8	ours q8/q8	ours q8/q4 (S0)
vt	4096	100	100	100	80
vt	32768	100	80*	96	0†
niah_single_1	4096	100	100	100	100
niah_single_1	32768	100	100	100	n/a†

*vanilla q8/q8 vt@32k=80 is a known vanilla-specific prompt fragility (ours q8/q8 there = 96). †ours q8/q4 at 32k is a CPU-FA timeout, not a quality result (re-confirmed 2026-06-18 on the current post-upstream-align binary, sha 5e1b3a53): q4_0-V has no GPU FA kernel at head_dim 512, so resident-mode prefill runs on CPU-FA (~22 tok/s, ~1440s for 32k) and exceeds the harness's 1200s/sample budget — vt 5/5 ReadTimeout, niah the 2 naturally-completing samples both ReadTimeout @1200s (the other 3 were ConnectionError from the reap, not data). This is the S0 WALL below, not a retrieval failure. niah@4k=100 (0/5 empty) is the decisive correctness signal and it holds.

vt@4096 = 80: both f16 and q8/q8 controls hold 100, so the 20pp drop isolates to V f16→q4_0 — one flipped sample at NS=5 ("direction real, magnitude not robust" per the baseline's own NS-sizing note). This is the textbook PolyKV signature: aggressive V-quant nicks hard multi-hop (vt) while single-needle (niah) should stay robust — and the exact motivation for TBQ3_0 (S2, MSE-optimal) over naive q4_0. niah@4k = 100 (re-confirmed 2026-06-18) is the decisive correctness signal — q4_0-V keeps the single needle, and the upstream-align rename did not regress plain gemma4 decode; niah@32k could not be measured on naive q4_0-V (CPU-FA timeout — see † and the S0 WALL).
Perf note (resident regime): at 4k/32k the KV is FULLY RESIDENT (no host tail, POSITION_WINDOW=0) so M7's dequant-on-lift does not engage; q8/q4 pays stock llama.cpp's head_dim-512 dequant tax (MMA kernel is f16-only → full-KV dequant before FA, every batch) → ~3–4× slower prefill than f16. The performance win lives in the OVERFLOW regime (256k cell), where M7 streams compressed tiles and amortizes dequant per-tile in-attention. Attempted 2026-06-18 (m6-s0-overflow-tps.sh, current binary 5e1b3a53): the 256k composition cell is NOT obtainable as milestone evidence on the current binary, for two independent reasons — (1) the harness's raw /completion prompt at 256k greedily emits an immediate end-token (RES_q8q8: GPU-fast prefill 222k tok @1122 tok/s, but eval = 1 token, needle=NO), so decode-tps is unmeasurable without a generation-capable 256k prompt; and (2) the q8/q4 overflow win is gated on the deferred q4_0-V GPU overflow kernel (S2 TBQ3_0) — the same head_dim-512 gap as the resident wall. The q8/q8 overflow path itself is already proven GPU-fast at 256k (M7 ship, bug-226: vt=96/niah=100). So the rolling-KV + asymmetric milestone evidence awaits the S2 kernel (and a generation-capable 256k harness), consistent with the finalized S0 verdict below. (The perf script was hardened this session: --max-time 900 on the completion curl + the resident-q4_0-V@256k config skipped, since it is the CPU-FA multi-hour path — see [[bug-564]].)
Pass = retrieval holds (≥ vanilla − 2pp on the robust niah signal; vt characterized at NS=5).

S0 WALL (2026-06-05): stock q4_0-V has NO fast GPU FA path at head_dim 512

The correctness gate surfaced a hard performance wall, not a quality result:

vt@4096 = 80, 316s/5 = 63s/sample, 0/5 empty — fast, valid (real q4_0-V quality at 4k).
vt@32768 = 0 is a TIMEOUT ARTIFACT, not a collapse: elapsed_secs=6001 = exactly 5×1200s (the ruler_runner --http-timeout 1200); server log shows srv stop: cancel task + stop processing: n_tokens=…, truncated=0 for every sample — cancelled mid-prefill, never decoded (no error/NaN/abort). The 32k q8/q4 prefill does not finish in 20 min.
GPU utilization ~5% during prefill (14 GB used = weights resident) ⇒ the quant-V attention runs on CPU (no GPU FA kernel for q4_0 V at head_dim 512). ~50× slower than f16. niah@32k (same slow prefill) is equally doomed → killed to stop wasting GPU.
Consequence: at RESIDENT (4k/32k/256k all fit compressed), stock FA handles quant-V on CPU → prohibitively slow at large ctx; the perf premise (smaller tail → higher tps) is swamped by the CPU-FA tax, and a 256k RULER/needle cell is infeasible (hours/timeout).
Salvage hypothesis (to test): M7's dequant-on-lift (S3d, to_fp16_nc) dequants q4_0→f16 on GPU per-tile, but ONLY engages when the window is active (overflow). Forcing --kv-residency-mode window / a tight --vram-target may route q4_0-V through the GPU dequant-on-lift — potentially faster than the resident CPU-FA path, which would make the composition both correct AND the perf win. Diagnostic: 32k forced-window q8/q4, compare prefill wall-time + GPU-util vs resident. If GPU-fast → composition viable; if still CPU/slow → naive q4_0-V is a dead end on Gemma-4 head_dim 512, and S0 needs V=q8_0 (memory ~1.9× not 2.4×) or S2's TBQ3_0 must ship a fused GPU dequant-FA kernel (raising S2 scope/risk).

S0 SALVAGE RESULT (2026-06-05): forced-window q4_0-V CRASHES → both paths blocked

Ran .opencoti/m6-s0-window-gpupath-probe.sh (32k, --kv-residency-mode window --vram-target 1700, q8/q4). The window engaged correctly (POSITION_WINDOW=5, window 20480 / host tail 20480 cells) but then aborted at fattn.cu:509: GGML_ABORT("fatal error") — the BEST_FATTN_KERNEL_NONE branch: ggml_cuda_get_best_fattn_kernel returns NONE for q4_0-V at head_dim 512 (MMA is f16-only; VEC needs head_dim ≤256; WMMA excluded at 512; TILE didn't apply to this tile shape). S3d's dequant-on-lift was validated only with q8_0 (bug-226 RULER vt@262144); q4_0-V tiles reach the kernel selector still-quantized → NONE → abort. So:

Resident q8/q4: stock TILE-kernel fallback, ~50× slower than f16, RULER-timeout at ≥32k.
Window q8/q4 (M7 GPU path): crashes at fattn.cu:509 (q4_0-V not handled by the streaming kernel selector / dequant-on-lift). Verdict: the "zero-new-kernel" naive-q4_0-V bootstrap is perf-blocked on Gemma-4 head_dim 512 — correct at 4k (niah=100, vt=80) but cannot be made GPU-fast at scale without kernel work. Note: rolling-KV itself is already proven GPU-fast at 256k with q8_0 (M7 ship, bug-226: vt=96/niah=100). The blocked piece is specifically the asymmetric q4_0/3-bit V. FORK (user decision pending): (A) fix the streaming kernel/dequant-on-lift for q4_0-V (DSO debug + rebuild) → proves asymmetric composition GPU-fast; (B) descope S0 to V=q8_0 (validated, GPU-fast; ~1.9× not 2.4×; loses asymmetry); (C) skip to S2 TBQ3_0 with a proper fused GPU dequant-FA kernel (the real PolyKV V-quant); (D) bank S0 correctness (4k proven) + proceed to S1 SharedKVPool (the "shared KV" core, host-side, orthogonal to the V-quant kernel gap; delivers an O(1)-in-agents memory-perf story independent of it).

DECISION (2026-06-05, user): q4_0-V GPU kernel DEFERRED ("we'll see later about the q4_0 kernel") — it folds into S2 TBQ3_0, which needs a custom fused dequant-FA kernel regardless. Proceeding to S1 SharedKVPool now (path D). S0 banked as: asymmetric architecture correct at 4k (niah=100, vt=80); large-ctx GPU-fast asymmetric compression awaits the S2 kernel. S1 uses the validated q8_0 KV (no kernel gap) and demonstrates the headline O(1)-in-agents sharing win.

S1 — SharedKVPool: shared read-only prefix across sequences (the multi-tenant core) ✅ DONE (2026-06-05)

N slots/seq_ids attend ONE physical copy of a common document prefix + a per-slot private suffix → O(1) prefix memory in agent count. Builds on M0 (prefix reuse), M7 (host residency), iSWA (two-cache). Single agent degenerates to S0.

Design (minimal-additive, rides native unified-mode sharing). llama.cpp already lets one KV cell belong to many seq_ids (cells.seq[i] is a bitset<LLAMA_MAX_SEQ>). In unified mode (--kv-unified, n_stream=1) all slots live on stream 0, so llama_memory_seq_cp(pool, slot, 0, P) takes the same-stream zero-copy branch (cells.seq_add — just OR the destination bit into the existing prefix cells; no buffer copy, no new allocation). Attention isolation is automatic: set_input_kq_mask_impl masks a cell for a seq unless cells.seq_has(j, seq_id), so shared-prefix cells (both bits) are attended by both agents while private-suffix cells (one bit) stay isolated. S1 is therefore a tagged, default-off server hook (shared_pool_slot/shared_prefix_n_tokens request params; gated prefill branch that seq_cp's the pool's [0,P) prefix into the sharer's seq and sets n_past=P to skip re-processing it) — no new storage. Counted-once is structural (one boot-fixed KV buffer, prefix cells bit-shared), proven by cell occupancy, not a MiB delta.

RUNTIME PREREQUISITE — --no-clear-idle (MANDATORY). Upstream clear_idle defaults on (with default cache_ram_mib=8192), so --kv-unified alone auto-enables the idle-slot prompt-cache offload (PR #16391): when the pool slot goes idle, the next task launch serializes its KV to a host blob and frees the device cells, evicting the prefix before any sharer's seq_cp runs → silent prefix miss (suffix/isolation still work; only the shared prefix is lost). Booting --no-clear-idle keeps the pool slot device-resident. The hook also self-guards: if seq_pos_max(pool) < P-1 it warns and falls through to a normal full reprocess (correct, just slower) instead of sharing nothing. Root cause: [[bug-367]] — it masqueraded as a cross-seq attention regression; the seq_cp/mask machinery was never at fault.

Gate (.opencoti/m6-s1-shared-pool-gate.sh) — PASS on Gemma-4 A4B iSWA, q8/q8, --parallel 3 --kv-unified --no-clear-idle: pool prefills an 874-token doc with needle QUASAR-4417 on slot 0; agents on slots 1/2 POST shared_pool_slot:0 + a distinct private officer (Alice/Bob). Results: both agents recall the needle (shared); each recalls only its own officer (isolation, R1 crux); cache_n=874≈P & prompt_n=39≈suffix per agent; Σ prompt_n {pool,A,B} = 953 ≈ P+sufA+sufB (950), not 3·P (2622) → prefix enters KV exactly once; pool slot intact after both sharers; shared_pool_slot:-1 default-off = full reprocess (cache_n=0) and still correct. iSWA two-cache seq_cp delegation (the R2 risk) confirmed across the 5-layer SWA + 25-layer global caches. Also verified on the non-iSWA path (Qwen2.5-1.5B-instruct).

Fan-out MEMORY win — measured (.opencoti/m6-s1-fanout-mem.sh, Qwen, N=8, fixed 8192-cell unified buffer). Physical KV-cell occupancy (read from the find_slot ... used = debug counter) for the SAME 8 agents over an 874-token shared doc: naive (sharing off) = 7183 cells (≈ 8×P, one full copy per seq) vs shared (S1) = 1041 cells (874 prefix counted ONCE + 167 cells for the 7 private suffixes ≈ 24 cells/agent). That's a 6.9× reduction at N=8, and projected onto the fixed buffer: the naive path fits ~9 agents before exhaustion while S1 fits ~306 — the O(1)-prefix headline (prefix amortized once; per-agent cost is just the private suffix). This is the memory counterpart to the gate's compute counted-once, so S1 is validated on both axes.

S2 — TurboQuant tier family (TBQ8_0 / TBQ4_0 / TBQ3_0): the MSE-optimal KV-quant ladder (HIGH RISK)

TurboQuant is one scheme — FWHT (fast Walsh-Hadamard) rotation to spread outlier energy across dims + Lloyd-Max (MSE-optimal) codebook — parameterized by bit-width. S2 ships the full tier ladder, not a one-off 3-bit kernel: new ggml types TBQ8_0, TBQ4_0, TBQ3_0 that share the same rotation + codebook machinery, each with its own dequant-on-lift entry. This lets the asymmetric scheme mix tiers freely (e.g. K→TBQ8_0, V→TBQ3_0) and supplies MSE-optimal replacements for the naive q8_0/q4_0 KV used in S0/S1. TBQ3_0 is the headline (replaces V=q4_0 → 3-bit at reference quality, +1.59% ppl @2k); TBQ8_0/TBQ4_0 are the precision/mid rungs and the lower-risk bring-up path — validate the shared rotation+codebook at 8-bit (near-lossless, easy cosine gate) first, then descend to 4-bit and 3-bit. Highest-risk milestone (new quant kernels + dequant-on-lift entries + ggml type registration + GGML_TYPE_COUNT bump + CUDA dequant).

Adoption source (2026-06-05): the AtomicBot-ai TurboQuant fork, not a from-scratch kernel. AtomicBot-ai/atomic-llama-cpp-turboquant (MIT, fork of TheTom/llama-cpp-turboquant) already ships a working, multi-platform (CUDA + Metal + CPU) TurboQuant family: runtime KV types turbo2/turbo3/turbo4 → TURBO2_0/TURBO3_0/ TURBO4_0 (their wording: "WHT + PolarQuant"; turbo3 = 3-bit ~4.3×, the recommended default, with a Metal "TurboFlash" flash-attention decode kernel) selected via -ctk/-ctv; plus quantize-time weight types TQ3_1S/TQ4_1S ("WHT-rotated Lloyd-Max", block_size=32) chosen as llama-quantize --type with masks under scripts/quantize-masks/. S2 becomes a study-and-adapt of that fork rather than a clean-room build: port its WHT rotation + low-bit codebook + the CUDA/Metal dequant/TurboFlash decode path into our vendored llama.cpp, mapped onto our dequant-on-lift (S3d) machinery and ggml-type registration. Reconcile naming on the way in (our TBQ3_0 ≡ their TURBO3_0/turbo3); decide whether to keep their turbo* spelling for CLI compatibility with the fork's -ctk turbo3 (and the draft -ctkd/-ctvd turbo3) so the published Gemma-4 GGUFs run unmodified. Caveat: the fork's base differs from our vendored tree + we build under cosmocc (no CMake/CUDA-nvcc in the host path; CUDA lives in the DSO) — this is a port, not a cherry-pick, and pairs with S4 (both adopted from the same fork, both use turbo3 KV).

Gate: cosine/RULER holds at each tier (TBQ8_0 near-lossless, TBQ4_0 ≈ q4_0-or-better, TBQ3_0 at reference); TBQ3_0 V compression → ~2.9× (matches reference 2.91×); the fork's published -ctk turbo3 Gemma-4 path reproduces "no measurable quality regression". DSO rebuild.

S2 InnerQ — per-channel equalization, PER-LAYER head_dim ✅ DONE + ACCEPTED (2026-06-05)

turbo3 as a K-quant (not just V) needs the missing correctness ingredient: InnerQ per-channel K-variance equalization before the WHT (x[ch] *= scale[ch] at K-write; dequant ×scale_inv[ch] after the inverse WHT). Without it turbo3-K is ~0.983 cosine — too lossy for softmax (K≫V asymmetry). opencoti uses calibrate-then-activate-from-token-0; scale_inv crosses the host↔DSO boundary as a file artifact (the turbo3 forward is GPU-only in the DSO; our active dequant is the CPU to_float on the lift path — both load the file at boot).

The mixed-head_dim problem (Gemma-4). ne00 of the K set-rows view is n_embd_gqa = head_dim × n_head_kv, not head_dim — and head_dim is NOT derivable from ne00 alone. Gemma-4 mixes head_dim per layer: SWA layers head_dim 256 (n_head_kv 8 → n_embd_gqa 2048) and global layers head_dim 512 (n_head_kv 2 → 1024), ~5:1. A single head_dim env mis-folds the minority layer-type and contaminates the shared scale_inv. Fix = per-WIDTH slots keyed by n_embd_gqa: the host supplies TURBO_INNERQ_HEADDIM_MAP="2048:256,1024:512"; the DSO carries per-slot scale/scale_inv/sq_accum[MAX_SLOTS=4][512] + slot_hd[], the dispatch maps ne00→slot and passes it as a kernel arg (no per-launch H2D), the kernel folds ch=(i_grp*128+j) % head_dim[slot] into that slot's accumulator (no cross-width contamination). Finalize writes a v2 file (TIQ2: nslots, per slot {n_embd_gqa, head_dim, float[head_dim]}); v1 (TIQ1) still loads as one wildcard slot. The CPU dequant selects its slot by k == n_embd_gqa. Empty map ⇒ one wildcard slot (uniform models, e.g. Qwen head_dim 128 → identity fold ch==j, byte-compatible). Files: ggml-cuda/set-rows.cu, ggml-cuda/turbo-innerq.cuh, ggml-turbo-quant.c; gate .opencoti/s2-innerq-gate.sh (+HEADDIM_MAP).

Fused-path invariant (S2-next). InnerQ's forward-equalize (×scale at K-write) is path-shared; the un-equalize is the only path-specific half and lives only in the lift's to_float (×scale_inv). The fused kernel keeps K equalized and moves the correction to a Q pre-scale (Q[c]*=scale_inv[c], Q_pre·K_eq = Q·K), reusing the same per-width table — so it must NOT route through to_float (no double-correction). Per-layer InnerQ is therefore a prerequisite the fused path consumes wholesale.

Acceptance (format-robust, NEVER greedy). A greedy short-coherence probe is invalid here — the ...-it coder model is thinking-tuned (<|channel>thought) and degenerates on raw short prompts even at f16. The valid gate is logit-equivalence vs f16 (.opencoti/s2-logit-equiv-gate.sh over the #355 harness). VERDICT PASS on Gemma-4 A4B, turbo3/turbo3 + per-layer InnerQ vs f16/f16, 12 needle prompts: real_frac=0.0 (≤0.15), answer-scoped recall_mismatch=0 (12/12 both recall), all 10 divergences post_answer-benign, mean_tv=0.0102/p95=0.0255 (catastrophe 0.30). turbo3-K + per-layer InnerQ is distributionally faithful to f16. Validated non-regressive on Qwen3-4B (uniform head_dim 128, identity fold). (#379)

S2 tier-family overview — quality × speed × memory, all 7 KV modes ✅ (2026-06-05)

Full sweep over {f16, q8_0, q4_0, turbo8, turbo4, turbo3, turbo2} on one rig (Gemma-4 A4B Q4_K_M, RTX 3090, 6025-tok prompt, 256-tok forced decode ignore_eos, ctx 8192, single stream). Harness: .opencoti/s2-tier-overview.sh + s2-overview-assemble.py. Quality = logit-equiv vs f16 (#355, n=12 needle); turbo verdicts from the tier-family gate, q8/q4 captured in the same pass.

mode    | verdict  real rec  meanTV  p95TV |  pre tps  dec tps  util% |  KV MiB    bpv   vsF16
--------------------------------------------------------------------------------------------
f16     | REF     0.000   0  0.0000 0.0000 |   2751     73.2     97  |   460.0  2.000  1.00x
q8_0    | PASS    0.000   0  0.0122 0.0785 |   2610     65.7     94  |   244.4  1.062  0.53x
q4_0    | PASS    0.000   0  0.0094 0.0007 |   2654     64.9     93  |   129.4  0.562  0.28x
turbo8  | PASS    0.000   0  0.0098 0.0157 |    994      2.7     54  |   233.6  1.016  0.51x
turbo4  | PASS    0.000   0  0.0152 0.0768 |   1069      3.2     58  |   122.2  0.531  0.27x
turbo3  | PASS    0.000   0  0.0075 0.0090 |   1003      3.1     53  |    89.8  0.391  0.20x
turbo2  | PASS    0.000   0  0.0228 0.1304 |   1129      3.3     58  |    61.1  0.266  0.13x

Quality: all 6 quant modes PASS (real_frac 0, recall_mismatch 0). turbo2's p95 TV 0.13 is the loosest but far under the 0.30 catastrophe line. Quality is not the differentiator — the whole ladder is usable.
Memory (the win): 460→61 MiB, a 7.5× span. turbo8 (234) and turbo4 (122) Pareto-beat q8_0 (244)/q4_0 (129) at matched bit-width — turbo carries one norm/128 vals (1.6% overhead) vs stock's scale/32 (6.25%). turbo3 (90) and turbo2 (61) reach footprints stock has no q3/q2 KV type for.
Speed (the catch — RESOLVED 2026-06-06 by the perf-path below): in this validation-path snapshot, native quants decoded 65–73 tps / 93–97% util while turbo decoded 2.7–3.3 tps / 53–58% util (~25× slower). The fingerprint: prefill dropped ~2.7× but decode collapsed ~25× — the CPU dequant-on-lift reconstructed the WHOLE KV cache on the host every step (prefill amortizes over the query batch, decode pays per single token). Root cause: the ggml_cast(turboN→f16) lift emitted a GGML_OP_CPY that CUDA's CPY supports_op rejected (no turbo src), so the scheduler spilled it to the CPU backend → whole compressed cache D2H + host dequant + H2D every token. Fixed by the GPU-resident dequant-on-lift (Level A, next subsection). (#371)

S2 perf-path Level A — GPU-resident InnerQ-correct turbo dequant-on-lift ✅ DONE + GATED (2026-06-06)

⚠ SUPERSEDED — historical. The ~41 tps plateau here is the whole-cache ggml_cast(turboN→F16) materialize path; it is broken open by Level B (in-register fused FA-VEC, next subsection) and again by the WS1/WS2 in-register turbo-V work (context.md §5x). The current turbo decode story lives there — treat these Level-A numbers as a milestone record, not the shipping figures.

The "Speed (the catch)" spill above is fixed. The lift stays a graph ggml_cast(turboN→F16) but now executes entirely on CUDA: a new block-cooperative kernel does the inverse WHT once per 128-element group in shared memory (vs the per-output-pair recompute in turbo-dequant.cuh), applies InnerQ ×scale_inv[ch] in true-channel space, and writes f16 in VRAM — stock flash-attention then runs unchanged. No host round-trip. Materialize-f16-then-stock-FA is preserved by design, so the ceiling is the f16 path (Level B = fused FA-VEC reading turbo in-register, a later milestone).

Edit surface (4 hook sites + 3 new files, all // opencoti-hook: turboquant perf-lift; 10 markers):

ggml/src/ggml-cuda/turbo-cpy.{cu,cuh} (NEW) — cpy_turbo{2,3,4,8}_f16 kernels + ggml_cpy_turbo_f16_cuda launcher (WHT-once cooperative inverse + InnerQ un-equalize + strided f16 write).
ggml/src/ggml-cuda/turbo-innerq-dev.cuh (NEW) — cross-TU bridge declaring turbo_innerq_active_device_scale(ne00, &d_scale_inv, &head_dim).
ggml-cuda/set-rows.cu — implements that accessor (owns the InnerQ device symbols; selects slot via the existing innerq_slot_for((int)ne00), returns a cudaGetSymbolAddress pointer + head_dim).
ggml-cuda/ggml-cuda.cu — CPY supports_op now accepts turbo{2,3,4,8}_0 → F16.
ggml-cuda/cpy.cu — ggml_cuda_cpy dispatches turbo→F16 to the launcher (InnerQ pointer from the bridge, or nullptr ⇒ identity).
src/llama-graph.cpp — collapses the old two-step turbo→F32→F16 cast to a single turbo→F16 (removes the f32 intermediate whose turbo→F32 leg was the CPU-spilling op).

Correctness model (R2, resolved empirically): the GPU lift is a faithful numeric replica of the CPU dequant — same graph tensor, same ne00, same innerq_slot_for/head_dim table, same inverse-WHT order (signs2 → butterfly → 1/√128·signs1) — so it makes the identical InnerQ decision as the CPU lift it replaces. The gate passes iff the CPU gate passed (it does). No K/V flag needed.

Build: build:llamafile:cuda (DSO, ~24min; restamp /root/.llamafile/v/0.10.1/ggml-cuda.so and /root/ggml-cuda.so byte-identical, bug-224) + build:llamafile:make (host, llama-graph.cpp). Scheduler proof: GGML_SCHED_DEBUG=1 shows graph splits = 2 (no CPU-spill fragmentation; the per-node dump is compiled out under -DNDEBUG).

Gates (both GREEN, RTX 3090, Gemma-4 A4B Q4_K_M, 6025-tok prompt, 256-tok forced decode ignore_eos):

mode    | verdict real rec | dec tps  was   speedup  util%  was |  KV MiB  (unchanged)
-------------------------------------------------------------------------------------
f16     | REF    0.00   0  |  73.8    73.2    1.0x     95    97  |  460.0
q8_0    | PASS   0.00   0  |  65.9    65.7    1.0x     95    94  |  244.4
q4_0    | PASS   0.00   0  |  65.9    64.9    1.0x     96    93  |  129.4
turbo8  | PASS   0.00   0  |  42.5     2.7   15.7x     96    54  |  233.6
turbo4  | PASS   0.00   0  |  40.6     3.2   12.7x     98    58  |  122.2
turbo3  | PASS   0.00   0  |  41.6     3.1   13.4x     96    53  |   89.8
turbo2  | PASS   0.00   0  |  42.2     3.3   12.8x     95    58  |   61.1

Decode: all turbo tiers 12.7–15.7× faster (3→~41 tps), every tier clears the ≥10× bar. Prefill also recovered (turbo8 994→2513 tps — the spill taxed prefill too); util 53–58% → 95–98%.
Quality: all four tiers stay logit-equiv PASS (real_frac 0, recall_mismatch 0) vs the SAME f16 ref — the GPU dequant reproduces the CPU dequant's InnerQ output distributionally (R1 validated).
Memory + regression: KV MiB identical to the validation-path table; f16/q8_0/q4_0 byte-unchanged.
Residual (root-caused 2026-06-06, #389): turbo plateaus at a uniform 41 tps regardless of tier (turbo8/4/3/2 all ~41 despite 234→61 MiB KV) — the tell-tale of a tier-independent bottleneck. It is the Level-A lift itself: build_attn_mha casts the whole turbo KV view to a full 460 MiB f16 buffer every decode step (ggml_cast(turboN→F16) over all n_kv, redundantly re-dequantizing unchanged positions), then runs stock f16 FA on it. The f16 output is the same size for every tier ⇒ flat ~41 tps; the turbo read (61–234 MiB) is not the bottleneck. This throws away turbo's bandwidth advantage — turbo decodes below q8/q4 (which the FA kernel reads in-register, no materialize). The AtomicBot fork proves the real ceiling: its CUDA path is the fused FA-VEC (fattn-vec.cuh reads TURBO{2,3,4}_0 in-register as "unquantized" K/V + sparse-V skip; full template-instance set). Fork benchmark (M4 Max Metal, fused): turbo3 = 88–92 % of f16 (61.84 vs 70.09 tps MoE; 19.71 vs 21.34 dense), 78.5 tps tg128 on M5 — i.e. ≈ q8/q4-class, not halved. Ratio-extrapolated to our rig (f16=74) the fused target is **63–66 tps turbo3**. Closing the gap = Level B = port the fork's fused FA-VEC turbo path (#390), existing vendored code, the next milestone. (A cheaper "incremental materialize into a persistent f16 shadow" would remove the per-step redundancy but costs a full f16 shadow buffer — negates the runtime memory win and is strictly dominated by the fused path; go straight to Level B.)
Patch capture: the 6-file Level-A edit surface is captured in 0073-turboquant-kv at S3 (#372), registered via the patch README row (same snapshot-diff mechanism as 0070). (S3pA / #383–388)

S2 perf-path Level B — fused FA-VEC turbo (in-register K/V, no f16 materialize) ✅ DONE + GATED (2026-06-06)

Level A's flat ~41 tps ceiling is broken. The whole-cache ggml_cast(turboN→F16) per decode step is gone for the value tiers: flash-attention now reads turbo2/3/4 K/V in-register (centroid×norm, in the WHT-rotated domain), and the expensive Walsh–Hadamard rotation runs only on the n_q-sized Q (forward) and the small FA output (inverse) — never on the KV cache. Acceptance = logit-equivalence, never greedy byte-equality.

New op GGML_OP_TURBO_WHT (ggml/include/ggml.h, ggml/src/ggml.c, ggml-cpu/ggml-cpu.c abort-stub, ggml-cuda/turbo-wht.{cu,cuh} + turbo-wht-core.cuh): direction 0 forward-rotates Q (×scale_inv → signs₁ → WHT → 1/√G·signs₂); direction 1 inverse-rotates the output. InnerQ folds into both WHT kernels via the device symbol turbo_innerq_active_device_scale(ne[0]=head_dim) (B-adapted — no graph-tensor scale src, unlike the fork's mctx tensor). Parseval: WHT(Q⊘s)·WHT(K⊙s) = Q·K, so the InnerQ scale cancels and the turbo K/V reads are InnerQ-free (pure centroid×norm).

Edit surface (all // opencoti-hook: turboquant perf-lift (… Level B)): fattn-turbo.cuh (NEW — in-register vec_dot_fattn_vec_KQ_turbo{2,3,4} + dequantize_V_turbo{2,3,4}), fattn-common.cuh (get_vec_dot_KQ/get_dequantize_V turbo cases), fattn-vec.cuh (is_turbo_K config; turbo rides the float Q path), fattn.cu (turbo→VEC gating, D∈{128,256}), ggml-cuda.cu (TURBO_WHT dispatch + supports_op), template-instances/fattn-vec-instance-turbo{2,3,4}_0 (+ generator), src/llama-graph.cpp (build_attn_mha fused gate). turbo8 + head_dim-512 global layers keep the Level-A materialize fallback (no vec instance / no fork reference).

Graph gate (build_attn_mha): fuse iff flash_attn && turbo2/3/4 && K==V && head_dim∈{128,256} and (L4.5 prefill-hybrid) per-stream n_q ≤ TBV_FUSE_MAX_NQ (default 8) — DECODE fuses, large-n_q PREFILL falls to the Level-A materialize→MMA path (turbo prefill on the vec kernel is ~2–3× slower than f16-MMA).

Gate (GREEN, RTX 3090, Gemma-4 A4B Q4_K_M, 6025-tok prompt, 256-tok forced decode):

mode    | verdict mean_tv | dec@6k  LevelA  low-ctx | prefill | KV MiB | path
----------------------------------------------------------------------------------
f16     | REF    —        |  72.4    73.8    77.6   |  2788   | 460.0  | native
q8_0    | PASS   0.0122   |  65.4    65.9    —      |  2617   | 244.4  | native
q4_0    | PASS   0.0094   |  64.6    65.9    70.2   |  2656   | 129.4  | native
turbo8  | PASS   0.0181   |  42.6    42.5    —      |  2502   | 233.6  | materialize (fallback)
turbo4  | PASS   0.0136   |  48.5    40.6    —      |   817*  | 122.2  | FUSED
turbo3  | PASS   0.0067   |  51.4    41.6    64.1   |  1113*  |  89.8  | FUSED
turbo2  | PASS   0.0179   |  56.0    42.2    —      |  1537*  |  61.1  | FUSED

* always-fuse prefill; the prefill-hybrid (default) restores turbo prefill to ~2474 tps (= materialize) while keeping 51.8 decode (A/B turbo3: T=0 2544/41.4, T=8 2474/51.8, T=huge 1099/52.2; hybrid logit-equiv PASS, mean_tv 0.0063).

Decode ceiling broken + scales with compression: turbo2 56 > turbo3 51 > turbo4 48 > turbo8-materialize 43 tps — the more compressed the tier, the less KV bandwidth/step, the faster decode (the in-register payoff). turbo2 = 86 % of q8_0, 1.31× the Level-A ceiling. GPU util 96–98 % (Level-A ~53 %).
Quality: 4/4 logit-equiv PASS (real_frac 0, recall_mismatch 0) vs the same f16 ref; turbo3 mean_tv 0.0067 is the tightest of any tier — a 3-bit cache beating 4-bit q4 (0.0094) on quality.
Decode-tps is ctx-dependent (turbo3 64.1 low-ctx → 51.4 @6k). Report the ctx.

Tier positioning (decided 2026-06-06, user): at matched bits, native q8_0/q4_0 dominate turbo8/turbo4 on speed AND quality at ~equal VRAM — TurboQuant's MSE centroids buy nothing at 4–8 bit (scalar quant is already near-lossless). Use q8_0/q4_0 for 8/4-bit KV. TurboQuant earns its place at 3-bit and below, where native has no KV type: turbo3 Pareto-beats q4 (90 vs 129 MiB, TV 0.0067 vs 0.0094) at ~80 % of q4 long-ctx decode; turbo2 is the smallest logit-equiv KV that exists (61 MiB, 56 tps) — the 256k play. turbo8/turbo4 stay as ladder-rung / materialize fallbacks; turbo8 is not fused.

Reference (animehacker/llama-turboquant, investigated #395): its headline "tb3 ≈ q4_0" is a regime artifact (Qwen-0.8B, tg128, Strix-Halo UMA APU → KV cost negligible); reproduced on our rig at low ctx (turbo3 64.1 vs q4 70.2 = 91 %). Its kernel does the same float centroid-LUT dot we do; the gap at long ctx is intrinsic centroid-decode compute (turbo3 reads less KV than q4 yet decodes slower). No kernel unlocks q4-parity at long ctx.

L4.5 optimization round: prefill-hybrid = WINNER, shipped (default TBV_FUSE_MAX_NQ=8). int8-dp4a (restructure the KQ dot into integer __dp4a + int8 centroid codebook) — correctness de-risked viable (.opencoti/ab-dp4a-unit2.cu: softmax TV 0.0005), but gather-bound (dp4a speeds the MAC, not the centroid LUT lookups, nor the float V path) and needs re-enabling the q8_1 Q path through the L4-green kernel — deferred to #397 (flag-gated follow-on, correctness already proven). AB1 (fold-WHT-into-FA) costed at ~1–2 % (60 tiny launches/step) — subsumed by #397's q8_1 path; AB2/AB4/AB5 dropped (centroid-LUT is the right primitive per the reference). ⚠️ Correction (user, 2026-06-07): the original AB4 write-off — "turbo8/4 demoted so D=512 turbo-vec moot" — was a non-sequitur. Head_dim-512 coverage is orthogonal to which tiers survive: turbo8/4 being demoted says nothing about whether the 512-dim global layers need a fused turbo path. They do — for turbo2/turbo3 (see below). AB4 should never have been dropped on that basis.

S2 perf-path D=512 fused turbo-vec for turbo2/turbo3 ✅ DONE + GATED (2026-06-07, #411) — ships as `0075-d512-turbo-vec`

Done. The "D=512 turbo-vec moot" verdict above was scoped to turbo8/4 (demoted in favour of native q8/q4). It did not hold for the sub-4-bit tiers: turbo3/turbo2 are the 256k play, and at head_dim 512 (Gemma-4 global layers, n_head_kv=2, ~1 in 6 layers) they fell back to the Level-A materialize→MMA path — the fused gate was head_dim∈{128,256} only. At long ctx that whole-cache ggml_cast(turboN→F16) per step is the decode tax that scales linearly with n_kv, biting hardest at 256k (our primary case).

The assumption that a flat D=512 would blow the VEC register budget (the original reason upstream caps can_use_vector_kernel at ≤256) was WRONG for the turbo path — no split / tiling was needed. The VEC kernel is already D-generic (nthreads/nthreads_V derive from D arithmetically; at D=512, ncols=1: nthreads_V=min(D/4,32)=32→V_cols_per_iter=1, KQ smem max(ne_KQ, nwarps·1·512) halfs ≈ 4–8 KB on the Ampere half2 path — well under the 48 KB static-smem cap — and VKQ ≈ 8 half2 ≈ 16 regs). A plain DECL_FATTN_VEC_CASE(512, turbo2/3) instance compiles + runs clean. Ships as 0075-d512-turbo-vec: fattn-vec.cuh extern decls + the two turbo template instances (+ generate_cu_files.py autogen line), fattn.cu kernel-selection relax (Q->ne[0]==512→VEC) + dispatch, llama-graph.cpp fused_turbo_512 fuse-gate extension. turbo4/turbo8 @ 512 keep Level-A (no D=512 instance).

Gate (.opencoti/p411-d512-gate.sh + p411-perf.sh): correctness = Parseval-EXACT (the WHT is block-diagonal over 128-blocks → D=512 = 4 independent Parseval-exact 128-blocks): fused-vs-materialize logit-equivalence real_frac=0.0 for BOTH turbo3 and turbo2 (bit-equivalent, not merely close). Perf: turbo3 @ ctx 27.7k, fused 41.0 vs materialize 29.7 tok/s = +38%, scaling with n_kv (larger at 256k). No new ABI / flag (auto-engages via the existing TBV_FUSE_MAX_NQ=8 decode gate). This unblocks the long RULER re-run.

Patch capture: the Level-B edit surface joins 0073-turboquant-kv at S3 (#372) — captured + byte-identical-proven (vendored source stays uncommitted per policy; DSO sha 19168f6a, both /root paths restamped byte-identical).

S4 — Speculative MTP draft head (`gemma4_assistant`) for Gemma-4 A4B — adopt AtomicBot fork (HIGH RISK, NEW 2026-06-05)

Pairs with S2 (same fork, same turbo3 KV). Gives ours llamafile a working Multi-Token Prediction speculative-decode path for Gemma-4 A4B: a ~0.4B assistant draft head predicts B-1 future tokens from the target's last hidden state, the 27B-MoE target verifies them in parallel → same output distribution (lossless), ~85-88% accept / claimed +30-50% short-prompt throughput on dense targets. The user frames the head as "EAGLE3 weight layout with low/mid/high feature taps"; the fork ships it as the custom gemma4_assistant architecture (EAGLE-family; eagle3 is a sibling --spec-type).

Draft artifact (published, ready to consume): AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF — F16 (816 MB) / Q8_0 (440) / Q5_K_M / Q4_K_M (310, recommended draft) / Q4_K_S. Target = any Gemma-4 26B-A4B-it GGUF (our gemma-4-A4B-98e-v6-coder-it-Q4_K_M or unsloth's). These GGUFs use the gemma4_assistant arch and will NOT load in stock llama.cpp — that's exactly the gap S4 closes.

Fork usage we must reproduce (CUDA/Linux):

llama-server -m TARGET.gguf --mtp-head ASSISTANT.gguf --spec-type mtp \
   --draft-block-size 3 --draft-max 8 --draft-min 0 -ngl 99 -ngld 99 \
   -ctk turbo3 -ctv turbo3 -ctkd turbo3 -ctvd turbo3 -fa on -c 16384

What S4 must adopt/port into our vendored cosmocc-built llama.cpp (study-and-adapt, not cherry-pick — fork base ≠ our base, and CUDA lives in our DSO):

gemma4_assistant architecture — arch enum + hparams + models/gemma4-assistant.cpp graph builder (the EAGLE3 low/mid/high feature-tap head + centroid LM head for E2B/E4B; dense tied head for 26B-A4B). Loader for the assistant GGUF's tensors.
In-context MTP scheduler — the fork loads the draft into the target context (no second llama_context, no second KV cache), on a dedicated scheduler; the head consumes the target's last hidden state, drafts a block, the target batch-verifies. This composes with M7 residency and S1 unified KV — design the seam carefully (one cache, one slot, draft tokens appended/verified).
Runtime flags (common/arg.cpp): --mtp-head (+ -md alias), --spec-type mtp, --draft-block-size, --draft-max, --draft-min, -ngld, and draft KV -ctkd/-ctvd (depends on S2 for turbo3; until S2 lands, draft KV can ride q8_0/f16).
TS adapter / inference-engine surface — expose the draft-head path through the opencoti tiered-inference config (later; not S4 core).

Gate: with the assistant draft loaded, decode is distribution-identical to no-draft greedy (teacher-forced KL ≈ 0 / top-1 agreement — reuse the #355 logit-equivalence harness), measured accept-rate in the published 70-88% band, and net throughput up vs no-draft on Gemma-4 A4B at RTX-3090. Draft-off byte-identical. NEVER greedy-needle alone.
Risks: (R-S4a) fork base divergence — the arch/scheduler may touch files heavily rewritten by our 0030-0042/0070 patches; budget a real port. (R-S4b) "no second KV cache" claim must be reconciled with our M7/S1 single-unified-cache model — verify the draft's appended tokens don't corrupt the shared/rolling cache. (R-S4c) cosmocc has no nvcc in the host path; any new CUDA kernel (TurboFlash for the draft) goes in the DSO build, not :make.

S3 — Compose-with-glue + bench + patches + docs ✅ DONE + SHIPPED (2026-06-06, #372)

The ship: prove PolyKV composes on the M5 advanced-KV stack, capture the vendored delta into reproducible patches, wire the user-facing TS knob, and land bookkeeping. S4 (MTP draft) is split out as its own future milestone — its 0062 patch is NOT part of this ship.

Adapter knob (P1). The PolyKV user-facing KV-type selector is -ctk/-ctv, surfaced on the opencoti-llamafile adapter as config.ctypeK/ctypeV (env OPENCOTI_LLAMAFILE_CTYPE_K/_V, validated against KNOWN_KV_TYPES incl. turbo{2,3,4,8}), emitted in buildServerArgs after --kv-residency-mode / before --neo-pipeline. Default-unset ⇒ no flag emitted (byte-identical to prior arg vector). S1's pool selection is a per-request JSON param (shared_pool_slot / shared_prefix_n_tokens), server-side, not a CLI flag. 98 adapter tests PASS + typecheck clean.

Compose bench (P1, perf/llamafile/advanced-kv-stack.bench.ts). Added config A4 = baseline

-ctk q8_0 -ctv q4_0 (S0 native asymmetric, greedy-needle-safe) and S′ = the full F5 stack + asymmetric KV. Live (Qwen 0.5B) result 11/12 hard PASS: A4 boots + recovers needle, A4 CUDA KV 4.89 vs baseline 12.00 MiB (0.41× ≤ 0.6× gate), S′ cached_n>0 ∧ needle_recovered, S′ per-cell mem < S per-cell (soft W3 PASS). The sole hard FAIL (C9, S 51.92 vs B 336.05 tps) is the pre-existing M2 head-split perf floor, not PolyKV — A4-alone decodes 128 tps, proving the asymmetric KV path itself is fast; C3/C7 SKIPPED (workload forced no shift this run).

Live turbo3 compose gate (P1b, .opencoti/m6-s3-glue-gate.sh) — the PolyKV ↔ HeadInfer-moot evidence. turbo3 and turbo2 K/V under the live F5 stack flags (--kv-unified --cache-reuse 256 --rest-kv-eviction --rest-kv-recent 256 --rest-kv-layer -1, --parallel 2, Gemma-4 A4B, InnerQ-calibrated) vs the same-stack f16 reference (isolates the KV-type effect, not the stack effect). Both PASS: turbo3 real_frac=0.0 mean_tv 0.0164, turbo2 real_frac=0.0 mean_tv 0.01784, recall_mismatch=0. PolyKV ↔ HeadInfer conflict is MOOT: M7 retired the head-split default (kvResidencyMode auto = window, not the M2 head axis), so TurboQuant's Level-B fused FA-VEC never collides with the M2 head-axis split — they cannot both be active. The gate is the proof: turbo fused decode composes cleanly with window residency + multi-slot reuse + retention eviction.

Patches (P2/P3) — captured + byte-identical-proven. The vendored PolyKV delta over the 0001-0042 + 0070 + 0071 baseline (snapshot-diff of a faithful reset → bootstrap → apply baseline vs the live tree) splits into two patches:

Patch	Stage	Touches	Acceptance
`0072-poly-kv-pool`	S1 SharedKVPool (#370)	3 server files (`tools/server/server-{task.h,task.cpp,context.cpp}`)	default-off byte-identical; gate `m6-s1-shared-pool-gate.sh`
`0073-turboquant-kv`	S2 TurboQuant (tiers+InnerQ+Level-A+Level-B, #371/#374–396)	32 files (13 NEW); `GGML_OP_TURBO_WHT` op; carries the outer `llamafile/build-functions.sh` host-source add	logit-equiv vs f16 (#355); gates `s2-tier-family-gate.sh` + `m6-s3-glue-gate.sh`

Both numbered 0072/0073 (NOT the plan's stale 0060/0061) because the build pipeline applies patches in pure lexical sort order (build-pipeline.ts, .sort()), and the TurboQuant FA-VEC edits sit on top of 0070's fattn.cu/llama-graph.cpp/ggml-cuda.cu rewrite — 0060/0061 would sort before 0070 and break the chain (M7 shipped before M6, so the original labels predate it). S0 asymmetric q8/q4 needs no patch (rides M7 dequant-on-lift natively). Proof: both reverse-apply clean against live (git apply --check -R), and a forward-apply onto a fresh baseline copy reproduces the live tree byte-for-byte (diff -r = 0 differing / 0 only-in-live / 0 only-in-fwd, outer build-functions.sh identical). The full 0001-0042+0070+0071+0072+0073 chain is strict-apply proven by composition (#366 + the baseline :apply + the 0072/0073 --check). Vendored source stays uncommitted per policy — only the .patch files commit. (DSO sha 19168f6a; both /root paths restamped byte-identical.)

S2 perf-path — turbo-MMA verify kernel + scalar-K/turbo-V fused decode ✅ SHIPPED OPT-IN default-OFF (2026-07-09, #628/#630, WS1, `0105-0107`)

WorkStream-1 = the asymmetric strong scalar-K (q8_0/q6_0/q5_0/q4_0) + cheap turbo/TCQ-V decode family — K read as scalar (true-space), V read in-register as rotated turbo/TCQ (no f16 materialize); the graph build_attn_mha wht_q/wht_o split applies the OUTPUT inverse-WHT only (K is true-space, V is FWHT-rotated). Two decode regimes:

Fused-VEC decode (n_q=1) — the foundation, unconditional. 16 new FA-VEC instances (fattn-vec-instance-q{4,5,6,8}_0-turbo{2,3}_{0,tcq}.cu) + FATTN_VEC_CASES_ALL_D dispatch + K≠V selector admission + build-functions.sh 3e default-build inclusion. Invariant K bit-width > V; reverse (turbo-K/scalar-V) not built. Ships in 0107.
turbo-MMA verify (n_q>1) — a turbo-aware tensor-core kernel that dequants q8_0-K + turbo2-V into the MMA smem tile (rotated V; graph wht_o keeps it exact) for MTP verify batches at head_dim 128 (Qwen full-attn), 256 (Gemma SWA-local), 512 (Gemma global). Three per-head-dim knobs WS1_TURBO_MMA_D128/D256/D512, all default-OFF — off ⇒ verify keeps the fused-VEC routing (the pre-#628 default + logit-equiv reference). WS1_DEROTATE (inverse-FWHT V in-load ⇒ graph skips wht_o) is a separate default-OFF experiment. bug-2130: !is_quant_kv for need_f16_K/V so launch_fattn reads raw quant blocks (f16 keeps true,true = byte-identical). Ships in 0107.

VERDICT (#630 — 3-model crossover map on bs2, A4B / omni27 / qw35a3, decode-tps + teacher-forced TV + niah, ctx 32k/128k/256k): turbo-MMA is mid-ctx-specialized and earns NO default-on anywhere.

model (attn)	32k	128k	256k	fidelity
A4B Gemma iSWA, D256-local + D512-global	+4.2% (241.4 vs 231.6)	+6.4% (249.1 vs 234.1)	−14.3% (190.3 vs 222.0)	niah 100
omni27 Qwen3.6-27B dense, D128 full	−0.1% (99.1 vs 99.2)	+0.3% (92.2 vs 91.9)	+0.1% (87.2 vs 87.1)	niah 100, real_frac 0
qw35a3 Qwen3.6-35B-A3B MoE, D128 full	+1.2% (204.8 vs 202.4)	+0.5% (187.7 vs 186.7)	+0.4% (168.0 vs 167.3)	niah 100, real_frac 0

Named mechanism (A4B, nsys --cuda-graph-trace=node): windowed-D256 cast-saving swamped by the ctx-growing D512 global-cast tail. turbo-MMA eliminates the cpy_turbo2_f16 materialize cast on the D256 SWA-windowed V-cache, paying instead ~2× slower D256 attention (in-kernel re-dequant) — both ctx-INDEPENDENT constants (windowed). The saving it must beat is the D512 GLOBAL materialize (cpy_turbo2_f16, still paid — D512 knob off), whose per-cast cost doubles with ctx (123µs@128k → 242µs@256k). So the eliminated (windowed) cast stays fixed while the total step balloons with the D512 tail: net-win @128k (cast eliminated ≈ 2744ms), net-loss @256k (only ≈ 211ms left to eliminate). And it is un-gatable — a per-op k->ne[1] context-gate sees the SWA WINDOW length, not session ctx (bug-2133). On Qwen D128 full-attention the per-op gate WOULD work (uniform n_kv), but there is nothing to gate for: turbo-MMA is bit-exact yet tps-flat because the dense/MoE FFN dominates the verify step — attention acceleration has no leverage.

Ship: OPT-IN, default-OFF (0107); the D128/D256/D512 instances are additive/correct/harmless. WS1's shipped decode wins are 0106 (bug-2126 boot correctness) + 0105 bug-2127 DCA per-tensor defer = the ~2× decode win — independent of turbo-MMA. Chain 0001..0107 strict-applies from clean 0.10.3 and reproduces the FINAL tree byte-identically (verified 2026-07-09).

This WS1 null does NOT transfer to DCA-on — see WS2 (#629). The turbo-MMA "no default-on anywhere" verdict is a noDCA, full-attention verify-batch result: the D512-global cpy_turbo2_f16 cast it had to beat was still paid (D512 knob off), and on Qwen the FFN swamps the attention win. WS2 (#629) is the opposite verdict on the DCA-on decode path: reading the rotated turbo/TCQ-V in-register inside the multichunk DCA fused flash-attn kernel (n_q≤16 gate) eliminates that context-growing whole-cache cast, and is a validated decode WIN of +68–95% @256k across A4B / 31B (D512) and omni27 / qw35a3 (D256), every turbo/TCQ tier, retrieval-lossless (niah 100) — now DEFAULT-ON. Full matrix + caveats: context.md §5x. WS1 turbo-MMA (verify-batch, noDCA) and WS2 in-register turbo-V (decode, DCA-on-fused) are distinct kernels with distinct verdicts.

S4 — Speculative MTP draft head ✅ DONE + SHIPPED (2026-06-07, #373/#404–410, `0074`)

The ship: the Gemma-4 A4B gemma4_assistant Multi-Token-Prediction draft head, loaded into the target (no second llama_context / KV cache), proven a decode-throughput win AND logit-equivalent, captured as 0074-speculative-mtp.patch, surfaced on the adapter, bookkept.

The binding requirement was throughput, and it is MET. MTP-on must BEAT MTP-off in decode tok/s — not just be correct. Final gate on the capture-candidate binary (.opencoti/p4-fused-gate.sh, Gemma-4 A4B v6-coder Q4_K_M @ RTX 3090, 200 tok × 2 prompts, temp 0):

mode	decode tok/s	vs off	draft accept
off (plain greedy)	78.34	—	—
seq (per-step MTP)	113.49	1.45×	0.769 / 0.606
fused (single-graph MTP)	114.03	1.46×	0.769 / 0.606

seq == fused byte-identical (the permitted refactor-equivalence use of byte-equality — two implementations of the same MTP computation). off != seq text divergence is NOT a regression: it is the documented benign batch-vs-sequential near-tie numerical artifact (bug-270) — plain greedy is bit-deterministic (verified off-vs-off A==B, 761 B identical), so the divergence is the deterministic float difference between the spec-decode's batched target-verify and sequential decode flipping an argmax within tolerance. The directive-mandated correctness metric is logit-equivalence, not greedy byte-equality: .opencoti/s4-p3-logit-equiv.sh (8 needle prompts, top-20 teacher-forced) gives real_frac=0.0, frac_full_agree=1.0, 100% draft acceptance (the VERDICT:FAIL string is the known mean_tv artifact for spec-accepted empty-top:[] tokens — judge on real_frac, per [[project_mtp_s4_p3]]).

Root cause of the early "MTP is ~19% SLOWER" result (bug-419) was weight placement, not cuda-graph. Stage-0 measurement (phase-split timers: submit=22 ms, sync=0.01 ms; GGML_SCHED_DEBUG=2) showed the assistant's dense LM head ran as a CPU island mid-graph, stalling the GPU ~~95% idle. The gemma4_assistant tok_embd (~~277 MiB) is the weight-tied dense output head, but llama.cpp classifies tok_embd as an INPUT tensor and dev_input is hard-pinned to the CPU buft — so the per-draft-step full-vocab (262144) output mul_mat executed on CPU. The fix is one loader edit: route the gemma4_assistant tok_embd through the OUTPUT buft (dev_output, GPU when the output layer is offloaded). cuda-graph replay was MOOT (sync already free); the originally-planned custom fused MTP-head kernel was unnecessary. See [[project_dso_cache_poisoning]] for the parallel DSO gotcha class.

Adapter knob. config.mtpHead / specType / draftNGpuLayers (env OPENCOTI_LLAMAFILE_MTP_HEAD / _SPEC_TYPE / _DRAFT_NGL), emitted in buildServerArgs as --mtp-head <path> / --spec-type mtp / -ngld <n> after --fused-moe-up-gate, before extraArgs. -ngld is REQUIRED for the win (a CPU-resident draft head re-seeds the stall). Default-unset ⇒ no draft (plain decode, byte-identical arg vector). 31 adapter tests PASS + typecheck clean.

Patch (0074-speculative-mtp) — captured + byte-identical-proven both directions. Snapshot-diff of the live tree vs a faithful base-mtp (= base-poly + apply(0072,0073) = 0001-0042+0070+0071+0072+ 0073, no MTP): 24 files, 1 NEW (src/models/gemma4-assistant.cpp), 0 outer-vendor delta. Forward-apply onto a fresh base-mtp reproduces the live tree byte-for-byte (0/24 differ); reverse-apply onto live reproduces base-mtp (0/23 differ + new file deleted). Three surgical upstream edits (marker opencoti F5 M6-S4 mtp) + additive surface — registered in docs/protocols/UPSTREAM_SYNC.md. The reverted sparse-argmax centroid fast path (dead for this dense use_ordered_embeddings=false GGUF) is banked at .opencoti/banked/mtp-sparse-argmax.md, not shipped (verification discipline: never ship an unexercised path). Vendored source stays uncommitted per policy — only the .patch commits.

P5 (#409) — POSITION_WINDOW MTP perf path: characterized, NOT implemented (route inapplicable). The task hypothesized routing the MTP cross-read through build_attn_mha_position_window if the get_k streamback under window residency is a bottleneck. Measurement (forced --vram-target 600, n_kv~2919) shows Gemma-4 A4B's iSWA cache is POSITION_WINDOW-ineligible (swa_type=1; eligibility requires swa_type==NONE) — overflow relief is GPU_STREAM, so the proposed window route never engages. Under GPU_STREAM overflow MTP is throughput-negative (MTP-on 30.3 vs off 46.3 tok/s; correct — needle recovered, accept 0.786). Refused to ship a dead position-window MTP route (same discipline as the reverted sparse-argmax). DISPOSITION: the supported long-context MTP path is turbo-KV-resident — S2 Level-B + the D=512 fused turbo instances (#411) keep the 256k cache in VRAM → GPU_RESIDENT → MTP keeps its 1.46× win; f16-overflow-streaming is a config to avoid (f16 KV doesn't fit 256k anyway). See [[bug-419]], bug-426.

Phase-0 fork recon (2026-06-05) — findings that shape S2 + S4

Shallow-cloned the fork (/srv/dev-disk-by-label-opt/dev/_recon/atomic-llama-cpp-turboquant) and diffed its TurboQuant + MTP surfaces against OURS. Headline: the fork's base is contemporaneous with ours, not years apart — both end the ggml type table at MXFP4=39, GGML_TYPE_COUNT=42; both carry the same recent model set (gemma4-iswa, minimax-m2, qwen35*); OURS already has LLM_ARCH_GEMMA4 and COMMON_SPECULATIVE_TYPE_EAGLE3 + a full ngram/draft speculative family. So both milestones are additive onto a near-identical base, far cheaper than a cross-era port.

S2 (TurboQuant) edit surface:

ggml types append cleanly at 42-46 (TURBO2_0/3_0/4_0, TQ3_1S/TQ4_1S), COUNT 42→47 — fork ggml.h:432, traits ggml.c:737. Block structs in ggml-common.h (TURBO3_0 = 14B/128-blk: fp16 norm + 2-bit qs[32] + 1-bit signs[16]).
CPU quant/dequant is ONE new file ggml/src/ggml-turbo-quant.c (~1026 lines: seeded WHT/Hadamard rotation TURBO_SEED_ROTATION=42 + PolarQuant/Lloyd-Max codebook) + dispatcher cases in ggml.c. Self-contained, low-conflict.
CUDA (DSO): turbo-wht.cu, turbo-quant.cuh, mmvq-tq.cu, vecdotq.cuh, + 12 template- instances/fattn-vec-instance-turbo*.cu. The fattn hook is the only CRITICAL conflict (OURS 0070 heavily rewrote fattn.cu).
CLI: kv_cache_types[] in common/arg.cpp maps "turbo3" → enum (keep the spelling for fork-GGUF compat).
Bring-up order: Gate-1 = TURBO3_0 CPU-only (enum + struct + ggml-turbo-quant.c + traits + dispatcher + CLI, NO CUDA, NO fattn) → validates WHT/codebook + type registration with zero fattn.cu risk. Gate-2 = CUDA dequant w/ dequant-then-compute fallback (reuse our S3d lift). Gate-3 = fused TurboFlash templates (the 0070 rebase).
Gate-2 design (2026-06-05, decided). The fork's convert.cu turbo3→f16 emits rotated f16 (it relies on the fused kernel to rotate q on-the-fly). Our S3d "dequant-on-lift → standard f16 FA" path does NOT rotate q, so the correct fallback un-rotates K/V instead: dequant turbo3 → rotated f16 → ggml_cuda_turbo_wht (direction=1, inverse) → true-space f16 K/V → unmodified standard f16 FA. Minimal additive port set: turbo-quant.cuh + turbo-wht.cu /.cuh (verbatim) → dequantize.cuh turbo3 accessor → convert.cu to_fp16 cases → graph insertion of the inverse-WHT pass on the lifted K/V → supports_op/type registration in ggml-cuda.cu → CUDA DSO rebuild → GPU cosine/needle gate vs vanilla f16.
- InnerQ stub-disabled for Gate-2 (decided, user). turbo-quant.cuh couples to InnerQ device globals (d_innerq_scale*, per-channel Q/V equalization) defined in turbo-innerq.cu. For the Gate-2 fallback we compile with identity InnerQ (scales = 1, d_innerq_active = 0) — either by porting turbo-innerq.cu in its dormant/identity state or by stubbing the globals — so K/V un-rotation is the plain inverse WHT. InnerQ is re-enabled in Gate-3 (the fused TurboFlash path), where the q-rotation + channel-scale fusion actually pays off. Identity InnerQ is numerically exact for the un-rotation (it only rescales channels pre-quant; with scale=1 the round-trip is the bare WHT codebook path Gate-1 already validated at cosine 0.983).
- Gate-2 is two halves (both mechanism-matched to our tree):
  - READ (dequant → true f16) — source DONE 2026-06-05. New ggml-cuda/turbo-dequant.cuh (InnerQ-free; folds the inverse WHT into the per-block dequant since turbo3 block=128=WHT group) + convert.cu to_fp16_cuda & to_fp16_nc_cuda TURBO3_0 cases (use dequantize_turbo3_0_inv) + QR_TURBO3 define in ggml-common.h. Routes through our type-agnostic GGML_OP_STREAMING_FLASH_ATTN (supports_op gates head_dim only; the op dequants K/V to f16 on the S3d lift at fattn.cu:748) → no standard-FA supports_op change, no graph node added. The gate must force the streaming path (residency-mode / low --vram-target) so turbo3 never hits the stock FLASH_ATTN_EXT (which lacks a turbo3 vec kernel until Gate-3).
  - WRITE (f32 → turbo3) — NEXT. Our KV store uses ggml_set_rows (cpy_k/cpy_v → ggml_set_rows, llama-kv-cache.cpp:3091/3188/3204) — the SAME mechanism as the fork's set-rows.cu. Port k_set_rows_turbo3 + k_set_rows_turbo3_tail + set_rows_cuda_turbo3 (~300 lines: group forward-WHT via warp shuffle, __ballot_sync sign pack, corrected-norm) into our set-rows.cu (which today has 0 turbo refs), + GGML_OP_SET_ROWS supports_op / dispatch for a turbo3 dst. Then ONE CUDA DSO rebuild + GPU cosine/needle gate vs vanilla f16.
Gate-1 — DONE (2026-06-05). Host build clean (ggml-turbo-quant.c.o linked; two splice bugs fixed: bug-333 truncated TQ4_1S type_traits entry, bug-334 decls trapped inside #ifdef __cplusplus). Registration proven by the boot line llama_kv_cache: … K (turbo3): 1.37 MiB, V (turbo3): 1.37 MiB + attn_rot_k = 1 on Qwen-1.5B CPU (-ngl 0 -ctk turbo3 -ctv turbo3) — i.e. -ctk turbo3 accepted, type name resolves, cache allocates at the compressed block size (~5× vs f16's ~7 MiB). CPU then segfaults at warmup — EXPECTED: dequantize_row_turbo3_0 is a rotated-space stub (no inverse WHT; the GPU FA kernel rotates q so (Rq)·(Rk)=q·k), so turbo3 attention compute is Gate-2/3 (CUDA) scope, not a Gate-1 fail. Gate-1c math validated out-of-band: vendored kernel primitives (turbo_cpu_fwht/_inverse/nearest_centroid_3bit/CENTROIDS_3BIT) are byte-identical (modulo GGML_API macro + a comment) to _recon/turbo3_selftest (cosine 0.983 round-trip). Gate script .opencoti/s2-gate1.sh. Next: Gate-2 (CUDA dequant-then-FA via S3d lift) — requires DSO rebuild.

S4 (MTP draft head) edit surface:

gemma4_assistant arch = new file src/models/gemma4-assistant.cpp (clean) + enum/hparams in llama-arch.{h,cpp} (n_embd_backbone, n_centroids, centroid_top_k, use_ordered_embeddings, requires_target_arch). EAGLE3 feature taps = target token-emb + prev backbone hidden → mtp_pre_projection → transformer → backbone hidden → (centroid head for E2B/E4B | dense head for A4B). Greedy argmax on device.
"No second KV cache" CONFIRMED in code: the assistant is loaded INTO the target via llama_model_load_mtp_from_file() (tensors prefixed mtp., stored as target->mtp_assistant); there is no second llama_context. MTP cross-attn reads the target's existing KV at the last full/SWA target layer (build_attn_mtp() → gemma4_mtp_kv_layer_last_in_range). Scheduler = common_speculative_state_mtp (common/speculative.cpp) + llama_decode_mtp_async/wait (llama-context.cpp), async depth-2 with lazy-wait overlap; MTP only READS target KV, never writes.
Flags additive in common/arg.cpp: --mtp-head/-md, --spec-type mtp, --draft-block-size (default 3), --n-draft-max/min, -ngld, -ctkd/-ctvd.
#1 integration risk (R-S4b, confirmed): build_attn_mtp() must read target KV through OUR M7 rolling-KV (POSITION_WINDOW) + S1 accessor dispatch, not the fork's direct cache_k/cache_v; and S1/M7 must not mutate the target prefix mid-MTP-snapshot. Mitigation: route MTP's attn through the same accessor as the regular forward; keep the slot in decode-state (no compress) while an MTP draft is pending. Fallback: dequant-on-lift the window into a temp buffer for the MTP read.
Bring-up order: P1 arch+loader (smoke-load the assistant GGUF, no decode) → P2 flags/params → P3 scheduler skeleton (async no-ops) → P4 KV-accessor merge (single-token MTP forward, cosine vs naive) → P5 async + S1 race-guard (turn-2 latency + logit-equivalence) → P6 docs/patch 0062.

Critical files (anticipated)

src/llama-kv-cache.{cpp,h} — pool/shared-prefix storage, type-asymmetric K/V, compression-ratio into the M7 tile-sizing.
ggml/src/ggml-cuda/fattn.cu + S3d dequant-on-lift — read compressed tiles (S0 reuses; S2 adds the TBQ8_0 / TBQ4_0 / TBQ3_0 dequant kernels).
ggml core — TBQ8_0 / TBQ4_0 / TBQ3_0 type registration (S2).
common/arg.cpp — pool/agent + (later) --poly-kv knobs; packages/opencoti-llamafile adapter.
Patches 0072-poly-kv-pool (S1) / 0073-turboquant-kv (S2); gates .opencoti/m6-s*-*.sh; perf/llamafile/advanced-kv-stack.bench.ts.

Verification

S0: single-tenant cosine/RULER == vanilla at asymmetric q8_0/q4_0; ~2.4× KV; M7 overflow path OK.
S1: 2-agent shared-prefix recall; O(1) shared-prefix memory.
S2: cosine/RULER holds per TurboQuant tier (TBQ8_0 / TBQ4_0 / TBQ3_0); TBQ3_0 ~2.9× compression.
S3: glue bench (A4/S′ + turbo3/turbo2 logit-equiv compose); patches 0072/0073 strict-apply in series after 0071, reverse-apply clean + forward-reproduce byte-identical; hooks registered; reap/DSO hygiene. ✅

Standing ops

OPENCOTI_NO_CCACHE=1; :cuda rebuilds DSO (~24min) only when S2 lands its kernels — S0/S1 are host-side; vendor-backup before any destructive vendor op; reap llamafile.*--server (never embedder 38092 / SSH); DSO restamp before each ours boot; GPU ≥10 GiB; commit only when asked; soft-fork additive + opencoti-hook: markers registered; RULER acceptance via cosine/semantic/RULER, never greedy byte-equality.