docs/evaluations/mtp.md

MTP / speculative decoding — evaluation

Status (2026-07-14): both flavours shipped, characterized, and at throughput parity-or-better vs upstream b9859. Gemma assistant-MTP runs upstream's dual-context execution (§3e — bug-858's two root causes found+fixed). #609 depth-3 CLOSED (bug-2103 TinyBLAS f32 mmvf fix). #675 (§3h, patch 0128) closed the last per-cycle host gap (bug-2180 common_sampler_clone ~1.8 MiB deep-copy under cosmo dlmalloc = 1.07 ms/cycle + mmvf f16/bf16 verify lift): Qwen 35B NextN −12% → −1.7%..+3%, and A4B assistant now BEATS b9859 at every depth (9-prompt decode mean +6.6/+12.9/+7.6% at n1/n2/n3), accept bit-identical (pure-latency fix). Residuals: accept deficit at depth ≥2 (#495-class chained-draft numerics, −4.4pp aggregate at n3) and 31B/27B −2..−6% decode. Recommend n2–n3 on A4B, n3 on Qwen NextN. Current cross-engine numbers incl. the 9-prompt per-prompt table: three-way-tps.md §2026-07-14.

This document is the rigorous record for Multi-Token Prediction (MTP) speculative decoding in the opencoti llamafile fork, across both flavours and all supported architectures. It is the MTP-specific companion to context.md (RULER retrieval / DCA long-context) — where context.md proves quality at length, this proves speed without quality loss from speculation.

Throughput (gen tok/s), draft acceptance, and the correctness gate are reported per model. The single invariant across everything here:

MTP speculative decoding is lossless by construction. The target model's verify pass decides every emitted token; the draft head only proposes. So the only correctness question is whether our wiring preserves that invariant — proven per model by logit-equivalence (MTP-on vs the plain target), never by greedy token-match (FP-unstable; bug-270/356).

---

1. The two MTP flavours

Post-0.10.3 the two flavours are distinct --spec-type strings (the old unified --spec-type mtp token is RETIRED) with distinct enums/graph types, dispatched in common/speculative.cpp / server-context.cpp: the Gemma external assistant is --spec-type draft-assistant (COMMON_SPECULATIVE_TYPE_MTP, our patch 0079), the Qwen NextN self-spec is --spec-type draft-mtp (COMMON_SPECULATIVE_TYPE_DRAFT_MTP, upstream-native). They differ in where the draft head lives:

Flavour	Models	Draft head	Cache	Rollback
External assistant (M6-S4)	Gemma-4 A4B/12B/27B/E2B/E4B	a separate gemma4_assistant GGUF loaded INTO the target (--mtp-head)	iSWA (attention, seq_rm-able)	FULL — plain seq_rm of the rejected tail
NextN self-spec (M6-S5)	Qwen3.5-9B (qwen35), Qwen3.6-27B-Omnimerge (qwen35), Qwen3.6-35B-A3B (qwen35moe)	baked into the target GGUF (last nextn_predict_layers layer) — no second model	hybrid (gated delta-net recurrent backbone + full-attention layers)	CKPT — recurrent state checkpoint + re-decode

Selection is explicit per engine: --spec-type draft-assistant --mtp-head <gemma4_assistant.gguf> for the Gemma external assistant; --spec-type draft-mtp (no --mtp-head; nextn_predict_layers>0 baked into the target) for Qwen self-speculation. The TS adapter emits the right string from specType:"draft-assistant" (+mtpHead) or specType:"draft-mtp" respectively (config.ts resolveEnvSpecType; the retired "mtp" value is no longer accepted).

1a. Why qwen NextN needs a different rollback (checkpoint; upstream-native on 0.10.3 — W3b was opencoti's pre-bump port)

The qwen NextN models are hybrid: most backbone layers are gated delta-net (a recurrent linear-attention whose state has no per-token history), and only the NextN/full-attention layers keep a seq_rm-able KV cache. On a partial draft acceptance, the recurrent backbone has already advanced through the rejected draft tokens and cannot erase its tail — llama_memory_seq_rm returns false, and the upstream gate would simply disable speculation for these models.

We do not disable it (locked user directive). Instead we port upstream PR#19493's checkpoint idea, adapted for self-spec (the NextN draft head is stateless/regenerating, so upstream's discard-and-retry would re-emit the identical draft and livelock):

1. Compat probe is three-way now (common_speculative_is_compat → NO / FULL / CKPT). A target that decodes but can't partial-erase its tail returns CKPT — spec stays enabled.
2. At draft time (CKPT mode) the server snapshots the committed-prefix recurrent state (llama_state_seq_get_data_ext, PARTIAL_ONLY → only the recurrent half of the hybrid cache).
3. On a partial accept the server restores that checkpoint, trims the attention KV to the prefix, then re-decodes the accepted tokens to re-advance both halves to exactly the accepted sequence (the last token carries logits to seed the next draft's h_prev).

Cost: in principle one extra small decode per partial-accept step + one PARTIAL_ONLY snapshot per draft step (the per-token ring buffer of PR#22673 would make both O(1) — tracked as #457). PROFILED 2026-06-18 (§4b): on the 0.10.3 binary this path is DORMANT and the cost is MEASURED = 0 — upstream's hybrid context supports bounded partial sequence removal (common_context_can_seq_rm), so for the default n_max=3 draft the recurrent rollback (n_rollback ≤ 3) is erased natively by seq_rm and the checkpoint-restore/re-decode branch (use_ckpt_*) is never entered. The "seq_rm returns false" premise above was the pre-bump W3b situation; the upstream-native 0.10.3 path no longer needs the re-decode at shipped draft widths.

On the post-0.10.3 binary the shipped draft-mtp path is the upstream-native common_speculative_impl_draft_mtp (NextN self-attention drafter + the checkpoint/recurrent-rollback above; server spec_ckpt, server-context.cpp), and it is n_seq-aware — its samplers / backend-chains / pending_h are all sized per seq, so it runs per-slot under --parallel (proven §5a/#485). This supersedes the --parallel 1 limit of opencoti's retired pre-bump W2/W3b CKPT self-spec. There are no // opencoti F5 M6-S5 source markers in the vendored tree: the NextN drafter and its rollback are upstream-native; opencoti's only NextN-side vendored change is the W1 loader fix (#436 — load the NextN head

• skip DeltaNet on the NextN layer). See the W2/W3b design record under .opencoti/reconcile/phase0-20260615/.

---

2. Hosts & methodology

Host	GPU	Role
solidpc	1× RTX 3090 (24 GB)	Throughput (off/on gen-tps) + correctness gates (logit-equiv). Q4_K_M / Q8_0 / IQ3 quantized models.
bs2	RTX 6000-class	256k-context MTP + DCA-on cells (large KV).

• Throughput: same fixed decode prompt, --temp 0, gen tok/s from the server completion timings.predicted_per_second. off = no --spec-type; on = self-spec / --mtp-head. Gate: .opencoti/w6-qwen-mtp-tps.sh.
• Acceptance: from the MTP NDJSON trace (LLAMA_MTP_ACC_TRACE) — accepted/drafted draft tokens and accept events (statistics mtp in the server log corroborates).
• Correctness: logit-equivalence — teacher-force shared prompts, compare per-position top-20 logprobs MTP-on vs plain; PRIMARY = real_frac (real, non-benign-tie argmax disagreements; target 0). mean_tv is advisory only (it false-fails on spec-accepted tokens whose server returns empty top-logprobs — see the gemma precedent below). Gate: .opencoti/w4-qwen-nextn-logit-equiv.sh.

---

3. Gemma external-assistant MTP (M6-S4) — shipped

3.0 Recommended config (the adapter productizes this)

MTP is the largest single decode-throughput lever for the turbo / turbo_tcq KV tiers (it amortizes the bandwidth-bound expert-weight reads across the draft window — see docs/evaluations/decode-levers.md L1/L2), and it is logit-equivalent (§3a). The engine needs three llamafile flags acting together — --mtp-head <gguf> --spec-type draft-assistant -ngld 99 — but as of #533 the opencoti adapter (packages/opencoti-llamafile/src/config.ts, buildConfig) derives the companions so you set just one knob:

You set	Adapter derives	Result
mtpHead: "<assistant.gguf>"	specType="draft-assistant", draftNGpuLayers=99	full gemma assistant-MTP, GPU-offloaded draft head
specType: "draft-mtp" (no mtpHead)	(nothing)	qwen NextN self-spec (rides the target's own offload)
(neither)	(nothing)	plain decode

Explicit values always win (e.g. set draftNGpuLayers: 40 to override the 99 default). Env equivalents: OPENCOTI_LLAMAFILE_MTP_HEAD, OPENCOTI_LLAMAFILE_SPEC_TYPE, OPENCOTI_LLAMAFILE_DRAFT_NGL. The -ngld 99 default matters: a CPU-resident draft head stalls the GPU and erases the win. Recommended pairing for turbo tiers: ctypeK/ctypeV = turbo3_tcq + mtpHead = <A4B assistant> → turbo3_tcq decode ≈ 89 tok/s on RTX 3090 (graphs+MTP, +69% over the 52.9 plain baseline). Covered by test/mtp-defaults.test.ts.

3.0a. Per-KV-tier MTP gain — re-confirmed on coherent output (#553, A4B v6-coder Q4_K_M, solidpc, 2026-06-24)

The §3.0 claim holds across the KV ladder. A4B chat coding prompt (in-domain, /v1/chat/completions, --temp 0, 256 gen tok, graphs default, coherent-mtp.sh):

KV tier	MTP-off tok/s	MTP-on tok/s	MTP gain	draft acceptance
f16	88.95	112.37	+26%	0.787
turbo3	69.66	94.44	+36%	0.827
turbo3_tcq	56.63	75.63	+34%	0.652

Re-confirmed on the fresh 0121 binary (#654, A4B v7-coder Q4_K_M, solidPC RTX 3090, ctx 8192, sp-mtp30a.sh, 2026-07-12): the per-tier win holds and is larger on the current engine —

KV tier	MTP-off tok/s	MTP-on tok/s	MTP gain	draft acceptance
f16	76.58	137.03	+79%	0.912
turbo3	65.67	97.10	+48%	0.864
turbo3_tcq	53.18	74.03	+39%	0.663

Same ordering (biggest on quantized tiers), same acceptance profile (turbo3_tcq's lower single-prompt 0.66 is the documented prompt-noise, not a deficit) — a clean reproduction on the shipping binary.

MTP is a real decode-tps win on every tier, biggest on the quantized-KV tiers (the §3.0 lever), and the two graph layers compose (graphs survive turbo3_tcq + sparse-V: +13.4% / +13.1% on today's DSO, matching the decode-levers L6 reference +13.1%). The single-prompt turbo3_tcq acceptance (0.65, BST prompt) is prompt noise, not a deficit: averaged over 3 diverse coherent coding prompts (coherent-mtp-multi.sh) the means are f16 0.745 / turbo3 0.757 / turbo3_tcq 0.703 — a ~0.05 spread within each tier's own prompt-to-prompt variance, and on one prompt turbo3_tcq (0.766) beat turbo3 (0.721) and f16 (0.729). Consistent with turbo3_tcq being the more accurate tier (§3a is logit-equivalent; 27B KLD 0.0346 < turbo3 0.047): no spec bug.

⚠️ Acceptance/tps MUST be measured on coherent output. Feeding an instruction-less prompt (e.g. a raw RULER/WikiText haystack to /completion) collapses greedy decode into a degenerate repetition loop where logits are near-tied; draft acceptance then becomes meaningless noise and even f16 shows greedy MTP-off ≠ MTP-on (impossible on real tasks — f16 is proven real_frac=0, §3a). An earlier sweep on that prompt mis-measured turbo3_tcq acceptance at 0.25 and read "MTP hurts turbo" — a pure harness artifact (bug-710), not a kernel/codebook defect. Use /v1/chat/completions with a real instruction. Gemma-4 thinking-coder writes its answer to choices[0].message.reasoning_content, not .content (.content is empty with finish_reason=length) — parse reasoning_content. Latent note: the TURBO_TCQ_CB runtime codebook override reaches only the FA-VEC TU's __constant__, not the MMA-lift TU (d_turbo3_tcq_cb_cpy); the two baked tables are byte-identical so default decode is consistent, but an A′ codebook sweep would diverge draft (VEC) vs verify (MMA-lift) — fix before relying on TURBO_TCQ_CB for spec/MTP runs.

3a. Correctness — logit-equivalence (A4B v6-coder Q4_K_M, solidpc)

s4-p3-logit-equiv (MTP verify path vs plain target, 8 teacher-forced prompts):

metric	value	meaning
real_frac	0.000	no real argmax divergence → logit-equivalent
real_disagreements	0	—
frac_full_agree	1.000	every position's argmax matched over the shared prefix
mean_top1_mae	0.000	top-1 logprob identical
mean_tv (advisory)	0.444	trips the catastrophe guard but is the empty-top artifact, NOT a real fail

Verdict: lossless. (The compare script's literal VERDICT printed FAIL purely on the advisory mean_tv guard — this is the documented spec-accepted-token artifact; the gate reads real_frac.) Acceptance on these prompts: 1.00 (e.g. statistics mtp … #acc drafts = N / #gen drafts = N).

3b. Throughput — MTP off vs on (all 5 Gemma targets) + ours-vs-upstream (#441/#442)

Two regimes. The canonical figure is the A4B RULER@32k milestone (long-context, the representative win). The series sweep (.opencoti/gemma-mtp-series-sweep.sh) fills all five targets at a fast, consistent 4k-ctx / n_predict-256 / temp-0 / 2-prompt regime, and — answering #441's "also include the llama.cpp PR benchmark" — is run a second time against upstream llama.cpp's native Gemma assistant-MTP (b9660, PR #23398/#24282) at the same regime on the same RTX 3090 (.opencoti/upstream-mtp-tps.sh), so ours-vs-upstream is apples-to-apples. accept = accepted/generated drafts (identical log metric both sides; upstream reads it from the completion JSON draft_n_accepted/draft_n, ours from the server draft acceptance line — same AtomicBot-lineage definition). Measured 2026-06-18 on solidpc RTX 3090.

Canonical (A4B, RULER@32k): off 56.6 → on 88.8 tok/s = 1.57× (#484, post-bump), accept 0.87. Absolute tps is lower than the 4k sweep below (long KV); the ~1.5–2× speedup is the regime-robust signal.

Series sweep — ours (llamafile 0.10.3 + #491 host-guard; single-context --mtp-head / --spec-type draft-assistant; FA on throughout):

model	arch	off tok/s	on tok/s	speedup×	acceptance
Gemma-4 A4B-26B	gemma4 (iSWA)	79.14	109.01	1.38×	0.631
Gemma-4 E2B	gemma4 (elastic)	132.84	195.23	1.47×	0.533
Gemma-4 E4B	gemma4 (elastic)	86.79	145.03	1.67×	0.448
Gemma-4 12B	gemma4	48.57	100.44	2.07×	0.792
Gemma-4 31B	gemma4	19.64	39.13	1.99×	0.662

Series sweep — upstream b9660 native (ctx_other engine; -md … --spec-type draft-mtp --spec-draft-n-max 3 -fa on; same HW + regime):

model	off tok/s	on tok/s	speedup×	acceptance	note
A4B	92.92	147.97	1.59×	0.839
E2B	157.81	264.43	1.68×	0.628
E4B	99.28	CRASH	—	—	-fa on aborts (D512/gqa=2 FA-kernel gap, bug-567); upstream's own documented workaround is global -fa off → 98.47 → 182.07 (1.85×, 0.542)
12B	51.70	137.59	2.66×	0.928
31B	20.45	51.09	2.50×	0.788

Upstream PR-published references (PR #23398/#24282, mixed HW — not directly comparable; cited as the "PR benchmark" the request asked for): 31B Q4 1.55× / Q8 1.97× (RTX PRO 6000); 31B 1.74× (2×RTX 3090, the closest HW match); DGX Spark wall 290.0 s → 120.7 s (≈2.4×), accept 0.588; "dense >2× on average, MoE none"; E4B accept 0.479 (125/261) — within noise of our 0.448. Repro: python3 scripts/mtp-bench.py against llama-server … --spec-type draft-mtp --spec-draft-n-max 4. Upstream's own caveats, both load-bearing for the §3d decision: quantized-KV → 0% acceptance (later fixed for Q only, still composition-fragile) and -fa on crashes E4B (PR #24282, verbatim).

Three honest readings.

1. The off-baseline gap is a base-version artifact, not an engine gap. Upstream out-decodes us even with MTP off (A4B 92.9 vs 79.1; 12B 51.7 vs 48.6) — that's newer b9660 kernels vs our 0.10.3 pin, the price of carrying the opencoti KV moat upstream's base cannot. The MTP speedup ratio is the engine-comparable number; there upstream's separate-context engine is modestly ahead (1.6–2.7× vs our 1.4–2.1×).
2. E4B is a capability divergence, not a 4k-speed win. Upstream crashes with FA on and must disable FA globally; opencoti's surgical #491 host non-FA guard disables FA only for the one uncovered op (D512/gqa≤2) and keeps it everywhere else → E4B runs at 1.67× with FA on. At 4k, upstream's FA-off workaround is actually faster in raw tps (182 vs our 145; newer base + no-FA penalty is negligible at short KV) — so the #491 advantage is latent at 4k; it pays off at long context (FA-off is O(n²)) and for FA-dependent composition, not in this short-ctx cell. Plainly: ours keeps a capability upstream sacrifices; upstream is faster on the trivial case.
3. The moat is composition, not raw speed. Upstream's ctx_other engine cannot compose with PolyKV / rolling-KV / M2 split / TurboQuant / DCA and collapses under quantized KV — see §3d. That, plus E4B-FA-on, is why opencoti keeps its single-context engine despite conceding the f16/short-ctx tps race.

Regime caveat: the sweep rows are 4k/n256 (fast, same-regime), not RULER@32k — absolute tps is higher at 4k (short KV). Compare speedups and the ours/upstream deltas, not absolutes, against the canonical A4B row; A4B appears in both as the cross-regime anchor.

3b′. am17an PR benchmark — A4B, single-harness ours-vs-upstream (consistent acceptance, 2026-06-18)

The §3b series tables read ours-acceptance from the per-request draft acceptance server line but upstream-acceptance from the cumulative completion JSON — an asymmetric aggregation, fine for the speedup column but not a like-for-like acceptance read. To settle ours-vs-upstream on one metric from one script, the canonical am17an PR benchmark (mtp-bench.py, gist 228edfb84 / PR #23398 — 9 fixed prompts, max_tokens=192, seed=42) is run verbatim against every engine; it reads timings.draft_n / draft_n_accepted identically for ours and upstream. A4B v6-coder Q4_K_M, ctx 8192, draft depth 2 (ours' measured optimum: --draft-block-size 3 ↔ upstream --spec-draft-n-max 2), --temp 0, FA on, RTX 3090 (.opencoti/am17an/run-gemma-a4b.sh):

engine	spec path	accept	drafted	accepted	mean tok/s	×ours-off
ours-off	base, no spec	—	0	0	78.04	1.000
ours-seq	--mtp-head draft-assistant, per-step sync	0.6628	1474	977	109.66	1.405
ours-fused	--mtp-head draft-assistant, fused (shipped)	0.6635	1474	978	110.07	1.410
up-off	b9660 base, no spec	—	0	0	82.88	1.062
up-mtp	b9660 -md … --spec-type draft-mtp (ctx_other)	0.7738	1344	1040	123.00	1.576

Upstream MTP-on is 1.118× ours MTP-on (123.00/110.07), which factors cleanly into two independent levers, ≈half each:

1. Base decode — 1.062× (base-version-bound). ours-off 78.04 vs up-off 82.88. CUDA graphs are reused on both sides (graphs reused = 190…1700+ in our server log), so per-token cost is GPU-kernel-bound, not launch overhead — it is purely newer kernels in b9660 (7dad2f1a1) vs our pinned base dbe9c0c8c (a confirmed direct ancestor; 20 CUDA commits between them). The A4B-relevant ones: #24087 enroll mul_mat_vec_q_moe into PDL — A4B is MoE, and that GEMV is its single hottest decode kernel — and #23729 MMVQ Turing parameter table (Q4_K_M decode matmuls); the broader PDL (Programmatic Dependent Launch) cluster trims dependent-launch stalls even inside captured graphs. This 6% is the cost of the KV-moat pin: we hold dbe9c0c8c so the 24-patch stack applies, foregoing these June kernel PRs. Recoverable only by a base bump (re-triggering the rebase friction).
2. MTP speedup ratio — 1.053× (engine/acceptance-bound). ours speedup-over-its-own-base 1.410× vs upstream 1.484× (123.00/82.88), driven by acceptance 0.664 vs 0.774 on the identical drafter GGUF. Note ours drafts more (1474) yet accepts fewer than upstream (1344 drafted, 1040 accepted): our single-context cross-read over-proposes lower-quality 2nd-step drafts where upstream's ctx_other recompute does not. Base-independent; a genuine future opportunity inside our control-plane (the same-draft-position hypothesis #492 was tested and proved inert — prompt-1 152→151 accepted, a 1-token regression; the lever is elsewhere, the 2nd-token hidden-state source). Not chased here.

Neither lever is the E4B FA crash — A4B is dense-MoE with gqa>2 and never hits bug-567; E4B's FA-on capability is the §3b row. This A4B block is the authoritative consistent-acceptance ours-vs-upstream comparison; the §3b series tables remain the 5-model coverage (with the noted aggregation caveat on their A4B acceptance cell).

3b″. E4B D512/gqa≤2 — real flash-attn, #491 non-FA fallback retired (#493, patch 0080, 2026-06-18)

The §3b E4B row ran on the #491 host stop-gap: a build_attn_mha guard that routed the drafter's single GLOBAL layer (head4/kv2 → gqa 2 at head_dim 512 — the bug-567 kernel hole) to the non-FA explicit-softmax branch, so -fa on no longer aborts but that one layer is "fa off". #493 (patch 0080) replaces that with a real D512 f16 flash-attn — a new FATTN_VEC_CASE(512, F16, F16) VEC instance + selector route + VEC dispatch case, and the #491 guard narrowed to fire only for the residual no-kernel case (quantized/bf16 D512/gqa≤2, which the assistant drafter never produces). The VEC kernel tiles Q->ne[1] (cols_per_block 1 decode / 2 prefill+verify), so it is a real FA at any n_q.

E4B -fa on, assistant-MTP	decode tps	acceptance	verified output
OLD (#491 non-FA soft_max on the global layer)	150.97	0.67910	reference
NEW (#493 D512 f16 VEC flash-attn)	147.86	0.67910	byte-identical

Correctness is decisive: assistant-MTP is exact spec-decode (the target re-validates every drafted token), so a wrong drafter FA cannot corrupt the verified greedy output but would collapse acceptance — equal acceptance (Δ=0) and a byte-identical verified prefix together prove the VEC FA numerically correct. Decode tps is within boot-to-boot noise (the global layer is one of many; −2% is not a regression). The two head-size switches in fattn.cu (selector and VEC dispatch) both need the D512 case or the selector picks VEC while the dispatch aborts at decode time (fattn.cu:363). A4B is gqa 8 (head16/kv2 on its D512 globals) → untouched MMA path, no #493 code; its raw-/completion "garbage" is a thinking-coder prompt-template artifact (bug-573), coherent via /v1/chat/completions. Gate: .opencoti/m493-fa-d512-gate.sh. DSO rebuilt byte-identical both paths (sha256 a8ddcd82).

3b‴. #654 re-run on the fresh 0121 binary — both hosts (2026-07-12)

Refreshes §3b on the current shipping engine (patch chain through 0121; the §3b "ours" table above was the retired #491 host-guard stopgap, now superseded by the #493 proper D512/gqa≤2 FA). Two hosts, both single-GPU pinned (no cross-GPU host contention), robust decode-tps method (8-tok cache_prompt warmup → 256-tok cache-reused measured completion, timings.predicted_per_second); accept = draft_n_accepted/draft_n. bs2 = §3b family (ctx 32768); solidPC = sp-mtp-family (ctx 8192).

bs2 (RTX 6000, mtp3b-family.sh, ctx 32768):

model	arch	off tps	on tps	speedup×	accept
Gemma-4 A4B	assistant	188.90	320.04	1.69×	0.903
Gemma-4 E2B	assistant	270.38	452.02	1.67×	0.985
Gemma-4 E4B	assistant	185.25	365.72	1.97×	1.000
Gemma-4 12B	assistant	127.61	266.17	2.09×	0.907
Gemma-4 31B	assistant	42.61	86.55	2.03×	0.563
Qwen3.6-27B-Omni	nextn	52.55	99.55	1.89×	0.637
Qwen3.6-35B-A3B	nextn	195.26	275.79	1.41×	0.868

The E4B -fa on cell is the direct #493/bug-567 verification: 1.97× at accept 1.000, no abort — confirming the D512/gqa≤2 FA gap is fixed on ours (the "-fa off" caveat in §3b belongs to upstream b9660, not opencoti).

solidPC (RTX 3090, sp-mtp-family.sh, ctx 8192):

model	off tps	on tps	speedup×	accept
Gemma-4 E2B	151.28	328.82	2.17×	0.959
Gemma-4 E4B	92.26	233.57	2.53×	0.964
Gemma-4 12B	48.43	117.84	2.43×	0.851
Gemma-4 A4B (iq4xs)	80.34	156.15	1.94×	0.990
Qwen3.6-27B-Omni	54.37	85.49	1.57×	0.629
Qwen3.5-9B NextN	45.29	67.62	1.49×	0.527
Gemma-4 31B (resident, --vram-target 23000)	18.93	31.88	1.68×	0.616
Gemma-4 31B (spilled, --vram-target 22000)	11.12	8.42	0.76× (loss)	0.605

The 3090's smaller absolute base tps leaves more headroom to hide draft cost → higher multipliers than bs2 (E4B 2.53×, 12B 2.43×). The Gemma-4 31B is a 1.68× win when the KV stays resident — the two rows above are the SAME model/drafter/prompt/accept (~0.61); the only difference is VRAM headroom, and it is decisive. This closes two earlier confusions:

1. Coherent-output fix (harness): the 31B is a thinking model (<|turn> framing, writes reasoning_content); the original bench fed it a raw RULER haystack via /completion with ignore_eos:true + greedy + no instruction (the §3.0a/bug-710 "measure on coherent output" violation), collapsing decode into a </u>×256 loop and a meaningless flat 11.14→11.28 / accept 0.926. Fixed by routing sp-mtp-family through /v1/chat/completions (chat template applied; chat response still carries the full timings incl. draft_n/draft_n_accepted).
2. "MTP slower on 3090" (config, NOT a bug): at the harness default --vram-target 22000 the drafter (0.51 GB) + its dual-ctx shared-KV path squeeze the resident-KV window to 0 MiB budget, so the whole KV streams from host over PCIe Gen3 (6.5 GB/s) — and with n_max=3 the drafter re-reads that spilled KV per draft step, so each cycle pays ~4 host-KV streams to produce ~1.8 accepted tokens (2× the autoregressive KV traffic) → net loss. Lift the cap 1 GB to 23000 and the KV fits resident (host tail 62 MiB, negligible): base decode 11→18.93 and MTP flips to 31.88 tps (+68%). So the 22000 default is simply too conservative for the 31B on a 24 GB card.

The GGUF/quant/template were never at fault (byte-identical to canonical bartowski/google_gemma-4-31B-it-GGUF; <|turn> is the genuine Gemma-4 turn format — vocab has no <start_of_turn>). On bs2 (RTX 6000, roomy VRAM, ctx 32768) the same pair is 2.03×; the 3090's 1.68× is the same win on a tighter card. bug-1116.

§4c niah ladder re-run (bs2, A4B-128e Q4_K_M native, DCA-on chunk 32768, q8_0/q8_0 KV, short3-niah.sh): niah_single_1 = 100.0 at 16k / 32k / 64k / 128k — clean retrieval at every length on the fresh binary.

3c. 0.10.3 restore (#423) — --spec-type draft-assistant, patch 0079

The assistant drafter is restored onto the llamafile 0.10.3 base as the first-class --spec-type draft-assistant (patch 0079-gemma-assistant-mtp), coexisting with upstream-native NextN (--spec-type draft-mtp) — distinct enums (COMMON_SPECULATIVE_TYPE_MTP vs …_DRAFT_MTP) and graph types (LLM_GRAPH_TYPE_MTP vs …_DECODER_MTP). Gemma-4 is iSWA with no NextN head, so the assistant is its ONLY MTP path — which is why the bump restores rather than drops it. Live boot-smoke on the rebuilt CUDA host binary (A4B v6-coder Q4_K_M target + 26B-A4B assistant Q8_0, solidpc RTX 3090, .opencoti/0079-boot-smoke.sh):

• server loads the assistant INTO the target (MTP assistant loaded into target) and the spec engine selects draft-assistant, not the draft-simple fallback that segfaulted (bug-517);
• decodes coherently and prefix-stable vs spec-off: 405-char byte-identical greedy prefix, then the expected benign FP-tie paraphrase split (verify-batch matmul reduction order ≠ single-token decode — the bug-263/270 artifact; free-running greedy is never the gate, §3a logit-equiv is);
• assistant actively drafts: 238/545 accepted over a 512-tok run (0.44; 0.83 over the first 96 tok).

Throughput here is NOT the §3b figure — at 512 tokens the per-step draft overhead is unamortized (on 47 vs off 78 tok/s); the representative win is RULER@32k (#414 1.46×; re-measured 1.57× / +57% after the #484 tok_embd→OUTPUT-buft re-claim — off 56.6 → on 88.8 tok/s, 87% accept; the 0.10.3 bump had dropped the head to the CPU buft, bug-540, restored via the additive create_tensor_output_head helper). The §5 composition cells (turbo / DCA / PolyKV / POSITION_WINDOW × assistant-MTP) remain the locked logit-equiv gate (#461), now unblocked by this restore.

3d. Upstream-alignment (#476) — adopt #23398/#24282 data-plane, keep our control-plane

Upstream merged Gemma-4 MTP (PR #23398 = b9549, + E-series #24282). To make a future llamafile bump onto a #23398 base a mechanical reconcile (clean delete of the identity layer), 0079 now carries upstream's data-plane identity while keeping the opencoti control-plane engine. Decision + rejection rationale: docs/protocols/UPSTREAM_SYNC.md (the 0079 block) and LLAMAFILE_UPGRADE.md G34.

• Adopted (data-plane): arch gemma4-assistant (hyphen); tensors nextn.pre_projection / nextn.post_projection + E-series masked_embd_centroids / masked_embd_ordering (bare, no .weight); KV %s.embedding_length_out + %s.nextn_predict_layers; matching internal C++ symbols (LLM_TENSOR_NEXTN_PROJ_*, LLM_TENSOR_MASKED_EMBD_*, hparams n_embd_out_impl). Drafter GGUFs re-formatted in place (Q8_0 bytes verbatim) via .opencoti/gguf-upstream-rename.py.
• Rejected — ctx_other: upstream's assistant runs a second llama_context aliasing the target KV (assumes vanilla single-owner/contiguous/unquantized KV; 0% accept under quantized KV upstream). It would break all five gate features (PolyKV / rolling-KV / M2 split / turbo / DCA), so we keep our single-context build_attn_mtp cross-read + ordered head + tok_embd→OUTPUT buft + fused N-step + --spec-type draft-assistant. ⚠ SUPERSEDED by §3e (2026-07): the single-context facade never reached upstream accept parity (bug-858), so the ctx_other execution model WAS ported after all (dual-ctx, OPENCOTI_MTP_DUAL_CTX=1) — with the read-only shared-cache guards extended to opencoti's own KV machinery (bug-2102).

Inert-rename proof (A4B v6-coder Q4_K_M, solidpc, canonical gemma4-assistant drafter): real_frac = 0 (s4-p3-logit-equiv.sh) — same as §3a, the rename changed no logits; boot-smoke routes draft-assistant, 44% accept, coherent. The 0079 patch is surgically confined: 16/23 file-stanzas byte-identical to the pre-rename 0079; only the 7-file data-plane surface changed. Chain proof (G33/G35): full 0001..0079 strict-applies from pristine (0 rejects), final tree == live byte-identical across 2941 source files, reverse round-trips. All five drafters re-formatted (E2B/E4B ordered head, 12B/31B/A4B dense) to the canonical gemma4-assistant identity and verified byte-faithful (per-tensor sha256 source==output, arch flipped, embedding_length_out + nextn_predict_layers present, no residual mtp.*); the transitional .upstream.gguf artifacts were renamed to the canonical <name>.gguf and the old underscore-arch sources removed. The full KV-composition matrix on the renamed engine is FOLLOW-ON (#461).

3e. bug-858 dual-ctx port + the two root causes — FIXED (2026-07-05, bs2, native A4B pair)

The #607 wholesale port runs the assistant as upstream does — a real second llama_context (ctx_dft, gated by OPENCOTI_MTP_DUAL_CTX=1) sharing the target's KV cells — reversing the §3d ctx_other rejection. Getting it to parity surfaced two independent defects, both ours:

1. bug-2101 — llm_graph_input_attn_kv_iswa::can_reuse base drift (llama-graph.cpp). The llamafile-0.10.3 base nested the kq-mask width check under if (self_k_idxs && …->buffer); the KV-less drafter's k/v_idxs are dead nodes → check skipped → the boot reserve graph (full-cache-width mask) was silently reused for every draft decode; set_input filled only n_kv columns, the rest read scheduler-pool garbage. Signature: accept collapsed with prompt length (0.83 @114 tok → 0.00 @1314; upstream flat). Fix = upstream-verbatim independent if-blocks (b9859 restructured exactly when KV-less graphs were introduced).
2. bug-2102 — opencoti lazy-clear on the aliased shared cache (llama-kv-cache.cpp). The F4-M3 residency calls ensure_cleared()/shrink_if_idle() in init_batch ran UNGUARDED on the draft cache, whose layers alias the target's k/v tensors (layer_share) with a zero n_cells_cleared watermark — the drafter's first decode memset the head of the target's live KV. Signature: deterministic dual-vs-plain trajectory flips at 4–17-logit margins, persisting with p_min=1.01 (zero drafts); isolated by a 2-factor probe (tap-on + drafter-decodes-skipped == plain byte-identical). Fix = if (other) return; read-only guards in both (upstream has neither function). Rule extracted: after adding cache sharing, EVERY mutating member needs the guard — including fork-specific ones upstream doesn't have.

(Also reverted an earlier fattn.cu Q->ne[1]<=2 VEC experiment back to upstream ==1 — its apparent win had been bug-2102's masked effect.)

Post-fix gates (native pair: google 26B-A4B Q4_K_M + HF-canonical assistant Q8_0, bs2, vs b9859):

• Accept (9-prompt compare, greedy 200-tok): n1 0.862 vs 0.876 (parity; 8/9 prompts within noise), n2 0.800 vs 0.792 (above), n3 0.679 vs 0.736 (#609-class target numerics — characterized and closed below; recommend n2 for A4B).
• Accept vs prompt length (filler×k): ours flat at/above upstream at every length (0.85/0.83/0.92/0.88/0.92/0.92 vs 0.87/0.83/0.91/0.88/0.87/0.87 at 34→2594 tok) — the bug-2101 cliff is gone.
• Losslessness (dual==plain, temp 0): now matches upstream's own flip profile — n1 fib/fr byte-identical end-to-end; the two residual flip prompts (clamp@43, forces@126) are positions where ours-plain also differs from up-plain (pre-existing #495 base numerics; upstream itself flips clamp@96). n3: 3/4 == upstream's 3/4.
• Throughput (mtp-matrix, 400-tok): base 204 vs 190 (ours +7%); MTP-on n2 314.7 vs 319.4 (−1.4%), n1 261 vs 288. Full matrix incl. 31B / Qwen-35B / Qwen-27B in three-way-tps.md §2026-07-05.

#609 depth-3 rung: CLOSED (bug-2103, 2026-07-05). The Qwen-35B n3 decode collapse was TinyBLAS's mmvf ne11<=3 cap dropping the 4-token verify batch's f32 GEMVs into a 1-block GemmEx; post-fix n3 is monotone (225→237→247) and ours' best Qwen rung. A4B n3 accept recovered to 0.784 cell / 0.688 9-prompt aggregate vs upstream 0.736 — the residual −4.8pp is spread thinly across prompts (#495-class target numerics, not depth machinery). 2026-07-05 final #614-binary re-verification (all four pairs — A4B + 31B assistant, Qwen 35B + 27B NextN): decode·accept unchanged within noise (A4B n1/n2/n3 257·0.975 / 302·0.845 / 329·0.784; 31B 100·0.849 / 119·0.791; q35 225·0.858 / 237·0.752 / 247·0.662; q27 99·0.752 / 105·0.626), and the 9-prompt aggregates hold at n1 0.856 vs 0.876, n2 0.784 vs 0.792, n3 0.688 vs 0.736 (per-prompt table in three-way-tps.md). Recommended depth: n2 on A4B/31B assistant, n3 on Qwen NextN. Historic diagnostic ladder: buglog bug-2101/2102/2103; probes at bs2:/srv/ml/opencoti-c1/gate{4,5,10,13,14,15,16,18}.sh, matrix at mtp614-out/. ⚠ The decode-tps columns above are SUPERSEDED by §3h (#675, 2026-07-14) — every ours spec cell gained +4–20% from the spec-clone-cheap fix; accepts are unchanged (bit-identical).

---

3f. #611 (2026-07-05): single-context facade REMOVED — dual-ctx is the only assistant path

The retired single-context in-target engine (llama_decode_mtp facade, build_attn_mtp cross-read, llm_build_gemma4_mtp, common_speculative_impl_draft_assistant, async worker) is deleted (patch 0096, ~1600 lines); the OPENCOTI_MTP_DUAL_CTX env gate is gone and --spec-type draft-assistant now ALWAYS runs the §3e dual context (server creates ctx_dft with ctx_other=target; CLI surface unchanged). Gate on the #611 binary (bs2, GPU0, no env): A4B n2 305.4 tps / 0.849 accept (== §3e reference), q35 NextN n3 248.8 / 0.662.

bug-2110 — RESOLVED-NOT-A-BUG (harness artifact, same day). The tps-gate harness showed accept 0.849→0.635 at --parallel 2 --kv-unified, but its prompt (160× repeated filler, temp 0, ignore_eos) drives the model into degenerate repetition loops; par2 merely shifts greedy decode onto a different garbage trajectory (#495-class knife-edge), and accept-on-garbage swings wildly between trajectories (upstream b9859 flips the same way on that harness). On the REAL 9-prompt mtp-bench (/srv/ml/opencoti-c1/2110-real/): ours par1 0.785 vs par2 0.783 aggregate (max per-prompt delta 2pp), upstream 0.793 vs 0.792 — the dual-ctx assistant HOLDS accept at --parallel 2 --kv-unified, within 1pp of upstream at both. PolyKV × assistant-MTP composes cleanly. Lesson (cerebrum Do-Not-Repeat): never read accept off the repetitive-filler tps harness.

3g. WS2 turbo-V in-register × MTP-on — the ship-gate closer (#629, A4B, bs2)

WS2 (#629) reads a rotated turboN V-cache in-register inside the multichunk DCA fused flash-attn kernel (killing the per-forward whole-cache cpy_turboN_f16 cast; n_q≤16 gate; DEFAULT-ON; full decode-tps matrix in context.md §5x). The remaining question for MTP was whether the in-register path stays lossless through the draft-assistant verify batch (n_q > 1). It does — airtight, by measurement not inference:

• Determinism floor (MTP-on, WS2=0 twice): byte-IDENTICAL. The WS2=0 reference path is deterministic, so any WS2=0-vs-1 diff is real signal, not sampling noise.
• n_q=1 teacher-forced logit-equiv (no MTP): real_frac 0.0, mean_tv 0.00041, ffa 1.0 → logit-lossless on the single-token decode.
• n_q>1 teacher-forced logit-equiv (MTP draft-assistant verify batch, n_q = draft+1): real_frac 0.0, mean_tv 2e-05, ffa 1.0 → logit-lossless in the multi-token verify batch. This is the airtight closer — the verify batch is exactly where an n_q-gated in-register path could have diverged, and it does not (measurement, not inference).
• Free-running divergence (MTP-on WS2=0 vs WS2=1): forks at char ~82 (and the n_q=1 no-MTP arm at char ~486), both coherent / greedy / same GGUF → a pre-existing #495-class base FP-tie autoregressive fork, NOT a WS2 defect. Accept parity: 90/90.
• Compose: WS2 default-on in-register (+89% decode, §5x A4B turbo2_0) composes with the MTP draft-assistant path; both are lossless and stack.

Ship-gate: WS2 × MTP is closed — in-register turbo-V is logit-lossless through the verify batch, accept holds at parity, and the only free-running divergence is the known base FP-tie fork. Cross-ref: context.md §5x (WS2 decode-tps matrix + caveats).

3h. #675 (2026-07-14): the last per-cycle host gap — bug-2180 spec-clone under cosmo dlmalloc (patch 0128)

After #609/#614, Qwen 35B NextN still trailed b9859 12% at every spec depth with base decode at parity — a constant per-cycle cost, invisible to nsys/GPU profiling. Per-phase wall-clock timers (OPENCOTI_SPEC_PROF, temporary) in BOTH servers produced a phase-parity table: draft, verify, and process each at parity; the whole gap sat in the process→sample seam. A direct timer nailed it: common_sampler_clone cost 1.071 ms/cycle on ours vs ~0.07 ms on b9859 — from IDENTICAL source. The clone deep-copies the n_vocab-sized cur scratch (1.8 MiB); under the cosmo dlmalloc runtime every ≥256 KiB allocation is a fresh mmap + page-fault-on-write + munmap, while b9859's glibc reuses the freed block. A runtime-allocator asymmetry, not an algorithmic one (bug-2180).

Fix (spec-clone-cheap, common/sampling.cpp): clone with empty cur/cur_p. Safe because cur is write-only scratch that set_logits() repopulates before any read, and the copied cur_p.data pointed into the SOURCE sampler's buffer anyway (latent upstream aliasing — no consumer can rely on the clone's copy). Restore via move-assign unaffected. Clone 1.071 → 0.005 ms/cycle. Companion fix in the same patch: the mmvf f16/bf16 ampere_mma TinyBLAS lift (ne11≤8, src0_small-guarded) — the bf16 shared-expert gate GEMVs (ffn_gate_inp_shexp) in the 2–4-token verify batch were the dominant residual tinyblasGE cliff after #609's f32 fix. Both fixes are output-identical: accept bit-identical pre/post at every depth.

Gate (bs2 GPU1, binary 05240e8e57c7191e, all diag stripped): q35 base 212.6 = parity (unregressed); n3 282.6 vs b9859 287.4 (−1.7%, was −12%); full same-day matrix — A4B 307.9/353.8/378.3 vs 289.1/321.8/369.1 (ours WINS all depths), q35 265.2/270.0/278.8 vs 261.3/277.7/270.7, 9-prompt decode mean 317.7/361.6/365.7 vs 298.1/320.2/339.8 — tables in three-way-tps.md §2026-07-14. Characterized non-blockers: ~0.15–0.2 ms/cycle pre-draft glue; #495-class accept deficit at depth ≥2 (chained-draft numerics), worth its own investigation only if the 31B/27B −6% matters. Diagnostic lesson (cerebrum): a constant ms/cycle host gap with GPU parity on identical source ⇒ suspect the RUNTIME (allocator mmap threshold), and measure with host wall timers, not nsys windows. Build gotcha hit during cleanup: bug-2181 (ccache+cosmocc fat-object stale .aarch64 sidecar — delete both .o twins

• CCACHE_DISABLE=1, verify the APE with strings).

4. Qwen NextN self-speculation (M6-S5) — all 3 models GATED (correctness + tps)

Status (2026-06-18). All 3 Qwen models gated on BOTH axes. Correctness (real_frac=0, §4a): 9B Q8_0 (K=48, full KV-composition sweep), 27B-Omnimerge Q4_K_M (K=256), 35B-A3B IQ3 MoE (K=256) — the spec path is lossless on every model (recall matches plain on all prompts). Throughput (§4b): dense 9B 1.92× / 27B 1.89× (NextN head survives the Omnimerge merge), MoE 35B-A3B 1.18× (MoE wins less). Fused draft (#454) ≡ sequential (no win — fusing draft-token graph steps buys nothing; the recurrent-snapshot cost #457 targets is a separate, unmeasured residual — not "moot"). CoT-budget finding (bug-560): the needle harness default K=48 spuriously inflates real_frac on reasoning models (plain decode cut off mid-<think>); K≥256 → real_frac=0. Both 256k inter-chunk-DCA cells now PASS real_frac=0 (§4c): Qwen NextN (27B-MTP Q6_K, 2026-06-29) and Gemma assistant-MTP (A4B-it + adapted drafter, 2026-06-30).

Models: Qwen3.5-9B-Q8_0 (qwen35, nextn@32) · Qwen3.6-27B-Omnimerge-v4-Q4_K_M (qwen35, nextn@64) · Qwen3.6-35B-A3B-UD-IQ3_XXS (qwen35moe, nextn@40). Loader fix #622 (W1); drafter graph W2; self-spec + CKPT rollback W3/W3b.

4a. Correctness — logit-equivalence (self-spec vs plain), solidpc

All 3 models DONE — real_frac=0 (9B at K=48; 27B/35B at K=256 — see CoT-budget caveat below). w4-qwen-nextn-logit-equiv.sh (a wrong recurrent rollback would surface here as large real_frac; it does not — the spec path is lossless on every model, recall matches plain on all prompts). The 9B cell also carries the full KV-composition sweep (the Qwen column of §5): base + asymmetric + DCA + q4/q8 + turbo2/3/4/8, all real_frac=0. VERDICT prints FAIL from the mean_tv advisory only — the known empty-top artifact (spec-accepted tokens return top:[]); the decisive signal is real_frac/real_disagreements (bug-263/270), both 0.

model	real_frac	frac_full_agree	draft acceptance	verdict
Qwen3.5-9B (Q8_0)	0.0 (K=48)	1.0	0.79–0.94	PASS (real_frac; TV-FAIL=empty-top artifact)
Qwen3.6-27B-Omnimerge (Q4_K_M)	0.0 (K=256)	0.875	0.685 (engaged, §4b)	PASS — K=48 showed 0.125–0.167, a CoT-budget artifact (bug-560), not a spec defect
Qwen3.6-35B-A3B (IQ3)	0.0 (K=256)	0.75	0.705 (engaged, §4b)	PASS — both recall 8/8; 2 benign divs (1 near-tie pos-7, 1 post-answer tail pos-193)

CoT-budget caveat (bug-560). The needle harness default K=n-predict=48 is too short for a reasoning/CoT model. The 27B-Omnimerge emits <think>…</think> before answering; at K=48 a benign pos-3 near-tie (tok 198→271 = "keep thinking" vs "close think+answer", margin 0.02–0.25 nats << tie_margin 0.7, top1_mae=0) tips plain decode toward verbose CoT, which then runs out of budget before emitting the needle → spurious ref_recall=False while the spec path answered correctly (cand_recall=True). That scored as real_frac 0.125 (n=8) / 0.167 (n=24) — but every divergence was cand-better, never cand-worse. At K=256 both paths recall 8/8 → real_frac=0.0: the spec path is lossless (the flip is the documented batch-vs-sequential FP micro-bias, surfacing on Q4_K_M where the Q8_0 9B had none). Run reasoning-model cells with K≥256.

4b. Throughput — off vs on, solidpc (ctx 8192, fixed decode prompt, --temp 0, 2026-06-18)

All 3 Qwen models DONE — self-spec is a decode win on every one; qwen-nextn-tps-confirm.sh (tps from server timings.predicted_per_second, bug-470). The win is large on the dense models (~1.9×) and small on the MoE (1.18×) — see the MoE note below.

model	arch	off gen tok/s	on gen tok/s	speedup×	draft acceptance
Qwen3.5-9B (Q8_0)	dense	44.2	84.7	1.92×	0.76–0.80
Qwen3.6-27B-Omnimerge (Q4_K_M)	dense	22.8	43.0	1.89×	0.685
Qwen3.6-35B-A3B (IQ3_XXS)	MoE (A3B)	76.8	90.9	1.18×	0.705

9B is two fixed prompts (off 44.23/44.09 → on 82.80/86.5 tok/s, 1.87×/1.96× @ accept 0.757/0.795); the table reports their mean. 27B confirms the win holds on a Q4_K_M merge whose NextN head survived the Omnimerge (draft engaged at 0.685).

MoE self-spec wins less. The 35B-A3B is a MoE with only ~3B active params, so its baseline decode is already fast (76.8 tok/s off, vs 44/22.8 for the dense 9B/27B) and memory-bandwidth-light per token. Speculation trades a fixed draft+verify overhead for skipped target steps; when each target step is already cheap (few active params) that overhead amortizes far less, so the net is 1.18× rather than ~1.9× — despite comparable acceptance (0.705). Self-spec still helps MoE, just modestly; the headline win is a dense-model effect.

Fused draft (#454) ≡ sequential — no win. OPENCOTI_MTP_FUSED=1 (one fused N-step graph launch) is byte-identical to the per-step sequential path and no faster (9B: seq 82.80 vs fused 83.15 tok/s = 1.004×; identical acceptance 0.75652 and identical per-iteration drafted-token NDJSON, w6b-…-equiv.sh). Fusing the draft-token graph steps buys nothing — the native drafter's per-step graph work is already minimal — so #454 lands as "sequential is optimal".

#454 ≠ #457 (don't conflate the two costs). #454 fused the draft-token graph steps (measured: no win). #457 targets a different cost: the hybrid backbone is recurrent (gated delta-net), so on a partial draft accept the server would restore a PARTIAL_ONLY recurrent-state checkpoint and re-decode the accepted tokens to re-advance past the rejected AR draft (server spec_ckpt, server-context.cpp ~3578-3606). PR#22673's ring-buffer rewind would make that snapshot/rollback O(1).

#457 PROFILED (2026-06-18, Qwen3.5-9B Q8_0, solidpc, .opencoti/m457-ckpt-profile.sh) → re-decode cost MEASURED = 0; PR#22673 port NOT justified at shipped config. Driving the binary's own LLAMA_TRACE=1 instrumentation over a 512-token decode (71.2 tps):

metric	value
spec iterations	183
partial accepts (accepted < drafted)	108 (59%) — dist: 40×0/3, 31×1/3, 37×2/3, 74×3/3
(restore checkpoint) re-decode events	0
draft acceptance	0.60 (328/547)
#457 speedup ceiling (1+P)	1.000×

The decisive log line is common_context_can_seq_rm: the context supports bounded partial sequence removal: upstream 0.10.3's hybrid context natively partial-rewinds the recurrent state up to a bound that covers the default n_max=3 draft, so all 108 partial rollbacks were handled by seq_rm and the checkpoint-restore/re-decode branch (use_ckpt_*, gated on ctx_*_seq_rm_type == FULL or rollback > llama_n_rs_seq) was never entered. The snapshot is also skipped (same gate), and even if taken it is an ON_DEVICE D2D copy of recurrent state (~µs). So the ~1.9× win has no re-decode residual to absorb — the rollback is already effectively O(1). Caveat (the only path that would revive #457): a draft width large enough that n_rollback > llama_n_rs_seq (e.g. raising --draft-max well past the bound) would start entering the checkpoint path; at that point re-profile before porting. Not a #462 blocker; closed.

The earlier "0.75× W6 regression" was a stale-token artifact (retired --spec-type mtp on the older binary → 32.46 tok/s); draft-mtp on the current binary gives the wins above (bug-557).

R-series closure — the "3-3.5×" headline was architecture-capped at the measured ~1.9×; #452/#453 are subsumed (2026-06-30). The Qwen NextN self-spec line was scoped (#451) with an aspirational 3-3.5× target; the delivered, dual-axis-gated ceiling is 1.92× (9B) / 1.89× (27B) / 1.18× (MoE 35B) — §4b above, post-#457. The gap is structural, not a missing optimization: the hybrid gated-delta-net backbone's verify step bounds the win, the fused-graph path is byte-identical-and-no-faster (#454), and the checkpoint/re-decode cost #457 would have absorbed is already 0 (native seq_rm partial-rewind covers the default draft width — re-decode events measured = 0, (1+P) ceiling = 1.000×). #452's nominal deliverable (DCA composing in the MTP block) is done — Qwen NextN 256k inter-chunk DCA-on PASSES real_frac=0 (§4c). #453's generalize/build/eval/ship is done — all three archs (qwen35 / qwen35moe) gated in §4b and shipped via 0079. No remaining work pushes 1.9× → 3.5× on this design; chasing it would need a fundamentally different (non-recurrent) drafter, not a continuation of this line. Closed as delivered-at-1.9×.

4c. MTP + DCA-on interaction (256k) — Qwen NextN PASS + Gemma assistant PASS (bs2, 2026-06-29/30)

Qwen NextN self-spec — PASS (real_frac=0, bs2 GPU1, 2026-06-29). Model: Qwen3.6-27B-Omnimerge-v4-Q6_K from the dedicated MTP-quant repo (ManniX-ITA/Qwen3.6-27B-Omnimerge-v4-MTP-GGUF, 22.4 GB, carries the 4 blk.64.nextn.* tensors — the plain …-v4-GGUF Q6_K has no NextN head and fails MTP load). Harness w4-dca256k-bs2.sh: -c 262144, --dca on --dca-chunk-size 32768 (8 chunks → genuine inter-chunk DCA on this native-256k model), ref = plain target, cand = --spec-type draft-mtp, both DCA-on so the comparison isolates MTP composition over DCA-laid-out KV + the W3b CKPT rollback. Verdict: real_frac=0.0, real_disagreements=0, frac_full_agree=1.0 → NextN self-spec is logit-equivalent to plain under inter-chunk DCA at 256k. (mean_tv_advisory=0.458 is the known empty-top-logprobs artifact for spec-accepted tokens — advisory only; real_frac is the gate, per [[mtp-s4-p3]].) Caveat: 1 of 2 ~228k prompts overflowed n_ctx_train; the compare ran over the 1 clean prompt × 64 decode positions (clean real_frac=0, sufficient for the composition gate which §4a + the 9B sweep already established mechanistically).

Gemma assistant-MTP — PASS (real_frac=0, bs2 GPU1, 2026-06-30, 2/2 non-empty). Target = Gemma-4 A4B-it (gemma-4-A4B-62e-…-it-Q6_K, arch gemma4/iSWA, native 262144); drafter = the adapted gemma4-assistant (regenerated on bs2 via gguf-upstream-rename.py from the stale underscore ollama_staging copy — the binary rejects the old gemma4_assistant underscore arch). cand adds --mtp-head <drafter> --spec-type draft-assistant -ngld 99; both boots --dca on --dca-chunk-size 32768. Verdict real_frac=0.0, real_disagreements=0, frac_full_agree=1.0 → assistant-MTP verify composes logit-equivalently with DCA-laid-out KV at 256k. (real_frac is robust to drafter/target quality — verify always uses target logits — so it isolates the MTP+DCA composition, not acceptance.)

---

5. Composition matrix — every KV development × MTP (the locked two-engine gate, #461)

MTP is TWO engines (§1) and both must compose with every other KV feature; where an engine is limited we extend it (never "N/A" — user directive 2026-06-17). Rows = features; cols = the engine that carries each model. The assistant-drafter restore (#423) is DONE — the engine boots, loads into the target, and drafts on 0.10.3 (§3c). The gemma gate scripts' mtp→draft-assistant refresh is done (#473); 9 of the Gemma cells PASS real_frac=0 (§5b, 2026-06-17), DCA composes, PolyKV composes under --kv-unified (#485, corrects the earlier bug-550 misdiagnosis). The last Gemma-side item — POSITION_WINDOW-at-overflow at long ctx — is architecturally N/A, not open: --kv-residency-mode window is ineligible for Gemma's iSWA caches (see the POSITION_WINDOW row of the composition matrix below); the cell stands on logit-equivalence.

feature	NextN-MTP (Qwen, --spec-type draft-mtp)	assistant-MTP (Gemma, --spec-type draft-assistant, 0079)
scalar f16 / q8_0 / q4_0	PASS real_frac=0 (GAP-1)	PASS real_frac=0 — f16 (#474) + q8_0/q4_0 (gemma-mtp-compose.sh, A4B v6-coder, 2026-06-17)
asymmetric q8_0/q4_0	PASS real_frac=0 (GAP-2)	PASS real_frac=0 (2026-06-17; V dequant-on-lift under the draft ctx)
turbo2/3/4/8 (fused FA-VEC)	PASS real_frac=0 (GAP-3)	PASS real_frac=0 all 4 tiers (2026-06-17; InnerQ tier-family scale, hd-map 2048:256,1024:512)
DCA-on INTRA (ctx ≤ n_ctx_train)	PASS real_frac=0	composes real_frac=0 (--dca on --dca-chunk-size 1024, 4 chunks @4096; no boot banner — genuine inter-chunk = 256k bs2)
PolyKV (SharedKVPool, --parallel ≥2)	PASS — composes (§5a)	PASS — composes under --kv-unified (which PolyKV forces); boot guard still rejects non-unified multi-slot. Re-proven on the #611 dual-ctx path with the real 9-prompt bench: accept 0.783 @ --parallel 2 --kv-unified vs 0.785 single-slot (upstream 0.792/0.793) — bug-2110 investigation, §3f
rolling-KV POSITION_WINDOW (iSWA)	n/a — Qwen non-iSWA	PASS on correctness real_frac=0 (logit-inert, §5b) + forward path proven by rolling-kv-equiv-gate + #409. 256k engagement is architecturally N/A on the post-bump binary (bs2, 2026-06-30): --kv-residency-mode window is ineligible for Gemma's iSWA caches — the only windowable cache is the SWA one (swa_type=1, bounded 1536 cells → never overflows), and window-mode is disqualified by swa_type=1/v_trans=0/offload=1 (llama_kv_cache: … ineligible … falling back to head/resident). The 2-KV-head global cache spills on the head-axis path (M2/#485), not position-window. So this cell stands on logit-equivalence; "engage POSITION_WINDOW at real 256k overflow" is not exercisable for Gemma without a binary-level eligibility change (not pursued — composition correctness already established)
256k stream / inter-chunk DCA	PASS real_frac=0 (27B-MTP Q6_K, chunk 32768/8 chunks @256k, #439, 2026-06-29 §4c)	PASS real_frac=0 (A4B-it + adapted assistant drafter, chunk 32768 @256k, #431, 2026-06-30 §4c)

§5b — Gemma assistant-MTP composition (solidpc, A4B v6-coder Q4_K_M, 2026-06-17). Eight cells PASS (real_frac=0, gemma-mtp-compose.sh → s4-p3-logit-equiv.sh with KVARGS/DRAFT_KVARGS hooks, #473): scalar f16/q8_0/q4_0, asymmetric q8_0/q4_0, turbo2/turbo3/turbo4/turbo8. ref and cand BOTH carry the feature flags so the feature's own accuracy cancels and real_frac isolates the draft/verify path; the renamed binary loads the canonical gemma-4-…-assistant-Q8_0.gguf drafter (post-reformat) under each KV type and routes draft-assistant. DCA-on (chunk 1024, 4 chunks) composes (real_frac=0) but has no boot banner and is positionally inert at ctx ≤ n_ctx_train (262144) — the decisive inter-chunk DCA test is 256k/bs2. POSITION_WINDOW is logit-inert (real_frac=0) yet the residency planner keeps the tiny 2-KV-head global iSWA cache fully resident at 4096 even under --vram-target 1/window (the historical --vram-target 1100+auto that windowed 256/4096 no longer forces overflow on the post-bump binary) — so its real regime is long-context overflow (256k). An engagement guard in the driver (MARK[] greps the cand server.log for the feature banner) prevents a silently-disengaged feature from masquerading as a vacuous f16 pass.

PolyKV × assistant-MTP — COMPOSES under --kv-unified (#485; corrects the earlier bug-550 misdiagnosis). gemma-polykv-probe.sh boots the full PolyKV combo (--parallel 2 --kv-unified --no-cache-idle-slots -ctk/-ctv q8_0 --mtp-head … --spec-type draft-assistant) → boots, routes draft-assistant, real_frac=0 + ~93% draft acceptance across the shared-prefix slots. PolyKV (SharedKVPool) forces --kv-unified, so the PolyKV config lands on the supported path by construction.

What the earlier session got wrong (bug-550). The first probe reported "BOOT FAIL / --parallel 1 only". That was a probe false-positive: the readiness loop grepped a bare failed to, which matched the benign, caught warning [spec] failed to measure draft model memory (the spec-fit pre-measure tries to build a standalone draft llama_context; gemma4-assistant correctly refuses to be a primary -m model, the warning is logged, and the server continues). The server was in fact booting fine. Two fixes folded into 0079: (a) skip the impossible standalone fit-measure for the assistant engine (spec_assistant_fit, server-context.cpp:809); (b) tighten the probe grep to genuine fatal signatures.

The one real limit — non-unified multi-slot — is now a clean boot guard, not a crash. --parallel >1 without --kv-unified selects the per-stream-split KV layout (patch 0031). The single-context assistant draft cross-reads the target KV via the shared build_attn_mha (build_attn_mtp → get_k/get_v), and build_attn_mha infers the batch's stream count from the target K tensor's 4th axis — const auto n_stream = k->ne[3] (llama-graph.cpp:2115) — then splits the query's token axis by it (:2117). Under the per-stream split get_k concats one view per covered stream on dim 3, so k->ne[3] = n_streams > 1; but the assistant draft query is single-stream, single-token (q->ne[2] = 1, gemma4_mtp_build_one_step builds [.,1] step tensors, ctx_dft==nullptr). Partitioning a 1-token query by n_streams (1/n_streams → 0) corrupts the stream/token geometry → ggml reshape nelements assert (ggml.c:3754) at the first draft decode. The boot guard (server-context.cpp:940) requires --kv-unified for --parallel >1 and fails clean with an actionable message instead of the mid-request SIGABRT. Under --kv-unified the cache is not per-stream-partitioned (n_stream=1, get_k returns ne[3]=1), the line-2117 split is the identity, and the cross-read matches the draft query exactly — the byte-identical path the engine is proven on. Making the draft itself n_seq-aware (one drafted token per active stream, like Qwen NextN §5a) would lift the non-unified limit too, but is not needed for PolyKV (which is unified) — tracked under the #469 extend-native-MTP family.

Head-axis KV residency × assistant-MTP — COMPOSES on a real 31B (Gemma-4-31B-it Q6_K, bs2 GPU1, 2026-06-30). This is the first 31B-specific MTP-composition cell (the §5b matrix above was A4B-only) and the first over a genuinely spilled head-split KV. headaxis-mtp-31b-bs2.sh boots the official google_gemma-4-31B-it-Q6_K.gguf at 128k with f16 KV and --kv-residency-mode auto --vram-target 7000, forcing the dense global cache to split GPU/host on the head axis (M2/PolyKV residency — the path the ineligible window mode falls back to on Gemma's iSWA caches, cf. §5 POSITION_WINDOW). Engagement is real, not vacuous: GPU1 sits at 33.4 GB (ref) / 33.9 GB (cand) vs ~40 GB for full f16-resident, i.e. ~6 GB of KV is held in pinned host RAM and lifted on read. ref-plain (head-split) vs cand-MTP (head-split + gemma-4-31B-it-assistant.Q8_0 drafter, --spec-type draft-assistant -ngld 99) → real_frac=0, real_disagreements=0, frac_full_agree=1.0 (PASS): the assistant draft/verify path is logit-identical even when half the global KV is dequant-on-lifted from host. (mean_tv_advisory=0.44 is the known empty-top-logprobs artifact for spec-accepted tokens, not a divergence — real_frac is the gate.) Closes the open "did we validate 31B on the head axis?" question: previously 31B was proven with MTP decode/accept (§1/§3) and sparse-V long-ctx (#566), but the head-axis residency × MTP cell had only been run on A4B.

5a. PolyKV × NextN-MTP — COMPOSES (Qwen3.5-9B Q8_0, solidpc, 2026-06-17)

Overturns the plan's GAP-5 "architecturally exclusive (parallel-1)". Native MTP is n_seq-aware by construction (common_speculative_impl_draft_mtp does smpls.resize(n_seq) + backend_chains.assign(n_seq); server-context.cpp:1066 inits spec with n_parallel), so it runs per-slot under --parallel. The "--parallel 1" limit was a property of our retired W2/W3b CKPT self-spec, not the native path. Probe .opencoti/polykv-mtp-probe.sh: --parallel 3 --kv-unified --no-cache-idle-slots --spec-type draft-mtp -ctk/-ctv/-ctkd/-ctvd q8_0.

• boot OK; spec engaged (MTP draft ctx 519 MiB; checkpoints max=32, min-spacing=256).
• 2 agents share a read-only prefix (P=873) via SharedKVPool with draft-mtp on: both recall the needle, isolation holds (A=Alice¬Bob / B=Bob¬Alice), counted-once (each cache_n=873≈P, prompt_n=36≈suffix → prefix entered KV once, seq-shared across 3 slots), hook fired ≥2.
• MTP actively drafting during the shared decodes: slots 1 & 2 draft acceptance = 0.19 (4/21).
• Re-verified 2026-06-18 on the post-#493/#485 binary (DSO a8ddcd82, host 13:11): byte-for-byte identical PASS — boot OK, spec engaged (MTP draft ctx 519 MiB), all Phase-2 asserts green (P=873, both recall needle, isolation holds, counted-once, hook ≥2), MTP drafting+accepting (slots 1 & 2 4/21) during the shared decodes. Confirms #485's two shared server-context.cpp spec hooks are inert for native NextN: spec_assistant_fit skip is gemma4-assistant-specific, and the --kv-unified-when---parallel>1 boot guard is satisfied by construction (SharedKVPool forces --kv-unified). #493's fattn D512-VEC case is unreachable here (Qwen is head_dim 128 ≠ 512). So the #469 composition scope is closed on the current binary; the only NextN remainder is the bs2-blocked 256k-stream / genuine inter-chunk DCA cell (#431/#439).

Flag note: the 0.10.1→0.10.3 bump renamed --no-clear-idle → --no-cache-idle-slots (bug-500; runbook G29). The PolyKV mechanism (0072) is unchanged — only the residency boot flag moved.

---

6. Provenance

• Source markers: // opencoti F5 M6-S4 mtp (gemma), // opencoti F5 M6-S5 nextn-mtp (qwen).
• Patches: 0079-gemma-assistant-mtp (gemma assistant, #423 — restored onto the 0.10.3 base; the pre-bump 0074/0076/0077 captures are RETIRED, #464). Qwen NextN is upstream-native (--spec-type draft-mtp) — no source capture beyond the W1 loader fix. Snapshot-diff (23 files, 0 index lines), reverse+forward byte-identical, registered in docs/protocols/UPSTREAM_SYNC.md.
• Build integrity (bug-460): every server boot aborted in common_params_parser_init (arg.cpp:2200, ggml_type_name(params.cache_type_v)) because the incremental cosmocc make did not track the common.h header dep, leaving a common_params layout split across TUs after W3b added use_checkpoints — the consumer read an uninitialized ggml_type ≥ GGML_TYPE_COUNT. Fixed by a clean host rebuild (rm -rf o/ + OPENCOTI_NO_CCACHE=1 build:llamafile:make). The DSO was never at fault; the q1_0-string proxy was a red herring (Q1_0 has no CUDA kernel, so its name is absent from a correct DSO too). Both /root/ggml-cuda.so paths remain byte-identical (67e3171b…).
• Upstream convergence: gemma assistant MTP restored onto the 0.10.3 bump as 0079 (#423, DONE); qwen CKPT rollback ports PR#19493 (enum names match upstream for clean re-sync).