docs/evaluations/three-way-tps.md

Three-way throughput: opencoti vs stock llamafile vs stock llama.cpp

What. Prefill and decode throughput of the opencoti llamafile fork ("ours") against the two upstreams it descends from — stock Mozilla llamafile 0.10.3 and stock llama.cpp — plus an ours-only speculative/MTP arm. The question is narrow and important: does the soft-fork regress raw tok/s versus upstream?

Harness. perf/llamafile/three-way-tps.sh — fully templated, nothing host-hardcoded (paths/knobs via env; models via a MODELS_MANIFEST file; see three-way-tps.models.example).

Method — and why it is reliable

One measurement path for all three engines, because all three speak the llama-server HTTP API:

• Universal metric = llama-server + POST /completion + the response timings block (prompt_per_second, predicted_per_second, draft_n/draft_n_accepted). This avoids the llama-bench‑vs‑server skew that makes cross-engine numbers meaningless — llamafile ships no llama-bench, only the server, so the server API is the only common ground.
• Prefill over a long prompt. A ~1760-token reference prefix is prepended to every prompt. Prefill tok/s over a ~40-token prompt is dominated by fixed overhead and is not a throughput measurement; the long prefix fixes that.
• Decode long and un-truncated. n_predict=512 with ignore_eos so the model cannot end the sample early. A short (EOS-truncated) generation gives a noisy, unreliable decode number — the first, discarded, run of this bench showed stock llamafile at "115 tok/s" purely because of a truncated 20-token sample; with a forced 512-token decode it measured 141.6.

Two structural facts shape the matrix:

1. Feature applicability differs. DCA and opencoti's Qwen-NextN / Gemma assistant MTP are opencoti-only; stock llama.cpp has neither DCA nor those, stock llamafile 0.10.3 has none. So the 3-way is base-only; DCA / MTP are ours-only deltas (MTP arm below). DCA throughput is inert at 4k ctx (it only engages past the chunk size) → covered in a separate long-ctx sweep, not here.
2. DSO-cache isolation. Stock llamafile is booted with an isolated HOME (STOCKLF_HOME) so it builds/uses its own bundled ggml-cuda.so, never a poisoned ~/.llamafile cache holding ours.

How to run

# copy + edit the manifest for your host (absolute GGUF paths)
cp perf/llamafile/three-way-tps.models.example /tmp/models.txt

MODELS_MANIFEST=/tmp/models.txt \
OURS_BIN=/path/to/opencoti/llamafile \
STOCK_LLAMAFILE=/path/to/llamafile-0.10.3 \
STOCK_LLAMACPP=/path/to/llama.cpp/build/bin/llama-server \
GPU=0 PORT=8272 NPRED=512 \
perf/llamafile/three-way-tps.sh
# results table → $OUT_DIR/summary.txt (default ./three-way-out/)

Any engine whose *_BIN is unset is skipped. The optional MTP_MANIFEST adds the ours-only speculative arm (see the example file for the line format).

Results

Host bs2 (RTX PRO 6000 Blackwell, sm_120, CUDA 13.2, driver 610.43.02), ctx 4096, ~1760-tok prompt, n_predict=512 ignore_eos, temp 0, 2-prompt average. opencoti build DSO 59f7d1bb, stock llamafile v0.10.3, stock llama.cpp /srv/ml/repos/llama.cpp-latest.

Base — 3-way parity

model	ours pp / tg	stock-llamafile pp / tg	stock-llama.cpp pp / tg
Qwen3-8B-Q8_0	13150 / 142.6	12727 / 141.6	13033 / 144.8
Qwen2.5-14B-1M-Q8_0	7711 / 84.0	7562 / 82.9	7524 / 84.7
Qwen3.6-27B-Omnimerge-Q6_K	2819 / 55.9	1469 / 54.0	2779 / 56.2
Gemma-4-A4B-v6-coder-Q4_K_M ⚠ v6 RETIRED (re-run on v7/A4B-128e)	5042 / 201.3	2516 / 130.1	4897 / 196.7
Gemma-4-31B-Q6_K	1444 / 44.6	1438 / 43.6	1473 / 44.7

pp = prefill tok/s, tg = decode tok/s.

Read. Two conclusions:

1. No regression vs upstream llama.cpp. ours lands within ~2% of stock llama.cpp on both prefill and decode across every model — the soft-fork adds no throughput cost, and its CUDA graphs engage (graphs reused observed live).
2. ours beats its own llamafile base on MoE. Stock llamafile 0.10.3 falls badly behind on the MoE models: Gemma-4-A4B decode 130 vs ours 201 (+55%), and both A4B and 27B prefill ~2× ours (2516→5042, 1469→2819). That gap is exactly the opencoti MoE-prefill / fused-MoE / graph work (#555, #554, 0084) — the fork is materially faster than the llamafile it descends from, and matches current llama.cpp. Dense models (14B, 31B) are at parity across all three, as expected.

Ours-only — speculative / MTP arm

model	base decode	MTP-on decode	speedup	accept
Qwen3.6-27B (NextN self-spec)	55.9	100.2	1.79×	0.599
Gemma-4-A4B (assistant-MTP)	201.3	242.1	1.20×	0.722
Gemma-4-31B (assistant-MTP)	44.6	87.7	1.97×	0.689
Qwen3.6-35B-A3B (NextN self-spec)	210.5	230.7 (n=2)	1.10×	0.731

Fresh 0121-binary MTP family re-run (#654, 2026-07-12) — full matrix in mtp.md §3b‴. Both hosts, single-GPU pinned, robust decode-tps method. The A4B row above is the retired v6-coder single-context result; the current dual-ctx engine gives, on bs2 (RTX 6000, ctx 32768): A4B 188.9→320.0 (1.69×, acc 0.903), E2B 270→452 (1.67×), E4B 185→366 (1.97×, acc 1.000 — the #493/bug-567 FA-on verification), 12B 128→266 (2.09×), 31B 42.6→86.6 (2.03×), Qwen-27B 52.6→99.6 (1.89×), Qwen-35B 195→276 (1.41×); on solidPC (RTX 3090, ctx 8192): E4B 2.53×, 12B 2.43×, E2B 2.17×, A4B 1.94×, and — after the harness fix (2026-07-12) — 31B 18.93→31.88 (1.68×, acc 0.616) when the KV stays resident (--vram-target 23000). Two fixes landed here: (1) the 31B is a thinking model and the original bench fed it a raw haystack via /completion with ignore_eos+greedy+no-instruction, collapsing decode into a </u>×256 loop (§3.0a/bug-710) → fixed by routing sp-mtp-family through /v1/chat/completions (chat template applied, timings still carried); (2) at the harness default --vram-target 22000 the drafter squeezes the resident-KV window to 0 MiB → whole KV streams from host over PCIe Gen3 and the n_max=3 drafter re-streams it per step → net loss 11.12→8.42 (0.76×). NOT a bug — pure spill: +1 GB headroom (23000) keeps it resident and MTP flips to +68%. The GGUF is fine (byte-identical to canonical bartowski/google_gemma-4-31B-it-GGUF). On bs2 (roomy VRAM) the same pair is 2.03× — the 3090's 1.68× is the same win on a tighter card. bug-1116.

Gemma MTP is opencoti-only (stock llama.cpp here rejects the drafter with unknown model architecture: 'gemma4-assistant' — that arch exists only in ours via #476), so it cannot be cross-checked against upstream. The Qwen NextN self-spec paths can be, and doing so on 35B-A3B surfaced a real regression.

⚠ Qwen-35B-A3B NextN self-spec — ours regresses vs stock llama.cpp

Same GGUF (UD-Q6_K), same greedy 400-tok decode, --spec-draft-n-max swept:

n_max	ours decode	stock llama.cpp decode	ours accept	stock accept
base	210.5	212.0	—	—
1	222.6 (+5.7%)	254.3 (+20%)	0.847	0.851
2	230.7 (+9.6%)	271.1 (+28%)	0.731	0.795
3	156.7 (−26%)	260.5 (+23%)	0.698	0.684

Base decode is at parity (210.5 vs 212.0) and acceptance is at parity — but ours' self-spec throughput is ~⅓ of stock's, and collapses at n=3. This is not a "fast base can't benefit from MTP" effect (my first read): stock proves the model gains +28% and stays positive at every depth. It is an ours-side spec-decode regression. The signature: ours' CUDA-graph reuse drops 161→128 from n=2→n=3 as decode collapses 231→157, while stock holds +23% at n=3 with a comparable graph count — i.e. ours thrashes graph rebuilds on the variable draft-batch shapes, plus carries a baseline per-step spec overhead even at n=2. Tracked for fix (see buglog bug-858, task #590). The genuine "MTP helps a slow base most" point still holds for the magnitude of the win (27B/31B big, A4B moderate) — but it is not why 35B underperforms here.

✅ SUPERSEDED 2026-07-05 (#609 / bug-2103) — the n=3 collapse and the "cap at 2" recommendation are OBSOLETE. The collapse was the TinyBLAS mmvf ne11≤3 1-block-GEMV cliff (bug-2103), fixed by gating the cap at ne11≤8. On the #614 binary Qwen-35B NextN n3 is now monotone and ours' best rung (see the §2026-07-05 matrix below): n1→n2→n3 rises, n3 no longer regresses. Do not cap --spec-draft-n-max at 2 anymore. The table above is retained as the pre-fix historical record only.

It is not Qwen-specific — ours' whole spec-decode path is behind upstream. Cross-checked against a freshly-built upstream b9859 (the Jun-1 reference predated the Gemma4-assistant merge; b9859 has it) on identical GGUFs/HW:

model (n=2)	ours MTP	upstream b9859 MTP	ours accept	b9859 accept
Qwen-35B-A3B (NextN self-spec)	230.7 (+9.6%)	271.0 (+28%)	0.731	0.770
Gemma-4-A4B (assistant-MTP)	242 (+20%)	265.9 (+32%)	0.722	0.637

On Gemma, ours has higher acceptance (0.722 vs 0.637) yet a smaller speedup — dispositive that the loss is per-step spec-loop overhead, not drafter quality, and that it affects both the NextN self-spec and the external-assistant paths. Parity fix is task #590.

#590 diagnostic (2026-07-02) — gap localized to per-cycle host-sync; acceptance ruled out

Three levers were chased to isolate the 35B-A3B NextN gap; the first three are dead ends, leaving one:

• Acceptance is at parity — not the driver. A fresh side-by-side on the LRU prompt (ours llamafile-592 vs upstream b9859, n_max=2, greedy 400-tok) measured ours accept 0.79–0.80 vs upstream 0.815 (~1.5pp), with identical cycle counts (draft_n 307 vs 303, accepted 244 vs 247) — yet upstream decodes 283 vs ours 244 t/s (+16%). Same acceptance, same number of draft/verify cycles, faster wall ⇒ the gap is per-cycle efficiency, not acceptance. (The 0.731/0.770 cells above were a single noisier run; the gap is real but acceptance is not its cause.) bug-867.
• Input-upload syncs — dead. An async-H2D spike (OPENCOTI_ASYNC_INPUTS) found the per-decode graph inputs live on a non-device buffer type, so set_tensor_async never engages (asserts unguarded). Byte-identical, zero movement. bug-866.
• Graph-thrash / OPENCOTI_MTP_FUSED on NextN — not applicable. The fused knob gates the Gemma-assistant path only (decode_mtp_fused requires model.mtp_assistant + llama_kv_cache_iswa); it is inert on NextN by design. On its own path fused works (default-on, byte-identical: 197.6 stable vs 161–192 sequential). bug-864, bug-866.

Conclusion. Base single-decode is at parity, acceptance is at parity, cycle count is equal — so the +16% is the host-sync/launch overhead of the un-fused NextN per-step draft loop (common_speculative_impl_draft_mtp: per draft token a llama_decode + a llama_get_embeddings_pre_norm_ith host readback). The one remaining validated lever is a fused NextN draft graph — the NextN analogue of decode_mtp_fused (multi-step in-graph argmax+embed chain, one launch + one readback per block). Design is scoped separately; payoff scales with n_max (modest at the n=2 cap, expected to fix the n=3 collapse). bug-867.

Gemma vs Qwen MTP use distinct --spec-type strings — get this right or the draft context fails to build. The Gemma external assistant is --spec-type draft-assistant --mtp-head <gemma4_assistant.gguf> -ngld 99; the Qwen NextN self-spec is --spec-type draft-mtp (no drafter file). An early run of this arm wrongly used the Qwen combo (-md … --spec-type draft-mtp) for the Gemma cells → load_model: [spec] failed to measure draft model memory: failed to create llama_context from model (Gemma has no NextN layers for draft-mtp to attach to). It is not an engine or drafter-format regression: with the correct draft-assistant flags both Gemma cells serve and accept normally, as above (matching the #481 A4B beachhead). The MTP-manifest example in three-way-tps.models.example carries the correct flags + a warning.

2026-07-05 refresh — post bug-858 fixes (ours vs upstream b9859, full MTP matrix)

⚠ MTP-arm + 9-prompt decode numbers below are SUPERSEDED by the 2026-07-14 refresh (post-#675 spec-clone-cheap fix — every ours spec cell gained +4–20%). Base + prefill numbers in this section remain current. Accept columns are identical in both runs.

Re-ran the ours-vs-upstream matrix on bs2 after the two bug-858 root-cause fixes landed (bug-2101 can_reuse reserve-graph reuse + bug-2102 ensure_cleared aliased-KV memset — see mtp.md §3e). Same method as above (ctx 4096, ~1760-tok prefix, n_predict=400 ignore_eos, temp 0, 2-prompt avg); ours = dual-ctx binary 20acc0722ea4f97c (OPENCOTI_MTP_DUAL_CTX=1 on the Gemma cells), upstream = b9859; models per the registry below (A4B = the NATIVE google pair, not retired v6-coder). Harness bs2:/srv/ml/opencoti-c1/mtp-matrix.sh → mtp-matrix-out/summary.txt.

Base — ours vs b9859

model	ours pp / tg	b9859 pp / tg	decode verdict
Gemma-4-26B-A4B-it Q4_K_M (MoE)	4818 7539 (r1 6296 · r2 8782) / 204.0	6316 (r1 5053 · r2 7579¹) / 190.1	ours +7%
Gemma-4-31B-it Q6_K	1582 2302 (r1 1938 · r2 2667) / 45.3	2240 (r1 2245 · r2 2235) / 44.8	parity
Qwen3.6-35B-A3B UD-Q6_K (MoE)	4683 6089 (r1 5917 · r2 6261) / 208.3	6406 (r1 5280 · r2 7531¹) / 212.1	parity (−1.8%)
Qwen3.6-27B-Omnimerge-MTP Q6_K	2820 2858 (r1 2799 · r2 2918) / 55.8	2922 / 56.0	parity

¹ b9859 r2 is prompt-cache-restore inflated (it removed the cache_prompt opt-out); ours' r2 is honest full recompute with a warm checkpoint buffer.

Decode: no regression anywhere; ours now beats b9859 on A4B MoE decode. Prefill (#614, 2026-07-05): the −25–35% gap was NOT upstream movement — it was THREE ours-side defects, all fixed. (1) bug-2104 TinyBLAS small-GEMM cliff: f32 GEMMs (Gemma MoE router ffn_gate_inp [2816×128]; q35 DeltaNet chunk GEMMs) skip MMQ/mmf/mmvf and land in the cuBLAS slot, where TinyBLAS's single 128×64-tile config launches only 8 blocks on a 142-SM GPU (274 µs/call vs cuBLAS cutlass tf32+splitK 8 µs). Fix opencoti-hook: tinyblas-small-gemm (tinyblas.cu): a 32×32-tile/2-warp config of the same deterministic warp2d kernel when the big config would launch <24 blocks (router 274→90.7 µs). (2) bug-2105 checkpoint_min_step 256→8192: our server port carried a stale early-upstream default, taking a SWA/recurrent state checkpoint (full state D2H into a zeroed host buffer, ~250–400 ms on 31B) every 512-token ubatch instead of every 8192 tokens. (3) bug-2107 checkpoint buffer recycle (opencoti-hook: ckpt-buf-noinit, server-context.cpp): each request allocated a fresh GB-class host vector for the snapshot (memset + page-fault storm + cold-page D2H); retired checkpoint buffers are now recycled through a small pool (ckpt_buf_pool), so steady-state requests write into warm pages — no memset, no faults. Measured dead end kept for the record: a non-zeroing allocator WITHOUT the pool made things worse (faults moved into the D2H copy loop; −8–13% all models). Final @1760-tok in the table above; steady-state (r2, warm) ours leads upstream's honest number on every model (31B 2667 vs 2240 = +19%; A4B 8782; q35 6261 vs 5280 cold). MTP accept/tps unchanged (spec rollback uses its own per-cycle spec_ckpt, not pooled). ⚠ Bench caveat: b9859 removed cache_prompt (always caches), so its r2 on identical prompts is inflated +43–50% on MoE (state restore, not compute); cold r1-vs-r1 ours LEADS on both MoE models and is −14% on 31B only for the very first request per slot (cold pool).

MTP arm — ours vs b9859 (decode t/s · accept)

Ours columns re-measured 2026-07-05 on the final #614 binary (tinyblas-small-gemm + checkpoint fixes in). Decode/accept is within run noise of the pre-#614 cells on every row — the #614 fixes are prefill-side and MTP-neutral, as designed. All four documented pairs covered: both Gemma assistant-drafter targets (A4B, 31B) and both Qwen NextN self-spec models (35B-A3B, 27B-omnimerge). Raw cells: bs2 /srv/ml/opencoti-c1/mtp614-out/.

model / config	ours n1	up n1	ours n2	up n2	ours n3	up n3
A4B assistant-MTP (dual-ctx)	257.4 · 0.975	287.7 · 0.939	302.1 · 0.845	319.4 · 0.890	329.1 · 0.784	366.6 · 0.900
31B assistant-MTP (dual-ctx)	—	—	100.3 · 0.849	107.4 · 0.912	118.6 · 0.791	131.6 · 0.881
Qwen 35B-A3B NextN self-spec	224.9 · 0.858	259.8 · 0.882	237.4 · 0.752	278.2 · 0.804	247.1 · 0.662	269.9 · 0.701
Qwen 27B NextN self-spec	—	—	99.1 · 0.752	107.5 · 0.763	104.7 · 0.626	116.2 · 0.654

MTP-config prefill also carries the #614 gains (2-request avg, same cells): A4B 6283–6476, 31B 1965–1968, q35 4915–5012, q27 2327–2348 tok/s.

9-prompt am17an bench — per-prompt accept, A4B native pair (bs2, 2026-07-05, final binary)

mtp-bench.py, 9 prompts × 192 predicted tokens each, greedy; same drafter GGUF both engines. Raw JSONs: bs2 /srv/ml/opencoti-c1/mtp614-out/b9_{ours,up}_n{1,2,3}.json.

prompt	ours n1	up n1	ours n2	up n2	ours n3	up n3
code_python	0.919	0.919	0.911	0.890	0.868	0.891
code_cpp	0.873	0.900	0.808	0.826	0.714	0.788
explain_concept	0.809	0.809	0.669	0.698	0.569	0.623
summarize	0.900	0.919	0.851	0.851	0.759	0.798
qa_factual	0.845	0.919	0.757	0.789	0.681	0.750
translation	0.969	0.969	0.890	0.871	0.736	0.834
creative_short	0.652	0.768	0.651	0.627	0.561	0.574
stepwise_math	0.939	0.891	0.808	0.871	0.718	0.759
long_code_review	0.837	0.809	0.762	0.757	0.653	0.688
aggregate accept	0.856	0.876	0.784	0.792	0.688	0.736
decode t/s (agg)	238	263	274	280	282	293

Read: n1/n2 accept is at parity (−2.0pp / −0.8pp aggregate; per-prompt ours wins 2–3 of 9 at each depth — stepwise_math, long_code_review, code_python/translation at n2). The n3 aggregate gap (−4.8pp) is spread thinly across prompts (worst: translation −9.8pp, code_cpp −7.4pp), consistent with the #495-class target-numerics residual, not a single broken prompt class. Recommended depth unchanged: n2 for A4B (best accept·tps balance on ours), n3 on Qwen NextN.

Read (supersedes the ⚠ sections above for current state):

1. A4B is FIXED. Pre-fix, ours' assistant path collapsed with prompt length (accept → 0.0 at 1.3k tok) and trailed −10pp at a single draft step. Post-fix: accept at/above upstream at n1/n2 (9-prompt am17an-style compare: n1 0.862 vs 0.876, n2 0.800 vs 0.792; flat vs prompt length to 2.6k tok), MTP-on decode 315 vs 319 at n2 (−1.4%), and dual-vs-plain losslessness now matches upstream's own flip profile. Recommended depth on ours: n2.
2. The 35B n3 decode collapse is FIXED (#609, bug-2103, 2026-07-05). Root cause was NOT CUDA-graph churn: node-level nsys (--cuda-graph-trace=node) showed ~38 f32 dot GEMVs/cycle ([K,1]×[K,4], fused-GDN surroundings) falling off mmvf's NVIDIA f32 cap (ne11<=3) into llamafile's TinyBLAS GemmEx — ONE thread block, ~174 µs vs upstream's cuBLAS GEMV ~1.2 µs ≈ 35% of GPU wall. Depth 3's 4-token verify batch is the first rung past the cap, hence n3-only. Fix: GGML_USE_TINYBLAS-gated ne11<=8 in should_use_mmvf (kernel max, upstream-faithful; cuBLAS builds unchanged). Post-fix: n3 247.9 · 0.662 (was 149.3), curve monotone 225→238→248 like upstream's; base 211 vs up 212 (parity). Residual ~8–13% at all spec depths = the pre-existing per-cycle leg, not depth-triggered. n3 is now ours' best rung. A4B n3 re-test with the fixed kernel: decode 312.9→330.2 (+5.5% — A4B's verify batch also had TinyBLAS-routed f32 GEMVs), accept 0.758→0.784. The remaining −12pp vs upstream 0.900 at n3 is the pre-existing #495-class target-numerics residual (knife-edge thinking-prompt class, see the A4B parity note), NOT depth machinery — n2 remains the recommended depth for A4B; n3 is now safe (and fastest) on Qwen NextN.
3. Qwen 27B NextN is at working parity (−7/−9% decode, accept parity) — the 35B n3 collapse is depth-triggered, not NextN-wide.
4. 31B accept trails ~6–10pp at n2/n3 (same adapted drafter GGUF on both engines) — the n3-class residual, milder; still 2.6× base at n3 (117 t/s).

2026-07-14 refresh — post #675 spec-cycle host parity (patch 0128): the MTP gap is CLOSED

Re-ran the FULL ours-vs-b9859 MTP matrix after the two #675 fixes landed (spec-clone-cheap in common_sampler_clone — the 1.07 ms/cycle cosmo-dlmalloc deep-copy that was the entire wall-clock deficit — plus the mmvf f16/bf16 TinyBLAS verify lift; see patch 0128 and mtp.md §3g). Same method/models/prompts as the 07-05 section; both engines re-measured same-day on the same GPU (bs2 GPU1, GPU0 occupied by an unrelated job — mild shared-host contention applies equally to both columns). Ours = binary 05240e8e57c7191e (dual-ctx on Gemma cells), upstream = b9859 (binary unchanged since 07-05). Harness bs2:/srv/ml/opencoti-c1/mtp675.sh → mtp675-out/summary.txt. Accept columns are bit-identical to the 07-05 run on every ours cell (0.975/0.845/0.784 …) — the #675 fixes are pure-latency, output-identical, as designed.

MTP arm — ours vs b9859 (decode t/s · accept), 2026-07-14

model / config	ours n1	up n1	ours n2	up n2	ours n3	up n3
A4B assistant-MTP (dual-ctx)	307.9 · 0.975	289.1 · 0.939	353.8 · 0.845	321.8 · 0.890	378.3 · 0.784	369.1 · 0.900
31B assistant-MTP (dual-ctx)	—	—	105.2 · 0.849	107.3 · 0.912	123.6 · 0.782	131.6 · 0.881
Qwen 35B-A3B NextN self-spec	265.2 · 0.858	261.3 · 0.882	270.0 · 0.752	277.7 · 0.804	278.8 · 0.662	270.7 · 0.701
Qwen 27B NextN self-spec	—	—	102.4 · 0.752	107.5 · 0.763	109.4 · 0.626	116.2 · 0.654

Delta vs the 07-05 (pre-#675) ours column: every spec cell gains +4–20% (A4B n2 302→354, n3 329→378; q35 n1 225→265, n3 247→279; 31B n3 119→124; q27 n3 105→109) — exactly the per-cycle host-glue reclaim, scaling with cycle rate. Upstream columns moved only by run-to-run/GPU1 noise vs 07-05.

Verdict per pair:

1. A4B assistant: ours now BEATS b9859 at every depth (n1 +6.5%, n2 +9.9%, n3 +2.5%) — ours' lower accept at n2/n3 is more than compensated by the cheaper cycle.
2. Qwen 35B NextN: parity-or-better (n1 +1.5%, n2 −2.8%, n3 +3.0% — the former −12% headline gap rung). bug-858 is CLOSED.
3. 31B assistant −2/−6%, 27B NextN −5/−6% at n2/n3 — residual is the #495-class accept deficit at depth (chained-draft numerics; same drafter GGUF both engines), not cycle cost.
4. Recommended depths unchanged: n2–n3 on A4B (all rungs now win), n3 on Qwen NextN.

9-prompt am17an bench — per-prompt accept, A4B native pair (bs2 GPU1, 2026-07-14, 0128 binary)

mtp-bench.py, 9 prompts × 192 predicted tokens each, greedy; same drafter GGUF both engines. Raw JSONs: bs2 /srv/ml/opencoti-c1/mtp675-out/b9_{ours,up}_n{1,2,3}.json.

prompt	ours n1	up n1	ours n2	up n2	ours n3	up n3
code_python	0.919	0.919	0.911	0.890	0.868	0.891
code_cpp	0.873	0.900	0.826	0.826	0.732	0.788
explain_concept	0.809	0.809	0.669	0.698	0.569	0.623
summarize	0.900	0.919	0.851	0.851	0.759	0.798
qa_factual	0.845	0.919	0.757	0.789	0.681	0.750
translation	0.969	0.969	0.890	0.871	0.754	0.834
creative_short	0.652	0.768	0.651	0.627	0.566	0.574
stepwise_math	0.939	0.891	0.808	0.871	0.718	0.759
long_code_review	0.837	0.809	0.762	0.757	0.653	0.688
aggregate accept	0.856	0.876	0.786	0.792	0.692	0.736
decode t/s (mean)	317.7	298.1	361.6	320.2	365.7	339.8

Read: accept is statistically unchanged vs 07-05 (aggregate deltas ≤0.5pp, per-prompt wins/losses in the same places — stepwise_math/long_code_review ours, creative_short/qa_factual upstream; the n3 −4.4pp aggregate residual remains the #495-class target-numerics spread). The new fact is throughput: ours' 9-prompt decode mean now LEADS upstream at every depth (+6.6% n1, +12.9% n2, +7.6% n3) — pre-#675 it trailed at all three (238/274/282 vs 263/280/293 on 07-05). Depth guidance for A4B: n2 and n3 are now effectively tied on ours (361.6 vs 365.7); n2 keeps the higher accept.

Pinned models — exact identity on each host (registry)

Why this table exists. The same model family exists at different quants across solidPC and bs2, and in two cases the same filename holds different bytes. Picking the wrong copy silently benches a different model. Always resolve a model through this registry (sha256) — never by hunting the disks. Authoritative eval host is bs2; solidPC copies are for 3090 / local dev. Captured 2026-07-03.

bs2 — RTX PRO 6000 (authoritative 3-way + MTP eval host), under /srv/ml/models/

role	file · dir	bytes	sha256
Qwen NextN self-spec — bug-858 headline	Qwen3.6-35B-A3B-UD-Q6_K.gguf · gguf/Qwen3.6-35B-A3B-MTP/	30011242784	49935b04ad883c2f3d4da61f65b609d447dad67d0b08453b90abb09a1bb35464
Qwen NextN self-spec — 27B (nextn baked)	Qwen3.6-27B-Omnimerge-v4-Q6_K.gguf · nextn/omnimerge-mtp/	22430999136	eb4532bde36550f7f9d99ffacec265d416d4ad3c77d0b3e657639254df4be3be
Qwen 27B base (NO nextn)	Qwen3.6-27B-Omnimerge-v4-Q6_K.gguf · gguf/Qwen3.6-27B-Omnimerge-v4-GGUF/	22082528832	8d574a27e5034861c45107397d5a179dac3a634f4efacc1b9e2274e36f39c5bc
Qwen 14B-1M base	Qwen2.5-14B-Instruct-1M-Q8_0.gguf · Qwen2.5-14B-Instruct-1M/	15701598272	68ee5c4845c61730a7d769cb31c13f62b2752042111497dcfa80753e04195149
A4B native target	google_gemma-4-26B-A4B-it-Q4_K_M.gguf · gguf/google-a4b-128e/	17035038112	e718536fe9b4bd505b07d44ded8f1595053a5d5407315bccf555ce592f33c140
A4B assistant drafter (native pair)	gemma-4-26B-A4B-it-assistant.Q8_0.gguf · nextn/drafters-hf/26B-A4B/	461767136	5963a2edfb516fbdff499b1d4b7f8158a7a811af7196eaae00be2119188e422e
A4B v6-coder — ⚠ RETIRED 2026-07-03 (deprecated)	gemma-4-A4B-98e-v6-coder-it-Q4_K_M.gguf · variants/v6coder/ (bs2 copies remain)	13239743776	debb603160d278fd7fd33028a0357ccbd06ee4c4bf03addfcffa2329f2f7a951
31B target	google_gemma-4-31B-it-Q6_K.gguf · gguf/gemma-4-31B-it-GGUF/	26726720128	7961268b989e16ec846fc7077e27f78212cc33553688d59066df4491f057f928
31B assistant drafter	gemma-4-31B-it-assistant.Q8_0.gguf · nextn/drafters-adapted/31B/	514687456	ca7820a7289d4c88b3e09626bf9bc8f44bd90ee5b54d33d42049b69d5c349a06
Qwen3-8B-Q8_0	ABSENT on bs2 (present at doc-eval time; removed since)	—	—

solidPC — RTX 3090, under .opencoti/models/ (symlink → spool)

role	file · dir	bytes	sha256
Qwen 8B	Qwen3-8B-Q8_0.gguf · .opencoti/models/	8709518112	408b955510e196121c1c375201744783b5c9a43c7956d73fc78df54c66e883d6
Qwen 14B-1M ⚠ Q6_K (not Q8_0)	Qwen2.5-14B-Instruct-1M-Q6_K.gguf · .opencoti/models/	12124684352	c2faeb8ebe703924668aceab5fd49afaf5cb13efef75dfc3d3796f5c6de1cf52
Qwen 27B ⚠ Q4_K_M (not Q6_K, no nextn)	Qwen3.6-27B-Omnimerge-v4-Q4_K_M.gguf · .opencoti/models/	16810713696	59e5c6fa3d44cb56937001696b6869b4b2cb2fe0643c0c12cd63a0a8221dfb79
Qwen 35B-A3B ⚠ IQ3_XXS (not Q6_K)	Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf · .opencoti/models/	13211155424	9c964e657212fea1f24905dd7b0a89b82fd807d19fab0b41da14251b07b88fbe
A4B native target ✓ == bs2	google_gemma-4-26B-A4B-it-Q4_K_M.gguf · .opencoti/models/	17035038112	e718536fe9b4bd505b07d44ded8f1595053a5d5407315bccf555ce592f33c140
A4B v7-coder (replaces v6; Q4_K_M re-synced from HF 2026-07-14)	gemma-4-A4B-98e-v7-coder-it-{Q4_K_M,IQ4_XS}.gguf · .opencoti/models/	Q4_K_M 13239743904	Q4_K_M 89b4038c… ✓ == HF-pub · IQ4_XS 6048770e… ⚠ still ≠ HF-pub 07c6406b
31B target ⚠ Q4_K_M (not Q6_K)	gemma-4-31B-it-Q4_K_M.gguf · .opencoti/models/	19598488192	35dacd53f9dde83fe20a4596efce2b2f72d2ebb6e4141186533419e9e98bc269
A4B assistant drafter ✓ == bs2/HF	gemma-4-26B-A4B-it-assistant-Q8_0.gguf · .opencoti/models/drafters/	461767136	5963a2edfb516fbdff499b1d4b7f8158a7a811af7196eaae00be2119188e422e
31B assistant drafter ✓ == bs2	gemma-4-31B-it-assistant-Q8_0.gguf · .opencoti/models/drafters/	514687456	ca7820a7289d4c88b3e09626bf9bc8f44bd90ee5b54d33d42049b69d5c349a06
12B assistant drafter ✓ == bs2/HF	gemma-4-12B-it-assistant-Q8_0.gguf · .opencoti/models/drafters/	465109440	223bb1d1e18b28efc196042b7e166321bc0b3e10e8ca0b97f7a88c5abe1654c1
E2B assistant drafter	gemma-4-E2B-it-assistant-Q8_0.gguf · .opencoti/models/drafters/	99423712	78996a998e4773247e789f02716e4fae64c9026c6cce755002d17d9f50882849
E4B assistant drafter	gemma-4-E4B-it-assistant-Q8_0.gguf · .opencoti/models/drafters/	100259296	e2c1426e6552ac7b9e73bf66991cf6aef62580243424860f82746ceb00fbb4cb

⚠ Cross-host alarms — same name ≠ same bytes

• Byte-identical both hosts (safe either): A4B native target e718536f; all 5 gemma4 drafters Q8_0 — HF-canonical on BOTH hosts after the 2026-07-03 re-sync (A4B 5963a2ed, 12B 223bb1d1, 31B ca7820a7, E2B 78996a99, E4B e2c1426e; source of truth = private ManniX-ITA/gemma-4-<size>-it-assistant-GGUF). Only the filename differs cosmetically — solidPC -Q8_0.gguf (dash) vs bs2/HF .Q8_0.gguf (dot); the bytes/hash match, which is what counts.
• Same filename, DIFFERENT bytes — never cross-substitute: A4B v7-coder IQ4_XS (solidPC local 6048770e ≠ HF-published 07c6406b) — re-fetch from HF before any cross-host IQ4_XS eval. (The Q4_K_M mismatch was RESOLVED 2026-07-14: solidPC re-synced from HF to 89b4038c, old 1607dce0 deleted, .sha256 sidecar written. v6-coder RETIRED 2026-07-03: removed on solidPC, deprecated on bs2. The A4B + 12B drafter byte mismatches were RESOLVED same day — solidPC re-synced from HF to 5963a2ed/223bb1d1.)
• Different quant, same family — solidPC holds the LOWER quant; never sub it for a bs2 eval: 35B-A3B (Q6_K vs IQ3_XXS), 27B (Q6_K vs Q4_K_M), 14B-1M (Q8_0 vs Q6_K), 31B (Q6_K vs Q4_K_M).
• One host only: Qwen3-8B-Q8_0 → solidPC only. 35B-A3B-UD-Q6_K, 27B nextn-baked (omnimerge-mtp/), A4B native drafter (drafters-hf/) → bs2 only.
• Coder-variant policy (2026-07-03): v6-coder RETIRED. Coder re-runs use v7-coder (canonical = HF ManniX-ITA/gemma-4-A4B-98e-v7-coder-it-GGUF, Q4_K_M 89b4038c) or the A4B-128e full-Google target (e718536f). The v6-coder bench rows above (Base 3-way + assistant-MTP) are re-run fresh, not reconciled.

bug-858 canonical pair (the two models the both-paths eval MUST use)

• Qwen NextN self-spec → bs2 gguf/Qwen3.6-35B-A3B-MTP/Qwen3.6-35B-A3B-UD-Q6_K.gguf (49935b04…), --spec-type draft-mtp (nextn baked, no drafter). solidPC has no Q6_K equivalent (only IQ3_XXS).
• A4B assistant-MTP → bs2 target gguf/google-a4b-128e/google_gemma-4-26B-A4B-it-Q4_K_M.gguf (e718536f…) + drafter nextn/drafters-hf/26B-A4B/gemma-4-26B-A4B-it-assistant.Q8_0.gguf (5963a2ed…), --spec-type draft-assistant --mtp-head <drafter> -ngld 99. solidPC drafters/gemma-4-26B-A4B-it-assistant-Q8_0.gguf now carries the SAME HF-canonical bytes (5963a2ed, dash-name only) after the 2026-07-03 re-sync — safe to use on either host.

Caveats

• Short-ctx only. Long-context and the DCA sweep are separate evaluations.
• 2-prompt average at temp 0 (deterministic); ignore_eos guarantees the full NPRED decode length so tg is stable. Bump NPRED for tighter numbers.
• Stock llamafile 0.10.3 may LOADFAIL on archs newer than its bundled llama.cpp vintage — captured per-cell, not fatal.