# Three-way throughput: opencoti vs stock llamafile vs stock llama.cpp

**What.** Prefill and decode throughput of the opencoti llamafile fork ("ours")
against the two upstreams it descends from — stock **Mozilla llamafile 0.10.3**
and stock **llama.cpp** — plus an ours-only speculative/MTP arm. The question is
narrow and important: *does the soft-fork regress raw tok/s versus upstream?*

**Harness.** [`perf/llamafile/three-way-tps.sh`](../../perf/llamafile/three-way-tps.sh)
— fully templated, nothing host-hardcoded (paths/knobs via env; models via a
`MODELS_MANIFEST` file; see
[`three-way-tps.models.example`](../../perf/llamafile/three-way-tps.models.example)).

## Method — and why it is reliable

One measurement path for all three engines, because all three speak the
`llama-server` HTTP API:

- **Universal metric** = `llama-server` + `POST /completion` + the response
  `timings` block (`prompt_per_second`, `predicted_per_second`,
  `draft_n`/`draft_n_accepted`). This avoids the llama-bench‑vs‑server skew that
  makes cross-engine numbers meaningless — llamafile ships **no** `llama-bench`,
  only the server, so the server API is the only common ground.
- **Prefill over a long prompt.** A ~1760-token reference prefix is prepended to
  every prompt. Prefill tok/s over a ~40-token prompt is dominated by fixed
  overhead and is *not* a throughput measurement; the long prefix fixes that.
- **Decode long and un-truncated.** `n_predict=512` with **`ignore_eos`** so the
  model cannot end the sample early. A short (EOS-truncated) generation gives a
  noisy, unreliable decode number — the first, discarded, run of this bench
  showed stock llamafile at "115 tok/s" purely because of a truncated 20-token
  sample; with a forced 512-token decode it measured **141.6**.

Two structural facts shape the matrix:

1. **Feature applicability differs.** DCA and opencoti's Qwen-NextN / Gemma
   assistant MTP are opencoti-only; stock llama.cpp has neither DCA nor those,
   stock llamafile 0.10.3 has none. So the **3-way is base-only**; DCA / MTP are
   ours-only deltas (MTP arm below). DCA throughput is *inert at 4k ctx* (it
   only engages past the chunk size) → covered in a separate long-ctx sweep, not
   here.
2. **DSO-cache isolation.** Stock llamafile is booted with an isolated `HOME`
   (`STOCKLF_HOME`) so it builds/uses *its own* bundled `ggml-cuda.so`, never a
   poisoned `~/.llamafile` cache holding ours.

## How to run

```sh
# copy + edit the manifest for your host (absolute GGUF paths)
cp perf/llamafile/three-way-tps.models.example /tmp/models.txt

MODELS_MANIFEST=/tmp/models.txt \
OURS_BIN=/path/to/opencoti/llamafile \
STOCK_LLAMAFILE=/path/to/llamafile-0.10.3 \
STOCK_LLAMACPP=/path/to/llama.cpp/build/bin/llama-server \
GPU=0 PORT=8272 NPRED=512 \
perf/llamafile/three-way-tps.sh
# results table → $OUT_DIR/summary.txt (default ./three-way-out/)
```

Any engine whose `*_BIN` is unset is skipped. The optional `MTP_MANIFEST` adds
the ours-only speculative arm (see the example file for the line format).

## Results

Host **bs2** (RTX PRO 6000 Blackwell, sm_120, CUDA 13.2, driver 610.43.02),
ctx 4096, ~1760-tok prompt, `n_predict=512` `ignore_eos`, temp 0, 2-prompt
average. opencoti build DSO `59f7d1bb`, stock llamafile v0.10.3, stock
llama.cpp `/srv/ml/repos/llama.cpp-latest`.

### Base — 3-way parity

| model | ours pp / tg | stock-llamafile pp / tg | stock-llama.cpp pp / tg |
|---|---|---|---|
| Qwen3-8B-Q8_0 | 13150 / 142.6 | 12727 / 141.6 | 13033 / 144.8 |
| Qwen2.5-14B-1M-Q8_0 | 7711 / 84.0 | 7562 / 82.9 | 7524 / 84.7 |
| Qwen3.6-27B-Omnimerge-Q6_K | 2819 / 55.9 | **1469** / 54.0 | 2779 / 56.2 |
| Gemma-4-A4B-v6-coder-Q4_K_M ⚠ v6 RETIRED (re-run on v7/A4B-128e) | 5042 / **201.3** | **2516 / 130.1** | 4897 / 196.7 |
| Gemma-4-31B-Q6_K | 1444 / 44.6 | 1438 / 43.6 | 1473 / 44.7 |

*pp = prefill tok/s, tg = decode tok/s.*

**Read.** Two conclusions:

1. **No regression vs upstream llama.cpp.** ours lands within **~2%** of stock
   llama.cpp on both prefill and decode across every model — the soft-fork adds
   no throughput cost, and its CUDA graphs engage (`graphs reused` observed
   live).
2. **ours *beats* its own llamafile base on MoE.** Stock llamafile 0.10.3 falls
   badly behind on the MoE models: **Gemma-4-A4B decode 130 vs ours 201
   (+55%)**, and both A4B and 27B **prefill ~2× ours** (2516→5042, 1469→2819).
   That gap is exactly the opencoti MoE-prefill / fused-MoE / graph work
   (#555, #554, 0084) — the fork is materially faster than the llamafile it
   descends from, and matches current llama.cpp. Dense models (14B, 31B) are at
   parity across all three, as expected.

### Ours-only — speculative / MTP arm

| model | base decode | MTP-on decode | speedup | accept |
|---|---|---|---|---|
| Qwen3.6-27B (NextN self-spec) | 55.9 | **100.2** | **1.79×** | 0.599 |
| Gemma-4-A4B (assistant-MTP) | 201.3 | **242.1** | **1.20×** | 0.722 |
| Gemma-4-31B (assistant-MTP) | 44.6 | **87.7** | **1.97×** | 0.689 |
| Qwen3.6-35B-A3B (NextN self-spec) | 210.5 | 230.7 (n=2) | 1.10× | 0.731 |

> **Fresh 0121-binary MTP family re-run (#654, 2026-07-12) — full matrix in
> [mtp.md §3b‴](mtp.md).** Both hosts, single-GPU pinned, robust decode-tps method.
> The A4B row above is the retired v6-coder single-context result; the current dual-ctx
> engine gives, on **bs2 (RTX 6000, ctx 32768):** A4B 188.9→320.0 (**1.69×**, acc 0.903),
> E2B 270→452 (1.67×), E4B 185→366 (**1.97×**, acc **1.000** — the #493/bug-567 FA-on
> verification), 12B 128→266 (2.09×), 31B 42.6→86.6 (2.03×), Qwen-27B 52.6→99.6 (1.89×),
> Qwen-35B 195→276 (1.41×); on **solidPC (RTX 3090, ctx 8192):** E4B **2.53×**, 12B 2.43×,
> E2B 2.17×, A4B 1.94×, and — after the harness fix (2026-07-12) — **31B 18.93→31.88 (1.68×,
> acc 0.616)** *when the KV stays resident* (`--vram-target 23000`). Two fixes landed here:
> (1) the 31B is a **thinking** model and the original bench fed it a raw haystack via
> `/completion` with `ignore_eos`+greedy+no-instruction, collapsing decode into a `</u>`×256
> loop (§3.0a/bug-710) → fixed by routing `sp-mtp-family` through `/v1/chat/completions`
> (chat template applied, `timings` still carried); (2) at the harness default
> `--vram-target 22000` the drafter squeezes the resident-KV window to 0 MiB → whole KV streams
> from host over PCIe Gen3 and the `n_max=3` drafter re-streams it per step → **net loss
> 11.12→8.42 (0.76×)**. NOT a bug — pure spill: +1 GB headroom (23000) keeps it resident and
> MTP flips to **+68%**. The GGUF is fine (byte-identical to canonical
> `bartowski/google_gemma-4-31B-it-GGUF`). On **bs2** (roomy VRAM) the same pair is **2.03×** —
> the 3090's 1.68× is the same win on a tighter card. bug-1116.

Gemma MTP is opencoti-only (stock llama.cpp here rejects the drafter with
`unknown model architecture: 'gemma4-assistant'` — that arch exists only in
ours via #476), so it cannot be cross-checked against upstream. The Qwen NextN
self-spec paths *can* be, and doing so on 35B-A3B surfaced a real regression.

### ⚠ Qwen-35B-A3B NextN self-spec — ours regresses vs stock llama.cpp

Same GGUF (UD-Q6_K), same greedy 400-tok decode, `--spec-draft-n-max` swept:

| n_max | ours decode | stock llama.cpp decode | ours accept | stock accept |
|---|---|---|---|---|
| base | 210.5 | 212.0 | — | — |
| 1 | 222.6 (+5.7%) | 254.3 (+20%) | 0.847 | 0.851 |
| 2 | 230.7 (+9.6%) | **271.1 (+28%)** | 0.731 | 0.795 |
| 3 | **156.7 (−26%)** | 260.5 (+23%) | 0.698 | 0.684 |

**Base decode is at parity (210.5 vs 212.0) and acceptance is at parity — but
ours' self-spec throughput is ~⅓ of stock's, and *collapses* at n=3.** This is
not a "fast base can't benefit from MTP" effect (my first read): stock proves the
model gains +28% and stays positive at every depth. It is an **ours-side
spec-decode regression**. The signature: ours' CUDA-graph reuse drops 161→128
from n=2→n=3 as decode collapses 231→157, while stock holds +23% at n=3 with a
comparable graph count — i.e. ours thrashes graph rebuilds on the variable
draft-batch shapes, plus carries a baseline per-step spec overhead even at n=2.
Tracked for fix (see buglog bug-858, task #590). The genuine "MTP helps a
slow base most" point still holds for the *magnitude* of the win (27B/31B big,
A4B moderate) — but it is not why 35B underperforms here.

> **✅ SUPERSEDED 2026-07-05 (#609 / bug-2103) — the n=3 collapse and the "cap at
> 2" recommendation are OBSOLETE.** The collapse was the TinyBLAS `mmvf` ne11≤3
> 1-block-GEMV cliff (bug-2103), fixed by gating the cap at ne11≤8. On the #614
> binary Qwen-35B NextN n3 is now **monotone and ours' best rung** (see the
> §2026-07-05 matrix below): n1→n2→n3 rises, n3 no longer regresses. Do **not**
> cap `--spec-draft-n-max` at 2 anymore. The table above is retained as the
> pre-fix historical record only.

**It is not Qwen-specific — ours' whole spec-decode path is behind upstream.**
Cross-checked against a freshly-built upstream **b9859** (the Jun-1 reference
predated the Gemma4-assistant merge; b9859 has it) on identical GGUFs/HW:

| model (n=2) | ours MTP | upstream b9859 MTP | ours accept | b9859 accept |
|---|---|---|---|---|
| Qwen-35B-A3B (NextN self-spec) | 230.7 (+9.6%) | **271.0 (+28%)** | 0.731 | 0.770 |
| Gemma-4-A4B (assistant-MTP) | 242 (+20%) | **265.9 (+32%)** | **0.722** | 0.637 |

On Gemma, **ours has *higher* acceptance (0.722 vs 0.637) yet a *smaller* speedup**
— dispositive that the loss is per-step spec-loop overhead, not drafter quality,
and that it affects both the NextN self-spec and the external-assistant paths.
Parity fix is task #590.

#### #590 diagnostic (2026-07-02) — gap localized to per-cycle host-sync; acceptance ruled out

Three levers were chased to isolate the 35B-A3B NextN gap; the first three are
**dead ends**, leaving one:

- **Acceptance is at parity — not the driver.** A fresh side-by-side on the LRU
  prompt (ours `llamafile-592` vs upstream `b9859`, n_max=2, greedy 400-tok)
  measured ours accept **0.79–0.80** vs upstream **0.815** (~1.5pp), with
  **identical cycle counts** (draft_n 307 vs 303, accepted 244 vs 247) — yet
  upstream decodes **283 vs ours 244 t/s (+16%)**. Same acceptance, same number
  of draft/verify cycles, faster wall ⇒ the gap is **per-cycle efficiency**, not
  acceptance. (The 0.731/0.770 cells above were a single noisier run; the gap is
  real but acceptance is not its cause.) `bug-867`.
- **Input-upload syncs — dead.** An async-H2D spike (`OPENCOTI_ASYNC_INPUTS`)
  found the per-decode graph inputs live on a non-device buffer type, so
  `set_tensor_async` never engages (asserts unguarded). Byte-identical, zero
  movement. `bug-866`.
- **Graph-thrash / `OPENCOTI_MTP_FUSED` on NextN — not applicable.** The fused
  knob gates the **Gemma-assistant** path only (`decode_mtp_fused` requires
  `model.mtp_assistant` + `llama_kv_cache_iswa`); it is inert on NextN by design.
  On its own path fused **works** (default-on, byte-identical: 197.6 stable vs
  161–192 sequential). `bug-864`, `bug-866`.

**Conclusion.** Base single-decode is at parity, acceptance is at parity, cycle
count is equal — so the +16% is the **host-sync/launch overhead of the un-fused
NextN per-step draft loop** (`common_speculative_impl_draft_mtp`: per draft token
a `llama_decode` + a `llama_get_embeddings_pre_norm_ith` host readback). The one
remaining validated lever is a **fused NextN draft graph** — the NextN analogue of
`decode_mtp_fused` (multi-step in-graph argmax+embed chain, one launch + one
readback per block). Design is scoped separately; payoff scales with n_max
(modest at the n=2 cap, expected to fix the n=3 collapse). `bug-867`.

**Gemma vs Qwen MTP use distinct `--spec-type` strings — get this right or the
draft context fails to build.** The Gemma external assistant is
`--spec-type draft-assistant --mtp-head <gemma4_assistant.gguf> -ngld 99`; the
Qwen NextN self-spec is `--spec-type draft-mtp` (no drafter file). An early run
of this arm wrongly used the Qwen combo (`-md … --spec-type draft-mtp`) for the
Gemma cells → `load_model: [spec] failed to measure draft model memory: failed
to create llama_context from model` (Gemma has no NextN layers for `draft-mtp`
to attach to). It is **not** an engine or drafter-format regression: with the
correct `draft-assistant` flags both Gemma cells serve and accept normally, as
above (matching the #481 A4B beachhead). The MTP-manifest example in
`three-way-tps.models.example` carries the correct flags + a warning.

## 2026-07-05 refresh — post bug-858 fixes (ours vs upstream b9859, full MTP matrix)

> **⚠ MTP-arm + 9-prompt decode numbers below are SUPERSEDED by the 2026-07-14 refresh**
> (post-#675 spec-clone-cheap fix — every ours spec cell gained +4–20%). Base + prefill
> numbers in this section remain current. Accept columns are identical in both runs.

Re-ran the ours-vs-upstream matrix on **bs2** after the two bug-858 root-cause fixes landed
(bug-2101 `can_reuse` reserve-graph reuse + bug-2102 `ensure_cleared` aliased-KV memset — see
[`mtp.md`](mtp.md) §3e). Same method as above (ctx 4096, ~1760-tok prefix, `n_predict=400`
`ignore_eos`, temp 0, 2-prompt avg); ours = dual-ctx binary `20acc0722ea4f97c`
(`OPENCOTI_MTP_DUAL_CTX=1` on the Gemma cells), upstream = **b9859**; models per the registry
below (A4B = the NATIVE google pair, not retired v6-coder). Harness
`bs2:/srv/ml/opencoti-c1/mtp-matrix.sh` → `mtp-matrix-out/summary.txt`.

### Base — ours vs b9859

| model | ours pp / tg | b9859 pp / tg | decode verdict |
|---|---|---|---|
| Gemma-4-26B-A4B-it Q4_K_M (MoE) | ~~4818~~ **7539** (r1 6296 · r2 8782) / **204.0** | 6316 (r1 5053 · r2 7579¹) / 190.1 | **ours +7%** |
| Gemma-4-31B-it Q6_K | ~~1582~~ **2302** (r1 1938 · r2 2667) / 45.3 | 2240 (r1 2245 · r2 2235) / 44.8 | parity |
| Qwen3.6-35B-A3B UD-Q6_K (MoE) | ~~4683~~ **6089** (r1 5917 · r2 6261) / 208.3 | 6406 (r1 5280 · r2 7531¹) / 212.1 | parity (−1.8%) |
| Qwen3.6-27B-Omnimerge-MTP Q6_K | ~~2820~~ **2858** (r1 2799 · r2 2918) / 55.8 | 2922 / 56.0 | parity |

¹ b9859 r2 is prompt-cache-restore inflated (it removed the `cache_prompt` opt-out); ours' r2
is honest full recompute with a warm checkpoint buffer.

**Decode: no regression anywhere; ours now beats b9859 on A4B MoE decode.** **Prefill (#614,
2026-07-05): the −25–35% gap was NOT upstream movement — it was THREE ours-side defects, all
fixed.** (1) **bug-2104 TinyBLAS small-GEMM cliff**: f32 GEMMs (Gemma MoE router `ffn_gate_inp`
[2816×128]; q35 DeltaNet chunk GEMMs) skip MMQ/mmf/mmvf and land in the cuBLAS slot, where
TinyBLAS's single 128×64-tile config launches only 8 blocks on a 142-SM GPU (274 µs/call vs
cuBLAS cutlass tf32+splitK 8 µs). Fix `opencoti-hook: tinyblas-small-gemm` (tinyblas.cu):
a 32×32-tile/2-warp config of the same deterministic warp2d kernel when the big config would
launch <24 blocks (router 274→90.7 µs). (2) **bug-2105 checkpoint_min_step 256→8192**: our
server port carried a stale early-upstream default, taking a SWA/recurrent state checkpoint
(full state D2H into a zeroed host buffer, ~250–400 ms on 31B) every 512-token ubatch instead
of every 8192 tokens. (3) **bug-2107 checkpoint buffer recycle** (`opencoti-hook:
ckpt-buf-noinit`, server-context.cpp): each request allocated a fresh GB-class host vector for
the snapshot (memset + page-fault storm + cold-page D2H); retired checkpoint buffers are now
recycled through a small pool (`ckpt_buf_pool`), so steady-state requests write into warm
pages — no memset, no faults. Measured dead end kept for the record: a non-zeroing allocator
WITHOUT the pool made things worse (faults moved into the D2H copy loop; −8–13% all models).
Final @1760-tok in the table above; steady-state (r2, warm) ours leads upstream's honest
number on every model (31B 2667 vs 2240 = +19%; A4B 8782; q35 6261 vs 5280 cold). MTP
accept/tps unchanged (spec rollback uses its own per-cycle `spec_ckpt`, not pooled).
⚠ Bench caveat: b9859 removed `cache_prompt` (always caches), so its r2 on identical prompts
is inflated +43–50% on MoE (state restore, not compute); cold r1-vs-r1 ours LEADS on both MoE
models and is −14% on 31B only for the very first request per slot (cold pool).

### MTP arm — ours vs b9859 (decode t/s · accept)

Ours columns re-measured **2026-07-05 on the final #614 binary** (tinyblas-small-gemm +
checkpoint fixes in). Decode/accept is within run noise of the pre-#614 cells on every row —
the #614 fixes are prefill-side and MTP-neutral, as designed. All four documented pairs
covered: both Gemma assistant-drafter targets (A4B, 31B) and both Qwen NextN self-spec
models (35B-A3B, 27B-omnimerge). Raw cells: bs2 `/srv/ml/opencoti-c1/mtp614-out/`.

| model / config | ours n1 | up n1 | ours n2 | up n2 | ours n3 | up n3 |
|---|---|---|---|---|---|---|
| **A4B assistant-MTP (dual-ctx)** | 257.4 · 0.975 | 287.7 · 0.939 | **302.1 · 0.845** | 319.4 · 0.890 | **329.1 · 0.784** | 366.6 · 0.900 |
| 31B assistant-MTP (dual-ctx) | — | — | 100.3 · 0.849 | 107.4 · 0.912 | 118.6 · 0.791 | 131.6 · 0.881 |
| Qwen 35B-A3B NextN self-spec | 224.9 · 0.858 | 259.8 · 0.882 | 237.4 · 0.752 | 278.2 · 0.804 | 247.1 · 0.662 | 269.9 · 0.701 |
| Qwen 27B NextN self-spec | — | — | 99.1 · 0.752 | 107.5 · 0.763 | 104.7 · 0.626 | 116.2 · 0.654 |

MTP-config **prefill** also carries the #614 gains (2-request avg, same cells): A4B 6283–6476,
31B 1965–1968, q35 4915–5012, q27 2327–2348 tok/s.

### 9-prompt am17an bench — per-prompt accept, A4B native pair (bs2, 2026-07-05, final binary)

`mtp-bench.py`, 9 prompts × 192 predicted tokens each, greedy; same drafter GGUF both engines.
Raw JSONs: bs2 `/srv/ml/opencoti-c1/mtp614-out/b9_{ours,up}_n{1,2,3}.json`.

| prompt | ours n1 | up n1 | ours n2 | up n2 | ours n3 | up n3 |
|---|---|---|---|---|---|---|
| code_python | 0.919 | 0.919 | 0.911 | 0.890 | 0.868 | 0.891 |
| code_cpp | 0.873 | 0.900 | 0.808 | 0.826 | 0.714 | 0.788 |
| explain_concept | 0.809 | 0.809 | 0.669 | 0.698 | 0.569 | 0.623 |
| summarize | 0.900 | 0.919 | 0.851 | 0.851 | 0.759 | 0.798 |
| qa_factual | 0.845 | 0.919 | 0.757 | 0.789 | 0.681 | 0.750 |
| translation | 0.969 | 0.969 | 0.890 | 0.871 | 0.736 | 0.834 |
| creative_short | 0.652 | 0.768 | 0.651 | 0.627 | 0.561 | 0.574 |
| stepwise_math | 0.939 | 0.891 | 0.808 | 0.871 | 0.718 | 0.759 |
| long_code_review | 0.837 | 0.809 | 0.762 | 0.757 | 0.653 | 0.688 |
| **aggregate accept** | **0.856** | **0.876** | **0.784** | **0.792** | **0.688** | **0.736** |
| **decode t/s (agg)** | 238 | 263 | 274 | 280 | 282 | 293 |

**Read:** n1/n2 accept is at parity (−2.0pp / −0.8pp aggregate; per-prompt ours wins 2–3 of 9
at each depth — stepwise_math, long_code_review, code_python/translation at n2). The n3
aggregate gap (−4.8pp) is spread thinly across prompts (worst: translation −9.8pp, code_cpp
−7.4pp), consistent with the #495-class target-numerics residual, not a single broken prompt
class. Recommended depth unchanged: **n2 for A4B** (best accept·tps balance on ours), n3 on
Qwen NextN.

**Read (supersedes the ⚠ sections above for current state):**

1. **A4B is FIXED.** Pre-fix, ours' assistant path collapsed with prompt length (accept → 0.0 at
   1.3k tok) and trailed −10pp at a single draft step. Post-fix: accept **at/above upstream at
   n1/n2** (9-prompt am17an-style compare: n1 0.862 vs 0.876, n2 **0.800 vs 0.792**; flat vs
   prompt length to 2.6k tok), MTP-on decode 315 vs 319 at n2 (−1.4%), and dual-vs-plain
   losslessness now matches upstream's own flip profile. Recommended depth on ours: **n2**.
2. **The 35B n3 decode collapse is FIXED (#609, bug-2103, 2026-07-05).** Root cause was NOT
   CUDA-graph churn: node-level nsys (`--cuda-graph-trace=node`) showed ~38 f32 dot GEMVs/cycle
   ([K,1]×[K,4], fused-GDN surroundings) falling off mmvf's NVIDIA f32 cap (`ne11<=3`) into
   llamafile's TinyBLAS GemmEx — ONE thread block, ~174 µs vs upstream's cuBLAS GEMV ~1.2 µs
   ≈ 35% of GPU wall. Depth 3's 4-token verify batch is the first rung past the cap, hence
   n3-only. Fix: `GGML_USE_TINYBLAS`-gated `ne11<=8` in `should_use_mmvf` (kernel max,
   upstream-faithful; cuBLAS builds unchanged). Post-fix: n3 **247.9 · 0.662** (was 149.3),
   curve monotone 225→238→248 like upstream's; base 211 vs up 212 (parity). Residual ~8–13%
   at all spec depths = the pre-existing per-cycle leg, not depth-triggered. n3 is now ours'
   best rung. A4B n3 re-test with the fixed kernel: decode 312.9→**330.2** (+5.5% — A4B's
   verify batch also had TinyBLAS-routed f32 GEMVs), accept 0.758→**0.784**. The remaining
   −12pp vs upstream 0.900 at n3 is the pre-existing #495-class target-numerics residual
   (knife-edge thinking-prompt class, see the A4B parity note), NOT depth machinery — n2
   remains the recommended depth for A4B; n3 is now safe (and fastest) on Qwen NextN.
3. **Qwen 27B NextN is at working parity** (−7/−9% decode, accept parity) — the 35B n3 collapse
   is depth-triggered, not NextN-wide.
4. 31B accept trails ~6–10pp at n2/n3 (same adapted drafter GGUF on both engines) — the
   n3-class residual, milder; still 2.6× base at n3 (117 t/s).

## 2026-07-14 refresh — post #675 spec-cycle host parity (patch 0128): the MTP gap is CLOSED

Re-ran the FULL ours-vs-b9859 MTP matrix after the two #675 fixes landed (spec-clone-cheap in
`common_sampler_clone` — the 1.07 ms/cycle cosmo-dlmalloc deep-copy that was the entire
wall-clock deficit — plus the mmvf f16/bf16 TinyBLAS verify lift; see patch `0128` and
[`mtp.md`](mtp.md) §3g). Same method/models/prompts as the 07-05 section; **both engines
re-measured same-day on the same GPU** (bs2 **GPU1**, GPU0 occupied by an unrelated job — mild
shared-host contention applies equally to both columns). Ours = binary `05240e8e57c7191e`
(dual-ctx on Gemma cells), upstream = b9859 (binary unchanged since 07-05). Harness
`bs2:/srv/ml/opencoti-c1/mtp675.sh` → `mtp675-out/summary.txt`. Accept columns are
**bit-identical** to the 07-05 run on every ours cell (0.975/0.845/0.784 …) — the #675 fixes
are pure-latency, output-identical, as designed.

### MTP arm — ours vs b9859 (decode t/s · accept), 2026-07-14

| model / config | ours n1 | up n1 | ours n2 | up n2 | ours n3 | up n3 |
|---|---|---|---|---|---|---|
| **A4B assistant-MTP (dual-ctx)** | **307.9 · 0.975** | 289.1 · 0.939 | **353.8 · 0.845** | 321.8 · 0.890 | **378.3 · 0.784** | 369.1 · 0.900 |
| 31B assistant-MTP (dual-ctx) | — | — | 105.2 · 0.849 | 107.3 · 0.912 | 123.6 · 0.782 | 131.6 · 0.881 |
| Qwen 35B-A3B NextN self-spec | **265.2 · 0.858** | 261.3 · 0.882 | 270.0 · 0.752 | 277.7 · 0.804 | **278.8 · 0.662** | 270.7 · 0.701 |
| Qwen 27B NextN self-spec | — | — | 102.4 · 0.752 | 107.5 · 0.763 | 109.4 · 0.626 | 116.2 · 0.654 |

Delta vs the 07-05 (pre-#675) ours column: **every spec cell gains +4–20%** (A4B n2 302→354,
n3 329→378; q35 n1 225→265, n3 247→279; 31B n3 119→124; q27 n3 105→109) — exactly the
per-cycle host-glue reclaim, scaling with cycle rate. Upstream columns moved only by
run-to-run/GPU1 noise vs 07-05.

**Verdict per pair:**

1. **A4B assistant: ours now BEATS b9859 at every depth** (n1 +6.5%, n2 +9.9%, n3 +2.5%) —
   ours' lower accept at n2/n3 is more than compensated by the cheaper cycle.
2. **Qwen 35B NextN: parity-or-better** (n1 +1.5%, n2 −2.8%, n3 **+3.0%** — the former −12%
   headline gap rung). bug-858 is CLOSED.
3. 31B assistant −2/−6%, 27B NextN −5/−6% at n2/n3 — residual is the #495-class accept deficit
   at depth (chained-draft numerics; same drafter GGUF both engines), not cycle cost.
4. Recommended depths unchanged: **n2–n3 on A4B** (all rungs now win), **n3 on Qwen NextN**.

### 9-prompt am17an bench — per-prompt accept, A4B native pair (bs2 GPU1, 2026-07-14, 0128 binary)

`mtp-bench.py`, 9 prompts × 192 predicted tokens each, greedy; same drafter GGUF both engines.
Raw JSONs: bs2 `/srv/ml/opencoti-c1/mtp675-out/b9_{ours,up}_n{1,2,3}.json`.

| prompt | ours n1 | up n1 | ours n2 | up n2 | ours n3 | up n3 |
|---|---|---|---|---|---|---|
| code_python | 0.919 | 0.919 | 0.911 | 0.890 | 0.868 | 0.891 |
| code_cpp | 0.873 | 0.900 | 0.826 | 0.826 | 0.732 | 0.788 |
| explain_concept | 0.809 | 0.809 | 0.669 | 0.698 | 0.569 | 0.623 |
| summarize | 0.900 | 0.919 | 0.851 | 0.851 | 0.759 | 0.798 |
| qa_factual | 0.845 | 0.919 | 0.757 | 0.789 | 0.681 | 0.750 |
| translation | 0.969 | 0.969 | 0.890 | 0.871 | 0.754 | 0.834 |
| creative_short | 0.652 | 0.768 | 0.651 | 0.627 | 0.566 | 0.574 |
| stepwise_math | 0.939 | 0.891 | 0.808 | 0.871 | 0.718 | 0.759 |
| long_code_review | 0.837 | 0.809 | 0.762 | 0.757 | 0.653 | 0.688 |
| **aggregate accept** | **0.856** | **0.876** | **0.786** | **0.792** | **0.692** | **0.736** |
| **decode t/s (mean)** | **317.7** | 298.1 | **361.6** | 320.2 | **365.7** | 339.8 |

**Read:** accept is statistically unchanged vs 07-05 (aggregate deltas ≤0.5pp, per-prompt
wins/losses in the same places — stepwise_math/long_code_review ours, creative_short/qa_factual
upstream; the n3 −4.4pp aggregate residual remains the #495-class target-numerics spread). The
new fact is **throughput: ours' 9-prompt decode mean now LEADS upstream at every depth**
(+6.6% n1, **+12.9% n2**, +7.6% n3) — pre-#675 it trailed at all three (238/274/282 vs
263/280/293 on 07-05). Depth guidance for A4B: n2 and n3 are now effectively tied on ours
(361.6 vs 365.7); n2 keeps the higher accept.

## Pinned models — exact identity on each host (registry)

**Why this table exists.** The same model family exists at *different quants* across
solidPC and bs2, and in two cases the *same filename holds different bytes*. Picking the
wrong copy silently benches a different model. **Always resolve a model through this
registry (sha256) — never by hunting the disks.** Authoritative eval host is **bs2**;
solidPC copies are for 3090 / local dev. Captured 2026-07-03.

### bs2 — RTX PRO 6000 (authoritative 3-way + MTP eval host), under `/srv/ml/models/`

| role | file · dir | bytes | sha256 |
|---|---|---|---|
| **Qwen NextN self-spec — bug-858 headline** | `Qwen3.6-35B-A3B-UD-Q6_K.gguf` · `gguf/Qwen3.6-35B-A3B-MTP/` | 30011242784 | `49935b04ad883c2f3d4da61f65b609d447dad67d0b08453b90abb09a1bb35464` |
| Qwen NextN self-spec — 27B (nextn baked) | `Qwen3.6-27B-Omnimerge-v4-Q6_K.gguf` · `nextn/omnimerge-mtp/` | 22430999136 | `eb4532bde36550f7f9d99ffacec265d416d4ad3c77d0b3e657639254df4be3be` |
| Qwen 27B base (NO nextn) | `Qwen3.6-27B-Omnimerge-v4-Q6_K.gguf` · `gguf/Qwen3.6-27B-Omnimerge-v4-GGUF/` | 22082528832 | `8d574a27e5034861c45107397d5a179dac3a634f4efacc1b9e2274e36f39c5bc` |
| Qwen 14B-1M base | `Qwen2.5-14B-Instruct-1M-Q8_0.gguf` · `Qwen2.5-14B-Instruct-1M/` | 15701598272 | `68ee5c4845c61730a7d769cb31c13f62b2752042111497dcfa80753e04195149` |
| **A4B native target** | `google_gemma-4-26B-A4B-it-Q4_K_M.gguf` · `gguf/google-a4b-128e/` | 17035038112 | `e718536fe9b4bd505b07d44ded8f1595053a5d5407315bccf555ce592f33c140` |
| **A4B assistant drafter (native pair)** | `gemma-4-26B-A4B-it-assistant.Q8_0.gguf` · `nextn/drafters-hf/26B-A4B/` | 461767136 | `5963a2edfb516fbdff499b1d4b7f8158a7a811af7196eaae00be2119188e422e` |
| A4B v6-coder — ⚠ RETIRED 2026-07-03 (deprecated) | `gemma-4-A4B-98e-v6-coder-it-Q4_K_M.gguf` · `variants/v6coder/` (bs2 copies remain) | 13239743776 | `debb603160d278fd7fd33028a0357ccbd06ee4c4bf03addfcffa2329f2f7a951` |
| 31B target | `google_gemma-4-31B-it-Q6_K.gguf` · `gguf/gemma-4-31B-it-GGUF/` | 26726720128 | `7961268b989e16ec846fc7077e27f78212cc33553688d59066df4491f057f928` |
| 31B assistant drafter | `gemma-4-31B-it-assistant.Q8_0.gguf` · `nextn/drafters-adapted/31B/` | 514687456 | `ca7820a7289d4c88b3e09626bf9bc8f44bd90ee5b54d33d42049b69d5c349a06` |
| Qwen3-8B-Q8_0 | **ABSENT** on bs2 (present at doc-eval time; removed since) | — | — |

### solidPC — RTX 3090, under `.opencoti/models/` (symlink → spool)

| role | file · dir | bytes | sha256 |
|---|---|---|---|
| Qwen 8B | `Qwen3-8B-Q8_0.gguf` · `.opencoti/models/` | 8709518112 | `408b955510e196121c1c375201744783b5c9a43c7956d73fc78df54c66e883d6` |
| Qwen 14B-1M ⚠ Q6_K (not Q8_0) | `Qwen2.5-14B-Instruct-1M-Q6_K.gguf` · `.opencoti/models/` | 12124684352 | `c2faeb8ebe703924668aceab5fd49afaf5cb13efef75dfc3d3796f5c6de1cf52` |
| Qwen 27B ⚠ Q4_K_M (not Q6_K, no nextn) | `Qwen3.6-27B-Omnimerge-v4-Q4_K_M.gguf` · `.opencoti/models/` | 16810713696 | `59e5c6fa3d44cb56937001696b6869b4b2cb2fe0643c0c12cd63a0a8221dfb79` |
| Qwen 35B-A3B ⚠ IQ3_XXS (not Q6_K) | `Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf` · `.opencoti/models/` | 13211155424 | `9c964e657212fea1f24905dd7b0a89b82fd807d19fab0b41da14251b07b88fbe` |
| A4B native target ✓ == bs2 | `google_gemma-4-26B-A4B-it-Q4_K_M.gguf` · `.opencoti/models/` | 17035038112 | `e718536fe9b4bd505b07d44ded8f1595053a5d5407315bccf555ce592f33c140` |
| A4B v7-coder (replaces v6; Q4_K_M re-synced from HF 2026-07-14) | `gemma-4-A4B-98e-v7-coder-it-{Q4_K_M,IQ4_XS}.gguf` · `.opencoti/models/` | Q4_K_M 13239743904 | Q4_K_M `89b4038c…` ✓ == HF-pub · IQ4_XS `6048770e…` ⚠ still ≠ HF-pub `07c6406b` |
| 31B target ⚠ Q4_K_M (not Q6_K) | `gemma-4-31B-it-Q4_K_M.gguf` · `.opencoti/models/` | 19598488192 | `35dacd53f9dde83fe20a4596efce2b2f72d2ebb6e4141186533419e9e98bc269` |
| A4B assistant drafter ✓ == bs2/HF | `gemma-4-26B-A4B-it-assistant-Q8_0.gguf` · `.opencoti/models/drafters/` | 461767136 | `5963a2edfb516fbdff499b1d4b7f8158a7a811af7196eaae00be2119188e422e` |
| 31B assistant drafter ✓ == bs2 | `gemma-4-31B-it-assistant-Q8_0.gguf` · `.opencoti/models/drafters/` | 514687456 | `ca7820a7289d4c88b3e09626bf9bc8f44bd90ee5b54d33d42049b69d5c349a06` |
| 12B assistant drafter ✓ == bs2/HF | `gemma-4-12B-it-assistant-Q8_0.gguf` · `.opencoti/models/drafters/` | 465109440 | `223bb1d1e18b28efc196042b7e166321bc0b3e10e8ca0b97f7a88c5abe1654c1` |
| E2B assistant drafter | `gemma-4-E2B-it-assistant-Q8_0.gguf` · `.opencoti/models/drafters/` | 99423712 | `78996a998e4773247e789f02716e4fae64c9026c6cce755002d17d9f50882849` |
| E4B assistant drafter | `gemma-4-E4B-it-assistant-Q8_0.gguf` · `.opencoti/models/drafters/` | 100259296 | `e2c1426e6552ac7b9e73bf66991cf6aef62580243424860f82746ceb00fbb4cb` |

### ⚠ Cross-host alarms — same name ≠ same bytes

- **Byte-identical both hosts** (safe either): A4B native target `e718536f`; **all 5 gemma4 drafters Q8_0 — HF-canonical on BOTH hosts after the 2026-07-03 re-sync** (A4B `5963a2ed`, 12B `223bb1d1`, 31B `ca7820a7`, E2B `78996a99`, E4B `e2c1426e`; source of truth = private `ManniX-ITA/gemma-4-<size>-it-assistant-GGUF`). Only the filename differs cosmetically — solidPC `-Q8_0.gguf` (dash) vs bs2/HF `.Q8_0.gguf` (dot); the bytes/hash match, which is what counts.
- **Same filename, DIFFERENT bytes** — never cross-substitute: A4B v7-coder **IQ4_XS** (solidPC local `6048770e` ≠ HF-published `07c6406b`) — re-fetch from HF before any cross-host IQ4_XS eval. *(The Q4_K_M mismatch was RESOLVED 2026-07-14: solidPC re-synced from HF to `89b4038c`, old `1607dce0` deleted, `.sha256` sidecar written. v6-coder RETIRED 2026-07-03: removed on solidPC, deprecated on bs2. The A4B + 12B drafter byte mismatches were RESOLVED same day — solidPC re-synced from HF to `5963a2ed`/`223bb1d1`.)*
- **Different quant, same family** — solidPC holds the LOWER quant; never sub it for a bs2 eval: 35B-A3B (Q6_K vs IQ3_XXS), 27B (Q6_K vs Q4_K_M), 14B-1M (Q8_0 vs Q6_K), 31B (Q6_K vs Q4_K_M).
- **One host only**: Qwen3-8B-Q8_0 → solidPC only. 35B-A3B-UD-Q6_K, 27B nextn-baked (`omnimerge-mtp/`), A4B native drafter (`drafters-hf/`) → bs2 only.
- **Coder-variant policy (2026-07-03):** v6-coder RETIRED. Coder re-runs use **v7-coder** (canonical = HF `ManniX-ITA/gemma-4-A4B-98e-v7-coder-it-GGUF`, Q4_K_M `89b4038c`) or the **A4B-128e** full-Google target (`e718536f`). The v6-coder bench rows above (Base 3-way + assistant-MTP) are re-run fresh, not reconciled.

### bug-858 canonical pair (the two models the both-paths eval MUST use)

- **Qwen NextN self-spec** → bs2 `gguf/Qwen3.6-35B-A3B-MTP/Qwen3.6-35B-A3B-UD-Q6_K.gguf` (`49935b04…`), `--spec-type draft-mtp` (nextn baked, no drafter). solidPC has **no** Q6_K equivalent (only IQ3_XXS).
- **A4B assistant-MTP** → bs2 target `gguf/google-a4b-128e/google_gemma-4-26B-A4B-it-Q4_K_M.gguf` (`e718536f…`) + drafter `nextn/drafters-hf/26B-A4B/gemma-4-26B-A4B-it-assistant.Q8_0.gguf` (`5963a2ed…`), `--spec-type draft-assistant --mtp-head <drafter> -ngld 99`. solidPC `drafters/gemma-4-26B-A4B-it-assistant-Q8_0.gguf` now carries the SAME HF-canonical bytes (`5963a2ed`, dash-name only) after the 2026-07-03 re-sync — safe to use on either host.

## Caveats

- Short-ctx only. Long-context and the DCA sweep are separate evaluations.
- 2-prompt average at temp 0 (deterministic); `ignore_eos` guarantees the full
  `NPRED` decode length so tg is stable. Bump `NPRED` for tighter numbers.
- Stock llamafile 0.10.3 may `LOADFAIL` on archs newer than its bundled
  llama.cpp vintage — captured per-cell, not fatal.