# Ornith-1.0-35B IQ4_XS Trained MTP Profile

Date: 2026-06-27

Hardware: single RTX PRO 6000 Blackwell Max-Q 96GB, `tp=1`.

Backend: patched local `llama.cpp` CUDA server, launched through
`scripts/serve_llamacpp_gpu0.sh`.

Target model: `ornith-1.0-35b-IQ4_XS.gguf`.

## Summary

The upstream Ornith checkpoint advertises MTP in config but does not ship a
loadable MTP/NextN head. This lab first created a bootstrap MTP draft to make
llama.cpp `draft-mtp` load, then trained a chain-corrected Qwen3.5 MoE MTP draft
head from target hidden-state captures.

The trained draft is operational and reaches the expected acceptance band:
the best profile accepts 67.19-67.22% of generated speculative tokens with
`n_max=2` on the c16/128 MTP fix-loop profile. However, it is still slower than
target-only IQ4_XS at c1, c4, c8, and c16. The current bottleneck is not draft
quality alone; llama.cpp still performs sequential draft and target verification
work with host-side sampling/control overhead. At higher concurrency the
target-only path already saturates the GPU better.

Recommendation: use target-only `Q4_K_M` or target-only `IQ4_XS` for production
throughput. Use the trained Q6_K MTP draft only for MTP development and
regression testing.

A later head-1 chain-prefix refinement sweep produced three additional Q6_K
draft GGUFs at 250, 500, and 1000 training steps. These trained cleanly offline,
but did not improve the online verifier path: all three landed at 283.74-284.62
tok/s and 65.79-65.91% draft-token acceptance on the matched c16/128 profile.
They are preserved as experimental artifacts and are not promoted over the
h2r9 Q6_K profile.

The patched efficient sequential-verification path fixes the deterministic
target-equivalence issue for MTP, but it is also not a production throughput
speedup. It measured 192.48/165.63/167.47 tok/s at c1/c4/c8 on the 256-token
low-concurrency prompt mix and 169.24 tok/s at c16/128, while matching the
target-only deterministic probe exactly.

## External Grafted MTP Comparison

We also tested `wang-yang/Ornith-1.0-35B-MTP-GGUF`, which is a single combined
Q6_K GGUF with a same-architecture MTP layer grafted into the model file. Its
model card reports 92.2% acceptance and a 1.26x speedup on an M3 Max
single-prompt code-continuation benchmark at `--spec-draft-n-max 1`.

On our RTX PRO 6000 Blackwell Max-Q c16/128 OpenAI-server benchmark, the grafted
head did reproduce the high acceptance behavior but not the throughput speedup:

| Profile | Requests | Output tokens | TPS | p95 TTFT ms | Draft acceptance |
|---|---:|---:|---:|---:|---:|
| wang-yang Q6_K-MTP GGUF, no MTP | 128 | 15,780 | 550.29 | 517.0 | n/a |
| wang-yang Q6_K-MTP GGUF, native `draft-mtp`, `n_max=1` | 128 | 15,782 | 257.74 | 868.5 | 90.67%; mean accepted len 1.91 |

The integrated graft is materially different from our current local profile:
llama.cpp creates an MTP draft context against the target model with only an
81.02 MiB estimated MTP context, while our local MTP profile uses an external
3.4G Q6_K draft GGUF plus a separate draft context. However, the matched c16
server run still slowed down because draft generation and target verification
remained sequential enough to dominate wall time. This makes the wang-yang model
useful as proof that a same-architecture graft can reach ~91-92% acceptance,
but it does not change the production recommendation for this server profile:
target-only IQ4_XS remains faster for batched single-GPU serving.

The important comparison is structural: the wang-yang Q6_K-MTP file and our
integrated IQ4_XS-MTP graft both have `qwen35moe.block_count=41`,
`qwen35moe.nextn_predict_layers=1`, and the same appended `blk.40.*` MTP tensor
set. The local IQ4_XS artifact changes the main-body quantization
(`general.file_type=30`) while preserving the appended MTP block at Q6_K/F16/F32.
The throughput gap is therefore a runtime/profile issue, not evidence that the
wang-yang head is doing a different kind of MTP.

## Integrated IQ4_XS Grafted MTP Artifact

We then requantized the wang-yang Q6_K integrated graft to an experimental
single-file IQ4_XS artifact:

`ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf`

Quantization:

```bash
llama-quantize --allow-requantize \
  --tensor-type-file runs/tensor-types-mtp-blk40-preserve.txt \
  Ornith-1.0-35B-Q6_K-MTP.gguf \
  ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf \
  IQ4_XS 24
```

The tensor override preserves the grafted MTP block `blk.40.*` at its original
Q6_K/F16/F32 types while requantizing the main model body to IQ4_XS. The
quantizer reported 27,844.13 MiB / 6.58 BPW source and 18,712.85 MiB / 4.42 BPW
output. Metadata verification confirmed `qwen35moe.block_count=41`,
`qwen35moe.nextn_predict_layers=1`, `general.file_type=30`, and all `blk.40`
MTP tensors present.

Matched c16/128 OpenAI-server benchmark:

| Profile | Requests | Output tokens | TPS | p95 TTFT ms | Draft acceptance |
|---|---:|---:|---:|---:|---:|
| Integrated IQ4_XS-MTP graft, no MTP, pre runtime patch | 128 | 15,767 | 594.96 | 502.4 | n/a |
| Integrated IQ4_XS-MTP graft, native `draft-mtp`, `n_max=1`, pre runtime patch | 128 | 15,782 | 283.34 | 729.1 | 89.67%; mean accepted len 1.90 |
| Integrated IQ4_XS-MTP graft, native `draft-mtp`, `n_max=2`, fast backend top-k=1, pre runtime patch | 128 | 15,656 | 326.8 | 815.8 | `(0.947, 0.812)`; mean accepted len 2.76 |
| Integrated IQ4_XS-MTP graft, no MTP, after recurrent row-index patch | 128 | 15,811 | 575.43 | 507.3 | n/a |
| Integrated IQ4_XS-MTP graft, native `draft-mtp`, `n_max=1`, fast backend top-k=1, after recurrent row-index patch | 128 | 16,232 | 296.89 | 797.9 | not logged in low-verbosity full run; prior matched `n_max=1` was 89.67% |
| Integrated IQ4_XS-MTP graft, native `draft-mtp`, `n_max=2`, fast backend top-k=1, after recurrent row-index patch | 128 | 16,195 | 337.40 | 818.1 | not logged in low-verbosity full run; trace confirmed high acceptance and graph reuse |
| Integrated IQ4_XS-MTP graft, native `draft-mtp`, `n_max=2`, process-timed rebuild | 128 | 16,005 | 322.46 | 826.1 | `(0.846, 0.688)`; mean accepted len 2.53 |
| Integrated IQ4_XS-MTP graft, native `draft-mtp`, `n_max=2`, target backend sampling enabled | 128 | 15,749 | 338.09 | 846.7 | `(0.934, 0.831)`; mean accepted len 2.77 |

Single-stream c1/256 check:

| Profile | Requests | Output tokens | TPS | p95 TTFT ms | MTP/AR ratio |
|---|---:|---:|---:|---:|---:|
| Integrated IQ4_XS-MTP graft, no MTP | 24 | 4,312 | 240.6 | 46.0 | n/a |
| Integrated IQ4_XS-MTP graft, native `draft-mtp`, `n_max=2`, fast backend top-k=1 | 24 | 4,312 | 313.8 | 54.0 | 1.30x |

This artifact is usable as a native `draft-mtp` GGUF and materially improves MTP
acceptance versus our locally trained external draft. It does provide a
single-stream speedup, but it still is not a production serving speedup in the
c16 profile. The bottleneck is now clearly in runtime scheduling/verification
overhead rather than MTP head compatibility.

The recurrent row-index patch changed Qwen35/Qwen35MoE MTP recurrent-state
writes from graph-baked head offsets to runtime row-index tensors for both
normal writes and rollback snapshots. A trace run after the patch showed no
remaining `mem_hybrid` or recurrent-state graph mismatch across head changes,
and graph reuse rose across the short c2/32 run. This fixed one real MTP runtime
issue, but it was not enough to overcome batched server overhead. The focused
diff is included as `patches/llamacpp-qwen35moe-mtp-recurrent-row-index.patch`.

Follow-up process timing showed that the missing cost was not
`common_speculative_process()`: the c16/128 process-timed run ended with
`#calls(b,p,g,a) = 128 436 433 6256` and `dur(b,p,g,a) = 0.100, 801.697,
1488.723, 7.496 ms`. Draft generation and MTP process bookkeeping were visible
but too small to explain the full gap to target-only. The larger issue is target
verification row shape.

Short c16/32 row-shape trace:

| Profile | Requests | Output tokens | TPS | Target decode calls | Target rows | Logit rows | Steady-state verifier shape |
|---|---:|---:|---:|---:|---:|---:|---|
| Target-only integrated graft | 16 | 512 | 389.09 | 34 | 931 | 512 | `16 seq x 1 row` |
| `draft-mtp n_max=2`, default verifier | 16 | 512 | 218.37 | 17 | 1,050 | 631 | `16 seq x 3 rows` |
| `draft-mtp n_max=2`, experimental batched-depth verifier | 16 | 512 | 159.21 | n/a | n/a | n/a | opt-in split by draft depth |

The default verifier halves target decode calls on this short probe but pushes
sampled plus draft rows through the target together. That creates more logit
rows than target-only and a `16 x 3` recurrent/MoE row shape that is slower on
this GPU than target-only's already-saturated `16 x 1` decode loop. An
experimental opt-in batched-depth verifier decoded sampled tokens across all
slots, then decoded only accepted draft-depth rows. It completed successfully
but measured only 159.21 tok/s on c16/32, so the extra decode launches cost more
than the rejected-row savings. The current conclusion is that integrated MTP is
valuable for c1 latency/throughput and runtime regression work, but not for the
recommended batched c16 serving profile.

## Draft Artifacts

The promoted h2r9 trained draft artifacts were produced from
`ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-bf16.gguf`.
The h2r10prefix rows are later head-1 refinement snapshots from the same draft
lineage.

| Draft file | Draft quant | Disk size | Estimated draft load | Quantization detail |
|---|---:|---:|---:|---|
| `ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-Q6_K.gguf` | Q6_K | 3.4G | 3176.45 MiB | `llama-quantize ... Q6_K`; source 8388.17 MiB / 16.01 BPW |
| `ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-Q5_K_M.gguf` | Q5_K_M | 3.1G | 2885.59 MiB | `llama-quantize ... Q5_K_M`; source 8388.17 MiB / 16.01 BPW, quantized 3090.01 MiB / 5.90 BPW |
| `ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-IQ4_XS.gguf` | IQ4_XS | 2.4G | 2246.29 MiB | `llama-quantize ... IQ4_XS`; source 8388.17 MiB / 16.01 BPW, quantized 2374.93 MiB / 4.53 BPW |
| `ornith-1.0-35b-mtp-chaincorr-h2r10prefix250-h1r2-h3r8fw-lr5e6-Q6_K.gguf` | Q6_K | 3.4G | 4143.45 MiB server estimate | Experimental head-1 chain-prefix refinement, snapshot step 250; `convert_hf_to_gguf.py --mtp --outtype bf16`, then `llama-quantize ... Q6_K`; source 8388.17 MiB / 16.01 BPW, quantized 3445.28 MiB / 6.57 BPW |
| `ornith-1.0-35b-mtp-chaincorr-h2r10prefix500-h1r2-h3r8fw-lr5e6-Q6_K.gguf` | Q6_K | 3.4G | 4143.45 MiB server estimate | Experimental head-1 chain-prefix refinement, snapshot step 500; `convert_hf_to_gguf.py --mtp --outtype bf16`, then `llama-quantize ... Q6_K`; source 8388.17 MiB / 16.01 BPW, quantized 3445.28 MiB / 6.57 BPW |
| `ornith-1.0-35b-mtp-chaincorr-h2r10prefix1000-h1r2-h3r8fw-lr5e6-Q6_K.gguf` | Q6_K | 3.4G | 4143.45 MiB server estimate | Experimental head-1 chain-prefix refinement, final step 1000; `convert_hf_to_gguf.py --mtp --outtype bf16`, then `llama-quantize ... Q6_K`; source 8388.17 MiB / 16.01 BPW, quantized 3445.28 MiB / 6.57 BPW |

The h2r9 Q6_K draft is the recommended MTP development draft despite the larger
footprint. Q5_K_M is slightly smaller but loses throughput and acceptance.
IQ4_XS is the smallest draft, but it drops acceptance to 59.20% and is not
recommended. The h2r10prefix rows are preserved as experimental rejected
refinements.

## Benchmark Profile

```bash
CUDA_VISIBLE_DEVICES=0 \
HOST=127.0.0.1 PORT=18237 QUANT=IQ4_XS \
MODEL_PATH=/path/to/ornith-1.0-35b-IQ4_XS.gguf \
ALIAS=ornith-iq4-xs-mtp-nmax2-fast-backend-topk1 \
CTX_SIZE=8192 PARALLEL=16 BATCH_SIZE=4096 UBATCH_SIZE=512 \
SPEC_TYPE=draft-mtp \
SPEC_DRAFT_MODEL=/path/to/ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-Q6_K.gguf \
SPEC_DRAFT_N_MAX=2 SPEC_DRAFT_N_MIN=0 \
SPEC_DRAFT_BACKEND_SAMPLING=1 \
MTP_PER_HEAD_CONTEXT=1 \
LLAMA_MTP_FAST_BACKEND_SAMPLE=1 \
LLAMA_MTP_DRAFT_TOP_K=1 LLAMA_MTP_DRAFT_TOP_P=1 LLAMA_MTP_DRAFT_TEMP=1 \
  scripts/serve_llamacpp_gpu0.sh
```

Load test:

```bash
scripts/bench_openai.py \
  --base-url http://127.0.0.1:18237/v1 \
  --model ornith-iq4-xs-mtp-nmax2-fast-backend-topk1 \
  --requests 128 --concurrency 16 --max-tokens 128 \
  --temperature 0 --top-p 1 --stream
```

## Results

Target-only comparison now includes the matched IQ4_XS c16/128-token benchmark
shape used for the MTP fix loop. The strict target-only rows use `CACHE_RAM=0`
to disable prompt-cache reuse, matching the per-head MTP runtime constraint. The
same c16/128 target-only shape with default prompt cache enabled measured
586.57 tok/s. The older refreshed c16/256 target-only run was 482.68 tok/s and
remains useful as a longer-generation reference, but 595.89 tok/s is the fair
c16/128 baseline for the per-head MTP row below.

Low-concurrency matrix:

| Profile | c1 tok/s | c4 tok/s | c8 tok/s | c16 tok/s | MTP/target ratio |
|---|---:|---:|---:|---:|---:|
| IQ4_XS target only, `CACHE_RAM=0` | 239.91 | 431.95 | 470.81 | 595.89 | n/a |
| IQ4_XS + Q6_K trained MTP draft, `n_max=2`, per-head contexts | 235.99 | 245.28 | 247.82 | 288.67 | 98.37% / 56.79% / 52.64% / 48.44% |
| IQ4_XS + Q6_K trained MTP draft, `n_max=2`, efficient sequential target verification | 192.48 | 165.63 | 167.47 | 169.24 | 80.23% / 38.34% / 35.57% / 28.40% |

The low-concurrency per-head MTP server run ended with 7,403 generated
speculative tokens and 4,031 accepted speculative tokens, or 54.45% aggregate
acceptance across the c1/c4/c8 prompt mix. Its acceptance by position was
`(0.622, 0.465)`. The c16/128 fix-loop run remains the best apples-to-apples
MTP acceptance measurement for this profile at 67.19%.

Deterministic output probe:

| Profile | Result versus target-only temperature-0 baseline | Notes |
|---|---|---|
| Target-only repeat | exact match, 3/3 prompts | Confirms the baseline is stable across server restarts. |
| Q6_K MTP, `n_max=2`, per-head contexts | mismatch, 2/3 prompts | Prompt 1 matched; prompts 2 and 3 diverged. |
| Q6_K MTP, `n_max=2`, standard contexts | mismatch, 2/3 prompts | Divergence is not caused by the per-head-context patch. |
| Q6_K MTP, `n_max=1` | mismatch, 2/3 prompts | Divergence appears even with one-token speculation. |
| Q6_K MTP, `n_max=2`, flash-attn off | mismatch, 2/3 prompts | Flash-attn is not the sole cause. |
| Q6_K MTP, `n_max=2`, `LLAMA_SPEC_VERIFY_SEQUENTIAL=1` | exact match, 3/3 prompts | Efficient sequential target verification avoids the older full checkpoint replay path. |

Sequential verification is a correctness/debug mode, not a production speed
profile. The older checkpoint-replay implementation measured 70.22 tok/s at
c1/256. The current efficient implementation measured 192.48 tok/s at c1/256,
165.63 tok/s at c4/256, 167.47 tok/s at c8/256, and 169.24 tok/s at c16/128.
The matched target-only no-prompt-cache profile measured
239.91/431.95/470.81/595.89 tok/s.

| Profile | Approx. loaded VRAM GiB | Requests | Output tokens | TPS | p95 TTFT ms | Draft acceptance |
|---|---:|---:|---:|---:|---:|---:|
| IQ4_XS target only, matched c16/128, `CACHE_RAM=0` | 19.34 | 128 | 15,742 | 595.89 | 494.5 | n/a |
| IQ4_XS target only, matched c16/128, default cache | 19.34 | 128 | 15,745 | 586.57 | 484.8 | n/a |
| IQ4_XS target only, refreshed c16/256 reference | 19.34 | 32 | 5,936 | 482.68 | 566.5 | n/a |
| IQ4_XS + Q6_K trained MTP draft, `n_max=2`, per-head contexts | ~25.5 | 128 | 15,656 | 288.67 | 845.1 | 67.19% |
| IQ4_XS + Q6_K trained MTP draft, `n_max=2`, per-head contexts, non-batched deferred replay | ~25.5 | 128 | 15,656 | 275.89 | 530.4 | 65.74% |
| IQ4_XS + Q6_K h2r10prefix head-1 refinement, step 250, `n_max=2`, per-head contexts | ~29.1 during benchmark | 128 | 15,656 | 284.62 | 540.2 | 65.91% |
| IQ4_XS + Q6_K h2r10prefix head-1 refinement, step 500, `n_max=2`, per-head contexts | ~29.1 during benchmark | 128 | 15,656 | 283.74 | 537.6 | 65.82% |
| IQ4_XS + Q6_K h2r10prefix head-1 refinement, step 1000, `n_max=2`, per-head contexts | ~29.1 during benchmark | 128 | 15,656 | 284.00 | 567.3 | 65.79% |
| IQ4_XS + Q6_K trained MTP draft, `n_max=2`, efficient sequential target verification | ~24.4 | 128 | 15,782 | 169.24 | 813.2 | 61.51%; exact target-only output |
| IQ4_XS + Q6_K trained MTP draft, `n_max=2` | 22.44 | 128 | 15,656 | 286.77 | 847.5 | 67.22% |
| IQ4_XS + Q5_K_M trained MTP draft, `n_max=2` | 22.16 | 128 | 15,656 | 285.32 | 834.1 | 66.61% |
| IQ4_XS + IQ4_XS trained MTP draft, `n_max=2` | 21.53 | 128 | 15,656 | 265.06 | 821.4 | 59.20% |
| IQ4_XS + Q6_K trained MTP draft, `n_max=1` | 22.44 | 128 | 15,782 | 262.76 | 801.9 | 68.12% |
| IQ4_XS + Q6_K trained MTP draft, `n_max=2`, batched deferred replay | 22.44 | 128 | 15,656 | 282.51 | 808.0 | 67.10% |
| IQ4_XS + Q6_K trained MTP draft, `n_max=3` | 22.44 | 128 | 15,572 | 270.19 | 845.3 | 58.54% |

Detailed llama.cpp draft statistics:

| Profile | Generated speculative tokens | Accepted speculative tokens | Acceptance by position | Draft timing `dur(b,g,a)` |
|---|---:|---:|---|---|
| Q6_K draft, `n_max=2`, per-head contexts | 13,269 | 8,915 | `(0.718, 0.621)` | `0.082, 1707.151, 8.103 ms` |
| Q6_K draft, `n_max=2`, per-head contexts, non-batched deferred replay | 13,435 | 8,832 | `(0.701, 0.610)` | `0.103, 1578.354, 2376.101 ms` |
| Q6_K h2r10prefix head-1 refinement, step 250, `n_max=2`, per-head contexts | 13,341 | 8,793 | `(0.693, 0.621)` | `0.080, 1756.073, 7.919 ms` |
| Q6_K h2r10prefix head-1 refinement, step 500, `n_max=2`, per-head contexts | 13,351 | 8,788 | `(0.692, 0.620)` | `0.089, 1734.566, 8.263 ms` |
| Q6_K h2r10prefix head-1 refinement, step 1000, `n_max=2`, per-head contexts | 13,355 | 8,786 | `(0.692, 0.620)` | `0.090, 1763.843, 8.316 ms` |
| Q6_K draft, `n_max=2`, efficient sequential target verification, c16/128 | 14,001 | 8,612 | `(0.683, 0.540)` | `0.084, 4252.843, 8.433 ms` |
| Q6_K draft, `n_max=2` | 13,265 | 8,917 | `(0.719, 0.621)` | `0.085, 1677.501, 8.474 ms` |
| Q5_K_M draft, `n_max=2` | 13,335 | 8,882 | `(0.715, 0.613)` | `0.078, 1646.867, 8.585 ms` |
| IQ4_XS draft, `n_max=2` | 14,317 | 8,475 | `(0.640, 0.541)` | `0.080, 1745.092, 8.985 ms` |
| Q6_K draft, `n_max=1` | 9,286 | 6,326 | `(0.681)` | `0.101, 1125.711, 11.183 ms` |
| Q6_K draft, `n_max=2`, batched deferred replay | 13,279 | 8,910 | `(0.718, 0.620)` | `0.085, 3082.099, 11.888 ms` |
| Q6_K draft, `n_max=3` | 16,905 | 9,896 | `(0.708, 0.644, 0.390)` | `0.085, 2282.005, 7.449 ms` |

## Engineering Notes

The working patches add:

- loadable MTP draft construction for missing upstream MTP tensors;
- MTP training from hidden-state captures;
- chain-corrected multi-position MTP training;
- optional fast backend sampling for unambiguous draft top-k=1 sampling;
- a correctness fix for per-head MTP contexts, guarded behind
  `LLAMA_MTP_PER_HEAD_CONTEXT=1`;
- an opt-in deterministic correctness guard, `LLAMA_SPEC_VERIFY_SEQUENTIAL=1`,
  that performs sequential target verification so temperature-0 MTP output
  matches target-only output. The current implementation samples the first
  verification token from the normal target batch, then decodes only accepted
  draft tails sequentially. This replaced the older full checkpoint-replay
  debug path and raised c1/256 from 70.22 tok/s to 192.48 tok/s.

Per-head contexts are the fastest measured MTP profile, but only by about 0.7%
over the standard nmax2 Q6_K run and still far behind the matched target-only
baseline. They require an additional draft context and disable prompt-cache
reuse/checkpoint behavior in the patched runtime. Batched deferred replay was
also tested and rejected:
acceptance stayed near 67%, but throughput fell to 282.51 tok/s. `n_max=3` was
rejected because acceptance dropped to 58.54% and throughput fell to 270.19
tok/s. Draft quantization also did not improve throughput: smaller draft files
reduced memory but harmed acceptance or tied/slightly trailed the Q6_K draft.
Non-batched deferred replay was also tested on the current h2r9 Q6_K per-head
profile and rejected: throughput fell to 275.89 tok/s, while the accept/replay
phase rose to 2376.101 ms in the aggregate llama.cpp draft timing. This confirms
that accepted-row replay needs to be fused or avoided; splitting it into many
small decodes is worse than full immediate replay.
The h2r10prefix head-1 refinement sweep did not improve the runtime profile
either: even though the training run reached step 1000 with loss 0.040322,
per-step perplexity 1.0411, and training top-1 of 0.99375, the deployed
acceptance stayed near the previous second-head plateau and TPS stayed below
the h2r9 Q6_K draft.

The remaining path to a production MTP speedup is a scheduler-level change: keep
draft sampling and target verification on-GPU with less host synchronization, or
pipeline draft and target work across independent requests. Simply moving the
draft to a second GPU is not expected to help in the current sequential
llama.cpp execution path.

For the native integrated graft specifically, the next bottleneck is no longer
draft quality or the MTP tensor graft. The high-acceptance c16 rows still lose
because the speculative loop pays additional draft plus target-verification
work per batch. The recurrent graph-reuse bug is fixed, so further work should
focus on reducing verifier passes, avoiding small sequential decode steps, or
fusing/pipelining draft and target work inside the server scheduler.

## Raw Artifacts

HF raw benchmark paths:

- `benchmarks/raw/llamacpp-iq4-xs-target-nocache-c1-256.jsonl`
- `benchmarks/raw/llamacpp-iq4-xs-target-nocache-c4-256.jsonl`
- `benchmarks/raw/llamacpp-iq4-xs-target-nocache-c8-256.jsonl`
- `benchmarks/raw/llamacpp-iq4-xs-target-nocache-lowc-server.log`
- `benchmarks/raw/llamacpp-iq4-xs-target-nocache-c16-128-r128.jsonl`
- `benchmarks/raw/llamacpp-iq4-xs-target-nocache-c16-128-server.log`
- `benchmarks/raw/llamacpp-iq4-xs-target-c16-128-r128.jsonl`
- `benchmarks/raw/llamacpp-iq4-xs-target-c16-128-server.log`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-perhead-q6-nmax2-c1-256.jsonl`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-perhead-q6-nmax2-c4-256.jsonl`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-perhead-q6-nmax2-c8-256.jsonl`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-perhead-q6-nmax2-lowc-server.log`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-nmax2-perhead-backend-topk1-c16-128-temp0.jsonl`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-nmax2-perhead-backend-topk1-server.log`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-h2r9-q6-nmax2-perhead-defer-c16-128-temp0.jsonl`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-h2r9-q6-nmax2-perhead-defer-c16-128-server.log`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-h2r10prefix250-q6k-c16-128-temp0.jsonl`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-h2r10prefix250-q6k-c16-128-server.log`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-h2r10prefix500-q6k-c16-128-temp0.jsonl`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-h2r10prefix500-q6k-c16-128-server.log`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-h2r10prefix1000-q6k-c16-128-temp0.jsonl`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-h2r10prefix1000-q6k-c16-128-server.log`
- `benchmarks/raw/llamacpp-wang-ornith-q6-mtp-ar-c16-128-temp0.jsonl`
- `benchmarks/raw/llamacpp-wang-ornith-q6-mtp-ar-c16-128-server.log`
- `benchmarks/raw/llamacpp-wang-ornith-q6-mtp-nmax1-c16-128-temp0.jsonl`
- `benchmarks/raw/llamacpp-wang-ornith-q6-mtp-nmax1-c16-128-server.log`
- `benchmarks/raw/quantize-iq4xs-mtp-graft-headq6.log`
- `benchmarks/raw/llamacpp-iq4xs-mtp-graft-headq6-ar-c16-128-temp0.jsonl`
- `benchmarks/raw/llamacpp-iq4xs-mtp-graft-headq6-ar-c16-128-server.log`
- `benchmarks/raw/llamacpp-iq4xs-mtp-graft-headq6-nmax1-c16-128-temp0.jsonl`
- `benchmarks/raw/llamacpp-iq4xs-mtp-graft-headq6-nmax1-c16-128-server.log`
- `benchmarks/raw/llamacpp-iq4xs-mtp-graft-headq6-ar-rollbackdst2-c16-128.jsonl`
- `benchmarks/raw/llamacpp-iq4xs-mtp-graft-headq6-ar-rollbackdst2-c16-128-server.log`
- `benchmarks/raw/llamacpp-iq4xs-mtp-graft-headq6-nmax1-fast-topk1-rollbackdst-c16-128.jsonl`
- `benchmarks/raw/llamacpp-iq4xs-mtp-graft-headq6-nmax1-fast-topk1-rollbackdst-c16-128-server.log`
- `benchmarks/raw/llamacpp-iq4xs-mtp-graft-headq6-nmax2-fast-topk1-rollbackdst-c16-128.jsonl`
- `benchmarks/raw/llamacpp-iq4xs-mtp-graft-headq6-nmax2-fast-topk1-rollbackdst-c16-128-server.log`
- `benchmarks/raw/llamacpp-iq4xs-mtp-graft-headq6-nmax2-fast-topk1-tracev-rollbackdst-lv3-c2-32.jsonl`
- `benchmarks/raw/llamacpp-iq4xs-mtp-graft-headq6-nmax2-fast-topk1-tracev-rollbackdst-lv3-c2-32-server.log`
- `benchmarks/raw/llamacpp-iq4xs-mtp-graft-headq6-nmax2-fast-topk1-proctime-c16-128.jsonl`
- `benchmarks/raw/llamacpp-iq4xs-mtp-graft-headq6-nmax2-fast-topk1-proctime-c16-128-server.log`
- `benchmarks/raw/llamacpp-iq4xs-mtp-graft-headq6-ar-proctime-trace-c16-32.jsonl`
- `benchmarks/raw/llamacpp-iq4xs-mtp-graft-headq6-ar-proctime-trace-c16-32-server.log`
- `benchmarks/raw/llamacpp-iq4xs-mtp-graft-headq6-nmax2-fast-topk1-proctime-trace-c16-32.jsonl`
- `benchmarks/raw/llamacpp-iq4xs-mtp-graft-headq6-nmax2-fast-topk1-proctime-trace-c16-32-server.log`
- `benchmarks/raw/llamacpp-iq4xs-mtp-graft-headq6-nmax2-batchdepth-trace-c16-32.jsonl`
- `benchmarks/raw/llamacpp-iq4xs-mtp-graft-headq6-nmax2-batchdepth-trace-c16-32-server.log`
- `benchmarks/raw/llamacpp-iq4xs-mtp-graft-headq6-nmax2-targetbackend-c16-128.jsonl`
- `benchmarks/raw/llamacpp-iq4xs-mtp-graft-headq6-nmax2-targetbackend-c16-128-server.log`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-seqverify-c1-256.jsonl`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-seqverify-fast-c1-256.jsonl`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-seqverify-fast-c4-256.jsonl`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-seqverify-fast-c8-256.jsonl`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-seqverify-fast-c16-128-r128.jsonl`
- `benchmarks/raw/ornith-iq4-xs-target-det-capture.json`
- `benchmarks/raw/ornith-iq4-xs-target-det-repeat-capture.json`
- `benchmarks/raw/ornith-iq4-xs-target-det-repeat-compare.json`
- `benchmarks/raw/ornith-iq4-xs-mtp-perhead-det-capture.json`
- `benchmarks/raw/ornith-iq4-xs-mtp-perhead-det-compare.json`
- `benchmarks/raw/ornith-iq4-xs-mtp-standard-det-capture.json`
- `benchmarks/raw/ornith-iq4-xs-mtp-standard-det-compare.json`
- `benchmarks/raw/ornith-iq4-xs-mtp-nmax1-det-capture.json`
- `benchmarks/raw/ornith-iq4-xs-mtp-nmax1-det-compare.json`
- `benchmarks/raw/ornith-iq4-xs-mtp-no-fa-det-capture.json`
- `benchmarks/raw/ornith-iq4-xs-mtp-no-fa-det-compare.json`
- `benchmarks/raw/ornith-iq4-xs-mtp-seqverify-det-capture.json`
- `benchmarks/raw/ornith-iq4-xs-mtp-seqverify-det-compare.json`
- `benchmarks/raw/ornith-iq4-xs-mtp-seqverify-fast-det-capture.json`
- `benchmarks/raw/ornith-iq4-xs-mtp-seqverify-fast-det-compare.json`
- `benchmarks/raw/llamacpp-iq4-xs-target-only-refresh-c16-256.jsonl`
- `benchmarks/raw/llamacpp-iq4-xs-target-only-refresh-server.log`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-nmax2-fast-backend-topk1-c16-128-temp0.jsonl`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-nmax2-fast-backend-topk1-server.log`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-nmax2-draftq5-fast-topk1-c16-128-temp0.jsonl`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-nmax2-draftq5-fast-topk1-server.log`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-nmax2-draftiq4-fast-topk1-c16-128-temp0.jsonl`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-nmax2-draftiq4-fast-topk1-server.log`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-nmax1-q6-fast-topk1-c16-128-temp0.jsonl`
- `benchmarks/raw/llamacpp-iq4-xs-mtp-nmax1-q6-fast-topk1-server.log`

Local source artifact paths:

- `runs/llamacpp-iq4-xs-target-nocache-c1-256.jsonl`
- `runs/llamacpp-iq4-xs-target-nocache-c4-256.jsonl`
- `runs/llamacpp-iq4-xs-target-nocache-c8-256.jsonl`
- `runs/llamacpp-iq4-xs-target-nocache-lowc-server.log`
- `runs/llamacpp-iq4-xs-target-nocache-c16-128-r128.jsonl`
- `runs/llamacpp-iq4-xs-target-nocache-c16-128-server.log`
- `runs/llamacpp-iq4-xs-target-c16-128-r128.jsonl`
- `runs/llamacpp-iq4-xs-target-c16-128-server.log`
- `runs/llamacpp-iq4-xs-mtp-perhead-q6-nmax2-c1-256.jsonl`
- `runs/llamacpp-iq4-xs-mtp-perhead-q6-nmax2-c4-256.jsonl`
- `runs/llamacpp-iq4-xs-mtp-perhead-q6-nmax2-c8-256.jsonl`
- `runs/llamacpp-iq4-xs-mtp-perhead-q6-nmax2-lowc-server.log`
- `runs/llamacpp-iq4-xs-mtp-nmax2-perhead-backend-topk1-c16-128-temp0.jsonl`
- `runs/llamacpp-iq4-xs-mtp-nmax2-perhead-backend-topk1-server.log`
- `runs/llamacpp-iq4-xs-mtp-h2r9-q6-nmax2-perhead-defer-c16-128-temp0.jsonl`
- `runs/server-mtp-h2r9-q6-nmax2-perhead-defer-c16-128.log`
- `runs/llamacpp-iq4-xs-mtp-h2r10prefix250-q6k-c16-128-temp0.jsonl`
- `runs/server-mtp-h2r10prefix250-c16-128.log`
- `runs/llamacpp-iq4-xs-mtp-h2r10prefix500-q6k-c16-128-temp0.jsonl`
- `runs/server-mtp-h2r10prefix500-c16-128.log`
- `runs/llamacpp-iq4-xs-mtp-h2r10prefix1000-q6k-c16-128-temp0.jsonl`
- `runs/server-mtp-h2r10prefix1000-c16-128.log`
- `runs/llamacpp-iq4-xs-mtp-seqverify-fast-c1-256.jsonl`
- `runs/llamacpp-iq4-xs-mtp-seqverify-fast-c4-256.jsonl`
- `runs/llamacpp-iq4-xs-mtp-seqverify-fast-c8-256.jsonl`
- `runs/llamacpp-iq4-xs-mtp-seqverify-fast-c16-128-r128.jsonl`
- `runs/ornith-iq4-xs-mtp-seqverify-fast-det-capture.json`
- `runs/ornith-iq4-xs-mtp-seqverify-fast-det-compare.json`
- `runs/llamacpp-iq4-xs-target-only-refresh-c16-256.jsonl`
- `runs/llamacpp-iq4-xs-target-only-refresh-server.log`
- `runs/llamacpp-iq4-xs-mtp-nmax2-fast-backend-topk1-c16-128-temp0.jsonl`
- `runs/llamacpp-iq4-xs-mtp-nmax2-fast-backend-topk1-server.log`
- `runs/llamacpp-iq4-xs-mtp-nmax2-draftq5-fast-topk1-c16-128-temp0.jsonl`
- `runs/llamacpp-iq4-xs-mtp-nmax2-draftq5-fast-topk1-server.log`
- `runs/llamacpp-iq4-xs-mtp-nmax2-draftiq4-fast-topk1-c16-128-temp0.jsonl`
- `runs/llamacpp-iq4-xs-mtp-nmax2-draftiq4-fast-topk1-server.log`
- `runs/llamacpp-iq4-xs-mtp-nmax1-q6-fast-topk1-c16-128-temp0.jsonl`
- `runs/llamacpp-iq4-xs-mtp-nmax1-q6-fast-topk1-server.log`