MTP KLD Evaluation — 2026-06-28
Source data for the dedicated MTP KL-divergence evaluation referenced by the model card. Numbers are transcribed verbatim; full precision is preserved in all tables.
Backend / config
Local llama.cpp CUDA server, tp=1, one RTX PRO 6000 Blackwell 96GB,
CTX_SIZE=8192, PARALLEL=1, CACHE_RAM=0, REASONING=off.
llama.cpp build/commit: TODO (not pinned for this eval).
2a. Next-token top-64 KLD — KL(P_bf16 || P_candidate)
32 coding prompts, n_predict=1, temperature=-1, n_probs=64.
| Candidate | Paired | Mean (nats) | Mean (bits) | P50 (nats) | P95 (nats) | Max (nats) | Top-1 |
|---|---|---|---|---|---|---|---|
| Integrated IQ4_XS-MTP graft, target-only | 32 | 0.0731382442 | 0.1055161823 | 0.0492811974 | 0.1593743642 | 0.3348258511 | 29/32 |
| Integrated IQ4_XS-MTP graft, active draft-mtp | 32 | 0.0731382442 | 0.1055161823 | 0.0492811974 | 0.1593743642 | 0.3348258511 | 29/32 |
| Active draft-mtp vs target-only graft | 32 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 32/32 |
Key facts:
- Active draft-mtp and target-only next-token distributions are byte-for-byte identical for the one-token API-visible probe.
- Graft vs plain IQ4_XS body: 0.0731382442 vs 0.1425748206 → 48.70% lower mean KLD, and top-1 improves 27/32 → 29/32.
2b. Sequence-level MTP probe
First 8 coding prompts, n_predict=64, temperature=-1, n_probs=64, target-only
baseline vs active native draft-mtp.
| Runtime | Client agg tok/s | Server decode tok/s | Draft acceptance | Exact 64-tok seqs | Token-position match |
|---|---|---|---|---|---|
| Target-only graft | 172.57 | ~210 (per-request timing) | n/a | 8/8 baseline | n/a |
Fast active draft-mtp, n_max=2 |
233.81 | 325.70 (/metrics) | 310/378 = 82.01%; per-position (0.884, 0.747) | 6/8 | 478/512 = 93.36% |
LLAMA_SPEC_VERIFY_SEQUENTIAL=1 |
164.34 | 204.88 (/metrics) | 313/375 = 83.47%; per-position (0.884, 0.772) | 5/8 | 455/512 = 88.87% |
First-divergence positions — fast active MTP: 6 of 8 sequences match all 64 tokens; the other two first diverge at positions 60 and 34. (Sequential verifier: divergences at 25, 43, 60 for three sequences; the rest match fully.)
Note: the fast verifier is here both faster and more target-matching than the sequential verifier — a non-obvious result worth stating plainly rather than smoothing over.
2c. Sequence logprob comparability (API limitation)
| Runtime | Paired positions | Comparable top-logprobs | Missing | Comparable same-token mean KLD (nats) | max (nats) |
|---|---|---|---|---|---|
| Fast active draft-mtp vs target-only | 512 | 12 | 500 | 0.0011215739 | 0.0040239103 |
| Sequential verifier vs target-only | 512 | 10 | 502 | 0.0012293172 | 0.0040239103 |
llama.cpp omits candidate top-logprobs for most accepted speculative tokens, so a naive "sequence KLD" looks enormous — that is an API/logprob-availability artifact, not model quality. On the comparable subset where the emitted token also matches, KLD is ≈ 0.001 nats (near zero). Full sequence KLD cannot be computed without changing the runtime to return target verifier logprobs for accepted speculative tokens.
2d. MTP draft-head distillation provenance — teacher KL vs cached target hidden states
| Draft checkpoint | Tokens | Windows | Teacher KL | Teacher argmax top-1 | Teacher argmax top-5 | Gold top-1 |
|---|---|---|---|---|---|---|
mtp-distill-kl-step500 |
22,637 | 128 | 1.9222025748 | 0.5360250917 | 0.7881786456 | 0.4425939833 |
snapshots-iq4-live-accepted-a/step-1000 |
58,754 | 413 | 1.5038724942 | 0.6876808388 | 0.8368451510 | 0.6764816013 |
snapshots-iq4-live-allrows-b/step-1000 |
32,264 | 2,048 | 0.7591610373 | 0.8908070915 | 0.9389722291 | 0.8076183982 |
The draft head was trained and measurably improved across checkpoints —
Teacher KL fell 1.92 → 1.50 → 0.76 and teacher argmax top-1 rose 0.54 → 0.69 →
0.89. These artifacts live under artifacts/mtp/ locally; referenced by name as
provenance (not assumed committed in the HF repo).