# MTP KLD Evaluation — 2026-06-28

Source data for the dedicated MTP KL-divergence evaluation referenced by the
model card. Numbers are transcribed verbatim; full precision is preserved in all
tables.

## Backend / config

Local llama.cpp CUDA server, `tp=1`, **one RTX PRO 6000 Blackwell 96GB**,
`CTX_SIZE=8192`, `PARALLEL=1`, `CACHE_RAM=0`, `REASONING=off`.

llama.cpp build/commit: TODO (not pinned for this eval).

## 2a. Next-token top-64 KLD — `KL(P_bf16 || P_candidate)`

32 coding prompts, `n_predict=1, temperature=-1, n_probs=64`.

| Candidate | Paired | Mean (nats) | Mean (bits) | P50 (nats) | P95 (nats) | Max (nats) | Top-1 |
|---|---:|---:|---:|---:|---:|---:|---:|
| Integrated IQ4_XS-MTP graft, target-only | 32 | 0.0731382442 | 0.1055161823 | 0.0492811974 | 0.1593743642 | 0.3348258511 | 29/32 |
| Integrated IQ4_XS-MTP graft, active draft-mtp | 32 | 0.0731382442 | 0.1055161823 | 0.0492811974 | 0.1593743642 | 0.3348258511 | 29/32 |
| Active draft-mtp **vs** target-only graft | 32 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 32/32 |

Key facts:

- Active draft-mtp and target-only next-token distributions are **byte-for-byte
  identical** for the one-token API-visible probe.
- Graft vs plain IQ4_XS body: **0.0731382442 vs 0.1425748206 → 48.70% lower mean
  KLD**, and top-1 improves 27/32 → 29/32.

## 2b. Sequence-level MTP probe

First 8 coding prompts, `n_predict=64, temperature=-1, n_probs=64`, target-only
baseline vs active native draft-mtp.

| Runtime | Client agg tok/s | Server decode tok/s | Draft acceptance | Exact 64-tok seqs | Token-position match |
|---|---:|---|---|---:|---|
| Target-only graft | 172.57 | ~210 (per-request timing) | n/a | 8/8 baseline | n/a |
| Fast active draft-mtp, `n_max=2` | 233.81 | 325.70 (/metrics) | 310/378 = 82.01%; per-position (0.884, 0.747) | 6/8 | 478/512 = 93.36% |
| `LLAMA_SPEC_VERIFY_SEQUENTIAL=1` | 164.34 | 204.88 (/metrics) | 313/375 = 83.47%; per-position (0.884, 0.772) | 5/8 | 455/512 = 88.87% |

First-divergence positions — **fast active MTP**: 6 of 8 sequences match all 64
tokens; the other two first diverge at positions **60** and **34**. (Sequential
verifier: divergences at 25, 43, 60 for three sequences; the rest match fully.)

Note: the **fast** verifier is here both faster *and* more target-matching than
the sequential verifier — a non-obvious result worth stating plainly rather than
smoothing over.

## 2c. Sequence logprob comparability (API limitation)

| Runtime | Paired positions | Comparable top-logprobs | Missing | Comparable same-token mean KLD (nats) | max (nats) |
|---|---:|---:|---:|---:|---:|
| Fast active draft-mtp vs target-only | 512 | 12 | 500 | 0.0011215739 | 0.0040239103 |
| Sequential verifier vs target-only | 512 | 10 | 502 | 0.0012293172 | 0.0040239103 |

llama.cpp omits candidate top-logprobs for most accepted speculative tokens, so a
naive "sequence KLD" looks **enormous** — that is an **API/logprob-availability
artifact, not model quality**. On the comparable subset where the emitted token
also matches, KLD is **≈ 0.001 nats** (near zero). Full sequence KLD cannot be
computed without changing the runtime to return target verifier logprobs for
accepted speculative tokens.

## 2d. MTP draft-head distillation provenance — teacher KL vs cached target hidden states

| Draft checkpoint | Tokens | Windows | Teacher KL | Teacher argmax top-1 | Teacher argmax top-5 | Gold top-1 |
|---|---:|---:|---:|---:|---:|---:|
| `mtp-distill-kl-step500` | 22,637 | 128 | 1.9222025748 | 0.5360250917 | 0.7881786456 | 0.4425939833 |
| `snapshots-iq4-live-accepted-a/step-1000` | 58,754 | 413 | 1.5038724942 | 0.6876808388 | 0.8368451510 | 0.6764816013 |
| `snapshots-iq4-live-allrows-b/step-1000` | 32,264 | 2,048 | 0.7591610373 | 0.8908070915 | 0.9389722291 | 0.8076183982 |

The draft head was trained and **measurably improved** across checkpoints —
Teacher KL fell 1.92 → 1.50 → 0.76 and teacher argmax top-1 rose 0.54 → 0.69 →
0.89. These artifacts live under `artifacts/mtp/` locally; referenced by name as
provenance (not assumed committed in the HF repo).