# Dedicated MTP KLD Evaluation

Date: 2026-06-28

Artifact under test: `artifacts/quant/ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf`

Backend: local `llama.cpp` CUDA server, `tp=1`, one RTX PRO 6000 Blackwell
96GB, `CTX_SIZE=8192`, `PARALLEL=1`, `CACHE_RAM=0`, `REASONING=off`.

## What Was Measured

Two different checks were run.

1. Next-token top-64 KLD over 32 coding prompts, comparable with the existing
   BF16-vs-quant table. This uses `n_predict=1`, `temperature=-1`, `n_probs=64`.
2. Sequence-level 8 prompt x 64 token deterministic generations to exercise
   real native `draft-mtp` acceptance and compare active MTP against the same
   GGUF running target-only.

Important limitation: llama.cpp does not return full candidate `top_logprobs`
for most accepted speculative tokens in the sequence-level response. Those
positions are reported as missing candidate logprob positions, not as valid KLD
measurements.

## Next-Token KLD

Approximate `KL(P_bf16 || P_candidate)` over top-64 next-token distributions.

| Candidate | Paired prompts | Mean KLD nats | Mean KLD bits | P50 nats | P95 nats | Max nats | Top-1 match |
|---|---:|---:|---:|---:|---:|---:|---:|
| Integrated IQ4_XS-MTP graft, target-only | 32 | 0.0731382442 | 0.1055161823 | 0.0492811974 | 0.1593743642 | 0.3348258511 | 29/32 |
| Integrated IQ4_XS-MTP graft, active `draft-mtp` | 32 | 0.0731382442 | 0.1055161823 | 0.0492811974 | 0.1593743642 | 0.3348258511 | 29/32 |
| Active `draft-mtp` vs target-only graft | 32 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 32/32 |

The active `draft-mtp` and target-only next-token distributions are byte-for-byte
identical for this one-token API-visible probe. The integrated MTP graft's
BF16 drift is also lower than the previous plain IQ4_XS body capture:

| Candidate | Mean KLD nats | Top-1 match |
|---|---:|---:|
| Plain IQ4_XS vs BF16, prior body KLD run | 0.1425748206 | 27/32 |
| Integrated IQ4_XS-MTP graft vs BF16, this run | 0.0731382442 | 29/32 |

That is a 48.70% lower mean KLD than the prior plain IQ4_XS capture.

## Sequence-Level MTP Probe

Shape: first 8 coding eval prompts, `n_predict=64`, `temperature=-1`,
`n_probs=64`, target-only baseline vs active native `draft-mtp`.

| Runtime | Client aggregate tok/s | Server decode tok/s | Draft acceptance | Exact 64-token sequences | Token-position match |
|---|---:|---:|---:|---:|---:|
| Target-only graft | 172.57 | about 210 from server per-request timing | n/a | 8/8 baseline | n/a |
| Fast active `draft-mtp`, `n_max=2` | 233.81 | 325.70 from `/metrics` | 310/378 = 82.01%; per-position `(0.884, 0.747)` | 6/8 | 478/512 = 93.36% |
| `LLAMA_SPEC_VERIFY_SEQUENTIAL=1` | 164.34 | 204.88 from `/metrics` | 313/375 = 83.47%; per-position `(0.884, 0.772)` | 5/8 | 455/512 = 88.87% |

Fast active MTP first divergence positions:

| Source id | First mismatch position | Matched prefix tokens |
|---|---:|---:|
| `630213` | none | 64 |
| `373803` | none | 64 |
| `148184` | none | 64 |
| `631801` | none | 64 |
| `211640` | none | 64 |
| `90023` | none | 64 |
| `611646` | 60 | 60 |
| `502142` | 34 | 34 |

Sequential verifier first divergence positions:

| Source id | First mismatch position | Matched prefix tokens |
|---|---:|---:|
| `630213` | none | 64 |
| `373803` | none | 64 |
| `148184` | none | 64 |
| `631801` | none | 64 |
| `211640` | 25 | 25 |
| `90023` | 43 | 43 |
| `611646` | 60 | 60 |
| `502142` | none | 64 |

## Sequence Logprob Comparability

The active-MTP sequence response omitted candidate top-logprobs for most accepted
draft tokens.

| Runtime | Paired token positions | Positions with comparable top-logprobs | Missing candidate top-logprobs | Comparable same-token mean KLD nats | Comparable same-token max KLD nats |
|---|---:|---:|---:|---:|---:|
| Fast active `draft-mtp` vs target-only | 512 | 12 | 500 | 0.0011215739 | 0.0040239103 |
| Sequential verifier vs target-only | 512 | 10 | 502 | 0.0012293172 | 0.0040239103 |

If missing candidate top-logprobs are treated as tail mass, the apparent
sequence KLD becomes extremely large. That number is not a valid model-quality
KLD; it is an API/logprob availability artifact. The valid comparable subset is
near zero when the emitted token also matches.

## MTP Draft-Head KL Context

The meaningful KL-style metric for standalone MTP draft heads is teacher KL
against cached target hidden states, not standard base-model next-token KLD.
Previously measured draft-head evals:

| Draft checkpoint | Cache | Tokens | Windows | Teacher KL | Teacher argmax top-1 | Teacher argmax top-5 | Gold top-1 |
|---|---|---:|---:|---:|---:|---:|---:|
| `artifacts/mtp/ornith-1.0-35b-mtp-distill-kl-step500` | `artifacts/mtp/cache-coding-4096x512-merged` | 22,637 | 128 | 1.9222025748 | 0.5360250917 | 0.7881786456 | 0.4425939833 |
| `artifacts/mtp/snapshots-llamacpp-iq4-live-accepted-a/step-1000` | `artifacts/mtp/cache-llamacpp-iq4-live-accepted-selfgen1000-a` | 58,754 | 413 | 1.5038724942 | 0.6876808388 | 0.8368451510 | 0.6764816013 |
| `artifacts/mtp/snapshots-llamacpp-iq4-live-allrows-b/step-1000` | `artifacts/mtp/cache-llamacpp-iq4-live-allrows-livea1000-b` | 32,264 | 2,048 | 0.7591610373 | 0.8908070915 | 0.9389722291 | 0.8076183982 |

## Conclusion

The integrated IQ4_XS-MTP graft is clean on next-token distribution quality:
active MTP does not change the API-visible one-token distribution, and its mean
BF16 KLD is 0.0731382442 nats over the 32-prompt top-64 probe.

The longer sequence probe is more mixed. Fast active MTP gives real single-user
speedup in this 8 x 64 test and high acceptance, but it is not fully
target-equivalent over deterministic 64-token generations. The response also
omits candidate top-logprobs for most accepted draft tokens, so full sequence
KLD cannot be computed from the current llama.cpp API response without changing
the runtime to return target verifier logprobs for accepted speculative tokens.

Raw files:

- `runs/kld-native-iq4-xs-mtp-graft-target-top64-32prompts.json`
- `runs/kld-native-iq4-xs-mtp-graft-target-vs-bf16.json`
- `runs/kld-native-iq4-xs-mtp-graft-active-top64-32prompts.json`
- `runs/kld-native-iq4-xs-mtp-graft-active-vs-bf16.json`
- `runs/kld-native-iq4-xs-mtp-graft-active-vs-target-only.json`
- `runs/kld-seq-iq4-xs-mtp-graft-target-top64-8prompts-64tokens.json`
- `runs/kld-seq-iq4-xs-mtp-graft-active-top64-8prompts-64tokens.json`
- `runs/kld-seq-iq4-xs-mtp-graft-active-vs-target-only-8prompts-64tokens.json`
- `runs/kld-seq-iq4-xs-mtp-graft-seqverify-top64-8prompts-64tokens.json`
- `runs/kld-seq-iq4-xs-mtp-graft-seqverify-vs-target-only-8prompts-64tokens.json`