# MTP Runtime Code Comparison: QuinsZouls llama-next vs Clean MTP llama.cpp

This appendix explains the most likely reason QuinsZouls `llama-next` ran faster than the clean MTP llama.cpp branch in the Qwen3.6-35B-A3B MTP benchmark. It is intended as teaching material: it starts from the mechanics of MTP, then maps those mechanics to the relevant code paths and benchmark counters.

## Source Repositories and Commits

The code references in this appendix link to fixed GitHub commits so line numbers remain stable.

- QuinsZouls runtime: [`QuinsZouls/llama-cpp-turboquant`, commit `1e5a46d`](https://github.com/QuinsZouls/llama-cpp-turboquant/tree/1e5a46dccb8dd9b8d52817030cf4e334f76a8725)
- Original TurboQuant runtime: [`TheTom/llama-cpp-turboquant`, commit `5aeb2fd`](https://github.com/TheTom/llama-cpp-turboquant/tree/5aeb2fdbe26cd4c534c6fa15de73cb5749bd0403)
- Clean MTP runtime: [`am17an/llama.cpp`, commit `e7b4848`](https://github.com/am17an/llama.cpp/tree/e7b4848151377395b1693d326d1cda3fcd61c2d9)
- Atomic reference runtime: [`AtomicBot-ai/atomic-llama-cpp-turboquant`, commit `0a635dc`](https://github.com/AtomicBot-ai/atomic-llama-cpp-turboquant/tree/0a635dcd92ba66c75fccfef91c3e106f4668f367)


## Executive Conclusion

The QuinsZouls runtime appears faster primarily because of its MTP execution strategy, not because Turbo KV compression was faster in isolation.

In the completed `q8_0/q8_0` runs, Quins was already much faster than clean MTP without changing the requested KV type. That means the largest effect cannot be attributed to `turbo3` or `turbo4` V-cache compression. The code comparison points to three stronger causes:

1. Quins uses a more direct target-to-MTP hidden-state handoff inside the target context decode path.
2. Quins loads the MTP head through a dedicated `qwen35moe_mtp` architecture override instead of treating MTP as a context type over the main Qwen architecture.
3. Quins drafts greedily with `top_k = 1` and appears to fill the configured draft budget more consistently, while clean MTP uses `top_k = 10` plus a `p_min` confidence gate that can stop drafting early.

The benchmark counters match that interpretation: Quins has a lower draft acceptance rate, but it attempts many more drafts and accepts more speculative tokens per generated token.

## Benchmark Symptom That Needs Explaining

The strongest apples-to-apples comparison is `q8_0/q8_0`, because it avoids Turbo KV as the differentiator.

| config | wall tok/s | decode tok/s | draft tokens | accepted draft tokens | predicted tokens | draft / predicted | accepted / predicted | accept rate |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| Q4 clean q8/q8 | 45.55 | 49.70 | 43,794 | 43,193 | 82,229 | 0.533 | 0.525 | 0.986 |
| Q4 Quins q8/q8 | 62.35 | 69.38 | 67,344 | 48,585 | 82,531 | 0.816 | 0.589 | 0.721 |
| IQ3 clean q8/q8 | 42.20 | 45.14 | 46,024 | 45,394 | 86,189 | 0.534 | 0.527 | 0.986 |
| IQ3 Quins q8/q8 | 53.31 | 58.01 | 69,330 | 49,785 | 84,720 | 0.818 | 0.588 | 0.718 |

This is the important pattern:

- Clean MTP is conservative: fewer draft attempts, very high acceptance.
- Quins MTP is aggressive: more draft attempts, lower acceptance, but more accepted speculative tokens overall.
- Quins wins wall throughput even when the requested KV cache is still `q8_0/q8_0`.

## Functional Additions vs Original TurboQuant

The original TheTom TurboQuant fork and the Quins fork are not equivalent runtimes with only different defaults. TheTom's local runtime is primarily a TurboQuant key-value cache implementation: it exposes compressed cache formats such as `turbo2`, `turbo3`, and `turbo4`, but it does not expose the Qwen multi-token prediction path used in this report. The local TheTom binary help listed:

```text
--spec-type [none|ngram-cache|ngram-simple|ngram-map-k|ngram-map-k4v|ngram-mod]
```

It did not list `mtp`, `draft-mtp`, or `nextn`. Source inspection likewise did not find the Qwen MTP-specific plumbing used by Quins, such as `COMMON_SPECULATIVE_TYPE_MTP`, `llama_set_mtp`, `handle_mtp_for_ubatch`, `qwen35moe_mtp`, or `qwen35_mtp`. This is why TheTom is recorded as a capability-incompatible runtime for the Qwen3.6 MTP benchmark, even though it is useful for TurboQuant key-value cache experiments.

Quins adds several functional layers on top of the original TurboQuant-style cache work:

| Functional addition | Original TheTom TurboQuant | QuinsZouls llama-next TurboQuant | Why it matters |
|---|---|---|---|
| Qwen MTP speculative mode | Not exposed by the local runtime's `--spec-type` help. | Adds `COMMON_SPECULATIVE_TYPE_MTP` and maps the mode to `--spec-type mtp`. | Makes the Qwen3.6 embedded multi-token prediction head callable from the server. |
| Dedicated MTP architecture loading | No Qwen MTP architecture override found in the inspected local source. | Loads the same GGUF as `qwen35moe_mtp` or `qwen35_mtp` using `override_arch`. | Lets the runtime construct a narrow MTP-head view of the model instead of treating the whole trunk as the draft path. |
| Direct MTP context registration | No `llama_set_mtp` path found in the inspected local source. | Registers the MTP context on the target context with `llama_set_mtp`. | Gives the target decode path a direct pointer to the MTP context. |
| Hidden-state handoff | Original TurboQuant does not implement the Qwen MTP hidden-state handoff used here. | Calls `handle_mtp_for_ubatch` during target decode and copies hidden-state rows into the MTP context. | Reduces orchestration overhead and keeps the MTP side context aligned with target decode. |
| Greedy MTP drafting | Generic speculative sampler uses a top-10 draft sampler. | MTP sampler uses top-1 proposals. | Makes the MTP draft path deterministic and more aggressive for the two-token draft budget. |
| MTP key-value synchronization | Not applicable without the Qwen MTP context. | Mirrors sequence removal into the MTP context and rolls back rejected draft tokens. | Prevents the target and MTP caches from drifting after partial acceptance or rejection. |
| GQA-aware asymmetric cache policy | Original TurboQuant provides Turbo key-value formats, but the inspected local version did not show the same high-GQA auto-upgrade policy. | Detects high grouped query attention ratios and can upgrade symmetric `turbo*/turbo*` requests to `q8_0/turbo*`. | Protects key precision on models where one key/value head serves many query heads. |

GQA means grouped query attention. In grouped query attention, several query heads share a smaller number of key/value heads. For example, if a model has 28 query heads and 4 key/value heads, each key/value head is reused by 7 query heads. That reuse makes key-cache error more dangerous: a badly compressed key can perturb multiple attention comparisons, not just one. The Quins auto-asymmetric policy is a safety rule for that situation. When a high grouped-query-attention ratio is detected and the user asks for symmetric Turbo key/value cache compression, Quins can keep the key cache at `q8_0` while still compressing the value cache with `turbo3` or `turbo4`.

The practical recommendation is therefore to describe the benchmark result as a Quins-specific runtime result, not a pure TurboQuant key-value cache result. TurboQuant key-value compression is part of the stack, but the rows in this report also depend on Quins' Qwen MTP mode, MTP architecture override, target-to-MTP hidden-state hook, greedy draft sampler, and cache synchronization behavior.

## Brief Intro: What MTP Needs

For this model, MTP is a small extra prediction head attached to the model. It is not a separate tiny draft model in the usual speculative decoding sense. To propose future tokens, the MTP head needs:

1. The current or previous token id.
2. The target model's internal hidden state at that position.

The hidden state is a vector representing the target model's internal understanding after processing the sequence. The main implementation challenge is therefore: how does the runtime move the target model's hidden state into the MTP head quickly and correctly?

Both runtimes implement that concept. They differ in how direct the handoff is.

## Side-by-Side Flow

### Clean MTP Flow

The clean branch routes MTP through the general speculative decoding system.

```text
target decode batch
  -> expose pre-norm hidden states
  -> server calls common_speculative_process()
  -> speculative MTP code copies hidden states into draft/MTP batch
  -> draft MTP context decodes that batch
  -> later common_speculative_draft() proposes future tokens
```

Relevant code:

- [`server-context.cpp:3002`](https://github.com/am17an/llama.cpp/blob/e7b4848151377395b1693d326d1cda3fcd61c2d9/tools/server/server-context.cpp#L3002)
  - calls `common_speculative_process(spec.get(), batch_view)` after target decode.
- [`speculative.cpp:380`](https://github.com/am17an/llama.cpp/blob/e7b4848151377395b1693d326d1cda3fcd61c2d9/common/speculative.cpp#L380)
  - implements `common_speculative_state_draft_mtp`.
- [`speculative.cpp:422`](https://github.com/am17an/llama.cpp/blob/e7b4848151377395b1693d326d1cda3fcd61c2d9/common/speculative.cpp#L422)
  - enables `llama_set_embeddings_pre_norm(ctx_tgt, true)`.
- [`speculative.cpp:501`](https://github.com/am17an/llama.cpp/blob/e7b4848151377395b1693d326d1cda3fcd61c2d9/common/speculative.cpp#L501)
  - reads target hidden states with `llama_get_embeddings_pre_norm(ctx_tgt)`.
- [`speculative.cpp:538`](https://github.com/am17an/llama.cpp/blob/e7b4848151377395b1693d326d1cda3fcd61c2d9/common/speculative.cpp#L538)
  - stores the last target hidden-state row for the next MTP step.

The clean branch is more framework-oriented. It supports multiple speculative implementations and sequence-oriented bookkeeping. That makes it flexible, but it creates more places for synchronization, copying, and conservative gating.

### QuinsZouls Flow

Quins registers the MTP context directly on the target context.

```text
target decode ubatch
  -> target context hook sees hidden-state tensor
  -> hook feeds hidden-state rows into MTP context
  -> MTP context remains aligned with target decode
  -> common_speculative_draft() runs a short greedy MTP autoregressive loop
```

Relevant code:

- [`speculative.cpp:639`](https://github.com/QuinsZouls/llama-cpp-turboquant/blob/1e5a46dccb8dd9b8d52817030cf4e334f76a8725/common/speculative.cpp#L639)
  - calls `llama_set_mtp(ctx_tgt, ctx_mtp)`.
- [`llama-context.cpp:1249`](https://github.com/QuinsZouls/llama-cpp-turboquant/blob/1e5a46dccb8dd9b8d52817030cf4e334f76a8725/src/llama-context.cpp#L1249)
  - calls `handle_mtp_for_ubatch(...)` during target decode.
- [`llama-context.cpp:3430`](https://github.com/QuinsZouls/llama-cpp-turboquant/blob/1e5a46dccb8dd9b8d52817030cf4e334f76a8725/src/llama-context.cpp#L3430)
  - implements the target-to-MTP hidden-state handoff.
- [`speculative.cpp:713`](https://github.com/QuinsZouls/llama-cpp-turboquant/blob/1e5a46dccb8dd9b8d52817030cf4e334f76a8725/common/speculative.cpp#L713)
  - uses `llama_context_get_t_mtp_out(ctx_mtp)` for later autoregressive MTP draft steps.

This is what we mean by a "tighter hidden-state hook": the MTP handoff is embedded inside the target context's decode machinery, rather than being a separate speculative-processing step after the target decode.

## Code Difference 1: Dedicated MTP Architecture vs MTP Context Type

Quins loads a dedicated architecture override for the MTP head.

```cpp
if (std::string(trunk_arch) == "qwen35moe") {
    mtp_arch = "qwen35moe_mtp";
} else if (std::string(trunk_arch) == "qwen35") {
    mtp_arch = "qwen35_mtp";
}

auto mparams_mtp = common_model_params_to_llama(params_base);
mparams_mtp.override_arch = mtp_arch;
model_mtp.reset(llama_model_load_from_file(params_base.model.path.c_str(), mparams_mtp));
```

Source: [`server-context.cpp:822-844`](https://github.com/QuinsZouls/llama-cpp-turboquant/blob/1e5a46dccb8dd9b8d52817030cf4e334f76a8725/tools/server/server-context.cpp#L822-L844)

The MTP-specific Qwen MoE architecture then marks the MTP context as KV-only for the NextN/MTP layers:

```cpp
// only the MTP layers get a KV cache, trunk layers are skipped.
hparams.kv_only_nextn         = true;
hparams.n_layer_kv_from_start = -1;
```

Source: [`qwen35moe_mtp.cpp:14-16`](https://github.com/QuinsZouls/llama-cpp-turboquant/blob/1e5a46dccb8dd9b8d52817030cf4e334f76a8725/src/models/qwen35moe_mtp.cpp#L14-L16)

Clean MTP creates an MTP context against the target model with a context type:

```cpp
auto cparams_mtp = common_context_params_to_llama(params_base);
cparams_mtp.ctx_type = LLAMA_CONTEXT_TYPE_MTP;
ctx_dft.reset(llama_init_from_model(model_tgt, cparams_mtp));
```

Source: [`server-context.cpp:763-772`](https://github.com/am17an/llama.cpp/blob/e7b4848151377395b1693d326d1cda3fcd61c2d9/tools/server/server-context.cpp#L763-L772)

Teaching interpretation:

- Clean says: "Use the target model with an MTP context mode."
- Quins says: "Load a separate MTP-only view of the model architecture."

The Quins approach gives the runtime a narrower graph for the MTP head and a clearer separation between trunk execution and MTP execution.

## Code Difference 2: Hidden-State Transfer

Clean exposes pre-norm embeddings/hidden states as an output buffer and lets the speculative module consume them.

```cpp
llama_set_embeddings_pre_norm(ctx_tgt, true);
llama_set_embeddings_pre_norm(ctx_dft, true);
```

Source: [`speculative.cpp:422-423`](https://github.com/am17an/llama.cpp/blob/e7b4848151377395b1693d326d1cda3fcd61c2d9/common/speculative.cpp#L422-L423)

Then the MTP speculative code reads those rows:

```cpp
const float * h_tgt = llama_get_embeddings_pre_norm(ctx_tgt);
std::memcpy(batch.embd + (size_t) 1 * n_embd, h_tgt, row_bytes * (n_tokens-1));
```

Source: [`speculative.cpp:501-502`](https://github.com/am17an/llama.cpp/blob/e7b4848151377395b1693d326d1cda3fcd61c2d9/common/speculative.cpp#L501-L502)

Quins instead registers the MTP context on the target and invokes a per-ubatch hook during target decode:

```cpp
llama_set_mtp(ctx_tgt, ctx_mtp);
```

Source: [`speculative.cpp:639`](https://github.com/QuinsZouls/llama-cpp-turboquant/blob/1e5a46dccb8dd9b8d52817030cf4e334f76a8725/common/speculative.cpp#L639)

The target context later calls:

```cpp
handle_mtp_for_ubatch(..., res->t_h_pre_norm);
```

Source: [`llama-context.cpp:1249`](https://github.com/QuinsZouls/llama-cpp-turboquant/blob/1e5a46dccb8dd9b8d52817030cf4e334f76a8725/src/llama-context.cpp#L1249)

Teaching interpretation:

- Clean exports hidden states and asks the speculative subsystem to process them.
- Quins mirrors hidden states to the MTP context as part of the target decode flow.

That does not change the mathematical purpose of MTP. It changes runtime placement. The Quins path appears more direct and specialized, which plausibly reduces host-side orchestration overhead.

## Code Difference 3: Draft Sampler and Draft Bucket Behavior

Both runs were configured with `--spec-draft-n-max 2`. That means the MTP path is allowed to propose up to two future tokens at a time. It does not mean every runtime always proposes two tokens.

Quins initializes the MTP draft sampler as top-1:

```cpp
sparams.top_k    = 1;
sparams.samplers = { COMMON_SAMPLER_TYPE_TOP_K };
```

Source: [`speculative.cpp:626-627`](https://github.com/QuinsZouls/llama-cpp-turboquant/blob/1e5a46dccb8dd9b8d52817030cf4e334f76a8725/common/speculative.cpp#L626-L627)

Then it loops up to `n_max` and pushes the best token each time:

```cpp
for (int32_t k = 0; k < n_max; ++k) {
    ...
    const llama_token best = common_sampler_sample(smpl, ctx_mtp, 0);
    common_sampler_accept(smpl, best, false);
    draft_tokens.push_back(best);
}
```

Source: [`speculative.cpp:698-738`](https://github.com/QuinsZouls/llama-cpp-turboquant/blob/1e5a46dccb8dd9b8d52817030cf4e334f76a8725/common/speculative.cpp#L698-L738)

Clean initializes MTP draft sampling with top-10:

```cpp
sparams.top_k    = 10;
sparams.samplers = { COMMON_SAMPLER_TYPE_TOP_K };
```

Source: [`speculative.cpp:417-418`](https://github.com/am17an/llama.cpp/blob/e7b4848151377395b1693d326d1cda3fcd61c2d9/common/speculative.cpp#L417-L418)

It also has a confidence gate:

```cpp
if (cur_p->data[0].p < params.p_min) {
    drafting[seq_id] = false;
    n_drafting--;
    continue;
}
```

Source: [`speculative.cpp:610-614`](https://github.com/am17an/llama.cpp/blob/e7b4848151377395b1693d326d1cda3fcd61c2d9/common/speculative.cpp#L610-L614)

The default `p_min` is `0.75`:

```cpp
float p_min = 0.75f;
```

Source: [`common.h:305`](https://github.com/am17an/llama.cpp/blob/e7b4848151377395b1693d326d1cda3fcd61c2d9/common/common.h#L305)

Teaching interpretation:

- Quins mostly fills the two-slot draft bucket.
- Clean often stops early if the MTP head is not confident enough.

That is why clean has a much higher acceptance rate but fewer total draft attempts. Quins is willing to make more guesses. Some are wrong, but enough are right that it accepts more speculative tokens per generated token.

## Why Lower Acceptance Can Still Be Faster

Speculative decoding is not only about the percentage of drafts that are accepted. It is also about how many draft opportunities the runtime creates.

Clean accepted almost everything it drafted:

```text
Q4 clean:
43,794 draft guesses
43,193 accepted
98.6% accepted
0.525 accepted draft tokens per generated token
```

Quins accepted a smaller share but drafted far more:

```text
Q4 Quins:
67,344 draft guesses
48,585 accepted
72.1% accepted
0.589 accepted draft tokens per generated token
```

This is the central performance explanation. Quins throws away more draft tokens, but it gets more accepted speculative tokens overall.

## What We Can Reproduce

The most reproducible findings from the current data are:

1. Under the same Q4_K_M MTP model, same 64k context, same MoE offload setting, same final target sampling, and same `q8_0/q8_0` KV request, Quins was substantially faster than clean MTP.
2. The Quins speedup appears before Turbo KV is introduced, so Turbo KV should not be described as the primary cause of the headline speedup.
3. Quins generated about 0.816 draft tokens per predicted token on Q4, compared with about 0.533 for clean.
4. Quins accepted about 0.589 draft tokens per predicted token on Q4, compared with about 0.525 for clean.
5. The code path differences provide a plausible mechanism: dedicated MTP architecture, direct target-context MTP hook, and greedy/top-1 MTP drafting without the same early confidence stop.

## What We Should Not Claim Yet

We should not claim that any single code change is proven to be the sole cause. The current result is an observational comparison between two forks with multiple differences.

Do not claim:

- "Turbo KV caused the 37% speedup." The q8/q8 result contradicts that.
- "Top-1 drafting alone caused the speedup." It is likely important, but not isolated.
- "The direct hidden-state hook alone caused the speedup." It is plausible, but not isolated.
- "These numbers generalize to all GPUs or all Qwen MTP builds." They are measured on this Windows CUDA workstation.

## Recommended Follow-Up Experiments

To isolate causality, run these targeted ablations:

1. **Draft sampler ablation**
   - Change Quins MTP draft sampler from `top_k = 1` to `top_k = 10` plus `p_min = 0.75`.
   - Or change clean MTP to use top-1 and no confidence gate.
   - Expected result: if draft aggressiveness is the main cause, draft tokens per predicted token should converge.

2. **Draft budget sweep**
   - Test `--spec-draft-n-max 1`, `2`, and possibly `4`.
   - Keep final target sampling constant.
   - Expected result: Quins should benefit more from larger buckets until rejection or overhead dominates.

3. **MTP handoff profiling**
   - Use Nsight Systems to measure CPU stalls, GPU synchronization, and `ggml_backend_tensor_get` transfer time.
   - Compare target decode, MTP hook/process, and draft decode sections.
   - Expected result: Quins should show less host-side orchestration cost around hidden-state handoff.

4. **MoE offload sensitivity**
   - Sweep `--n-cpu-moe` at `30`, `32`, `34`, and `36` for both runtimes.
   - Faster RAM or lower MoE offload may change the relative advantage.

5. **Reasoning-on vs reasoning-off**
   - Current published numbers are reasoning-on.
   - Repeat with reasoning disabled to test whether token predictability changes draft acceptance and throughput.

## Recommended Public Wording

Use this wording for a public summary:

> In our Qwen3.6 MTP tests, the fastest runtime was not faster primarily because of Turbo KV. QuinsZouls llama-next was already much faster at the same `q8_0/q8_0` KV setting. Code inspection suggests its advantage comes from a more aggressive and specialized MTP path: it loads a dedicated Qwen MTP architecture, wires target hidden-state transfer into the target decode flow, and uses top-1 greedy draft proposals that fill more of the configured two-token draft budget. This lowers acceptance percentage but increases accepted speculative tokens per generated token.