# IQ4_XS Integrated Graft MTP Adaptive Profile

Date: 2026-06-27

Hardware: one NVIDIA RTX PRO 6000 Blackwell Max-Q 96GB GPU, `tp=1`.

Model: `ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf`

This file was produced by requantizing the MIT-licensed
`wang-yang/Ornith-1.0-35B-MTP-GGUF` Q6_K graft to IQ4_XS while preserving the
appended `blk.40.*` MTP block at Q6_K/F16/F32.

## Structural Comparison To wang-yang

Both the wang-yang Q6_K MTP file and this IQ4_XS graft have:

| Key | Value |
|---|---|
| `general.architecture` | `qwen35moe` |
| `qwen35moe.block_count` | `41` |
| `qwen35moe.nextn_predict_layers` | `1` |
| MTP tensors | one appended `blk.40.*` block with `blk.40.nextn.*` tensors |

The difference is quantization and serving policy:

| File | Quant/layout | File size |
|---|---|---:|
| `wang-yang/Ornith-1.0-35B-MTP-GGUF` | Q6_K body + Q6_K/F16/F32 MTP block, `general.file_type=18` | 29.2 GB decimal |
| `ornith-1.0-35b-IQ4_XS.gguf` | IQ4_XS body, no MTP | 18.9 GB decimal |
| `ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf` | IQ4_XS body + preserved Q6_K/F16/F32 MTP block, `general.file_type=30` | 19.6 GB decimal |

The MTP block adds about 0.69 GB decimal over the plain IQ4_XS artifact.

## Wang-yang Reference Number

The wang-yang model card reports a single-prompt real-code-continuation tg128
benchmark on M3 Max:

| Mode | tok/s | Speedup | Acceptance | Mean accepted length |
|---|---:|---:|---:|---:|
| AR | 66.6 | 1.00x | n/a | n/a |
| `draft-mtp n_max=1` | 83.8 | 1.26x | 92.2% | 1.92 |

That is a low-concurrency MTP number. It is not a saturated multi-slot server
benchmark.

## Local Low-Concurrency Result

The local IQ4_XS graft reproduces the same low-concurrency behavior.

| Profile | Shape | tok/s | Notes |
|---|---|---:|---|
| Target-only AR | c1/128 | 221.80 | baseline |
| `draft-mtp n_max=2` | c1/128 | 279.36 | 1.26x vs AR |
| Adaptive MTP throttle | c1/128 | 319.53 | MTP active; cumulative acceptance by position `(0.953, 0.865)` |

The adaptive c1 run used four prompts. Server-side cumulative MTP stats were:

```text
#gen tokens = 339, #acc tokens = 309
#mean acc len = 2.82
#acc rate/pos = (0.953, 0.865)
```

## High-Concurrency Result

At c16 the target-only server is already GPU-saturated. Always-on active MTP
reduces target decode call count, but it increases verifier row shape and adds
draft work. That is why high acceptance does not automatically translate into a
throughput win at c16.

| Profile | Shape | tok/s | Notes |
|---|---|---:|---|
| Target-only AR | c16/64 | 568.57 | matched AR control |
| Active MTP, `n_max=2`, target+draft backend sampling | c16/64 | 323.12 | best always-on active-MTP c16 profile measured in this loop |
| Adaptive MTP throttle, `LLAMA_SPEC_MAX_DRAFTING_SLOTS=1` | c16/64 | 564.48 | MTP disabled during saturated c16; effectively matches AR |

The c16 adaptive server log confirms the throttle behavior: draft-MTP counters
were zero for saturated c16 requests, while the later c1 run on the same server
used MTP normally.

## Threshold Sweep

A later matched sweep used the same server settings, `max_tokens=64`, and
thresholds 1, 2, and 4. The goal was to see how far MTP can remain enabled
before it starts hurting multi-slot throughput.

| `LLAMA_SPEC_MAX_DRAFTING_SLOTS` | c1 tok/s | c2 tok/s | c4 tok/s | c8 tok/s | c16 tok/s |
|---:|---:|---:|---:|---:|---:|
| 1 | 271.12 | 316.46 | 424.14 | 564.24 | 592.45 |
| 2 | 267.04 | 313.25 | 422.88 | 558.17 | 592.39 |
| 4 | 269.23 | 313.06 | 328.45 | 547.56 | 591.50 |

Threshold 1 and 2 are effectively tied in this short-output sweep. Threshold 4
is too permissive: c4 drops from 424.14 tok/s to 328.45 tok/s because active
MTP is allowed too far into the batched regime. The recommended default remains
`LLAMA_SPEC_MAX_DRAFTING_SLOTS=1`; use `2` only if the deployment has known
two-request interactive bursts and accepts the small c8 tradeoff.

## Recommended Command

For a single-GPU OpenAI-compatible llama.cpp server:

```bash
export CUDA_VISIBLE_DEVICES=0
export LLAMA_SPEC_MAX_DRAFTING_SLOTS=1
export LLAMA_MTP_FAST_BACKEND_SAMPLE=1
export LLAMA_MTP_DRAFT_TOP_K=1
export LLAMA_MTP_DRAFT_TOP_P=1
export LLAMA_MTP_DRAFT_TEMP=1

llama-server \
  -m ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf \
  --alias Ornith-1.0-35B-GGUF-IQ4_XS \
  --host 127.0.0.1 --port 8002 \
  -c 8192 -np 16 -b 4096 -ub 512 -ngl 99 \
  --flash-attn on \
  --reasoning off --reasoning-format deepseek \
  --spec-type draft-mtp \
  --spec-draft-ngl 99 \
  --spec-draft-n-max 2 \
  --spec-draft-n-min 0 \
  --spec-draft-backend-sampling \
  --cont-batching \
  --metrics \
  --cache-ram 0 \
  --ctx-checkpoints 0
```

Use this profile when traffic includes low-concurrency interactive requests and
bursty high-concurrency batches. Leave MTP always on only for investigation,
because always-on active MTP is still slower than target-only on saturated c16.

## Raw Local Artifacts

Raw benchmark files from this run:

- `runs/llamacpp-mtp-adaptive-throttle1-cache0-ctxcp0-c16-64-20260627-184852.jsonl`
- `runs/llamacpp-mtp-adaptive-throttle1-cache0-ctxcp0-c1-128-20260627-184907.jsonl`
- `runs/server-mtp-adaptive-throttle1-cache0-ctxcp0-c16-64-20260627-184715.log`
- `runs/llamacpp-mtp-adaptive-throttle1-c*-64-20260627-185823.jsonl`
- `runs/llamacpp-mtp-adaptive-throttle2-c*-64-20260627-185705.jsonl`
- `runs/llamacpp-mtp-adaptive-throttle4-c*-64-20260627-185939.jsonl`

Relevant patch artifacts in this repo:

- `patches/llamacpp-qwen35moe-mtp-recurrent-row-index.patch`
- `patches/llamacpp-mtp-adaptive-serving-and-recurrent-rollback.patch`