Qwen3.6-35B-A3B-MTP-IQ4_XS-GGUF

Qwen3.6-35B-A3B (MoE, 3B active parameters) with NextN/MTP (Multi-Token Prediction) speculative-decoding head, quantized to IQ4_XS for single-GPU 24 GB inference.

What this is

A self-contained GGUF carrying both the trunk and the NextN MTP head. With a patched llama.cpp it serves at ~2× decode of the same trunk without MTP, and at long contexts the MoE+MTP combination wins on every dimension we measured: faster than the dense 27B, better long-ctx behavior, lower VRAM footprint than non-MTP equivalents.

This is what we use as our "orchestrator" / chair model for council pipelines on a 24 GB card.

Build pipeline

Source: Qwen/Qwen3.6-35B-A3B (HF safetensors, with NextN tensors).

1. Clone llama.cpp at the crucible-mtp branch on llama.cpp (patched) (Aman Gupta's MTP fork) — adds LLM_ARCH_QWEN35MOE_MTP + the NextN draft path.
2. Run convert_hf_to_gguf.py against the HF repo. Produces a BF16 GGUF with arch qwen35moe_mtp and the NextN tensors fused in.
3. Quantize to IQ4_XS via llama-quantize.

python convert_hf_to_gguf.py /path/to/Qwen3.6-35B-A3B \
    --outfile Qwen3.6-35B-A3B-MTP-bf16.gguf
llama-quantize Qwen3.6-35B-A3B-MTP-bf16.gguf \
    Qwen3.6-35B-A3B-MTP-IQ4_XS.gguf IQ4_XS

Optimal serving config (RTX 3090 Ti, 24 GB)

Cherry-pick PRs #20819 + #20822 for cross-process KV-slot save/restore (sub-second resume on long contexts — material when this model is used as a long-running chair / orchestrator).

llama-server \
  -m Qwen3.6-35B-A3B-MTP-IQ4_XS.gguf \
  -ngl 999 -fa on \
  --spec-type mtp --spec-draft-n-max 4 \
  --no-mmap \
  --ctx-size 200000 \
  --batch-size 1024 --ubatch-size 512 \
  -ctk q4_0 -ctv q4_0 \
  --parallel 1 --kv-unified \
  --ctx-checkpoints 8 --checkpoint-every-n-tokens 2048 \
  --cache-ram -1 --cache-idle-slots \
  --metrics --jinja

Why these flags:

• --spec-type mtp: enables NextN-head draft path. The whole point of the MTP variant.
• --spec-draft-n-max 4: empirical sweet spot. n=2 underutilises the head; n=6+ drops accept rate faster than throughput grows. n=4 lands at ~85% accept × ~101 tok/s.
• --ctx-size 200000: sized to leave ~1.5 GB VRAM headroom on 24 GB at q4/q4 KV. Native max is 262 K — push to that only if you don't need headroom for activations.
• --no-mmap: required for KV-slot persistence and zero perf hit on this rig.
• -ctk q4_0 -ctv q4_0: dense KV cache; q8 KV at this ctx exceeds 24 GB during warmup.
• --parallel 1: MTP path is single-stream upstream.

What NOT to set:

• -ot "exps=CPU" (expert offload) — kills speedup; only consider if running alongside another model.
• -ctk q8_0 at full 200K ctx — VRAM will OOM at warmup.

Performance (RTX 3090 Ti, 350 W power limit)

Measured 2026-05-06 with persistence + MTP on:

Metric	Value
Decode tok/s (short ctx, no thinking)	213.7 (live measured 2026-05-06, n=4 spec)
Decode tok/s (steady mean over 200 K ctx run)	183 mean / 203 peak
Prefill tok/s	3,078
Draft accept rate (n=4)	97% (live), 80–85% under steady mixed workload
KV slot restore (190 K tokens)	357 ms — ~525,000 tokens/s restore rate
Cold load (model → ready)	~5–6 s
Power sweep optimum	350 W (0.298 t/s/W; vs. 480 W default ≈ -7% throughput, +30% energy)
Cross-process slot save/restore	working — verified across model swaps via PRs 20819+20822
Combined gain vs starting baseline	6.1× (33 → 200+ t/s, MTP × ngl 999 × power × n=4)

Memory footprint at 200 K ctx: ~18.5 GB (model) + ~3 GB (KV q4/q4) + scratch ~22.5 GB. Headroom for activation spikes.

Tokenizer

Qwen3.6 tokenizer (248,320 vocab, qwen35 pre-tokenizer). Same chat template as upstream Qwen3.6-35B-A3B-Instruct. If your runtime errors on Jinja Exception: System message must be at the beginning, use a loosened jinja template (single line: replace the strict-position raise with a no-op).

Why MTP matters here

Qwen3.6-35B-A3B in plain GGUF form decodes at ~50 tok/s on this rig (single-stream MoE, 3B active). With MTP enabled and ~85% accept rate at draft n=4, we observe ~100 tok/s — exactly the 2× the NextN paper claims. On the 3090 Ti this is the difference between "useful chair model" and "infuriatingly slow."

The MoE-MTP combination also has a non-obvious benefit: long-context decode degrades much more gracefully than the 27B dense at the same VRAM budget. At 190 K tokens, 35B-A3B-MTP holds ~50 tok/s where 27B drops below 30.

License

Apache 2.0 (inherited from Qwen3.6).

Provenance

Built on Crucible: 9950X / 96 GB DDR5 / RTX 3090 Ti. Sibling: localweights/Qwen3.6-27B-MTP-IQ4_XS-GGUF.