---
license: apache-2.0
base_model: Qwen/Qwen3.6-27B
tags:
  - qwen
  - qwen3.6
  - gguf
  - llama.cpp
  - mtp
  - speculative-decoding
  - iq4_xs
language:
  - en
quantized_by: localweights
---

# Qwen3.6-27B-MTP-IQ4_XS-GGUF

Qwen3.6-27B with NextN/MTP (Multi-Token Prediction) speculative-decoding head, quantized to IQ4_XS for single-GPU 24GB inference.

## What this is

The published Qwen3.6 family ships with native NextN MTP heads embedded in the safetensors. Most public GGUFs strip these. This conversion preserves them, producing a single GGUF that:

- Loads as `qwen35moe_mtp` arch in a patched llama.cpp
- Serves at ~2× decode speed vs. the same trunk without MTP
- Fits in ~15 GB VRAM at IQ4_XS + q4/q4 KV
- Native 262K context

## Build pipeline

Source: `Qwen/Qwen3.6-27B` (HF safetensors, with NextN tensors).

1. Clone llama.cpp at the `crucible-mtp` branch on `llama.cpp (patched)` (Aman Gupta's MTP fork) — adds `LLM_ARCH_QWEN35MOE_MTP` + the NextN draft path.
2. Run `convert_hf_to_gguf.py` against the HF repo. Produces a BF16 GGUF with arch `qwen35moe_mtp` and the NextN tensors fused in.
3. Quantize to IQ4_XS via `llama-quantize`.

```
python convert_hf_to_gguf.py /path/to/Qwen3.6-27B \
    --outfile Qwen3.6-27B-MTP-bf16.gguf
llama-quantize Qwen3.6-27B-MTP-bf16.gguf \
    Qwen3.6-27B-MTP-IQ4_XS.gguf IQ4_XS
```

## Optimal serving config (RTX 3090 Ti, 24 GB)

Cherry-pick PRs `#20819` + `#20822` for cross-process KV-slot save/restore (we use these for sub-second resume on long contexts).

```bash
llama-server \
  -m Qwen3.6-27B-MTP-IQ4_XS.gguf \
  -ngl 999 -fa on \
  --spec-type mtp --spec-draft-n-max 4 \
  --no-mmap \
  --ctx-size 262144 \
  --batch-size 1024 --ubatch-size 512 \
  -ctk q4_0 -ctv q4_0 \
  --parallel 1 --kv-unified \
  --ctx-checkpoints 8 --checkpoint-every-n-tokens 2048 \
  --cache-ram -1 --cache-idle-slots \
  --metrics --jinja
```

**Why these flags:**
- `--spec-type mtp`: enables NextN-head draft path (this is the whole point of the MTP variant).
- `--spec-draft-n-max 4`: empirically the sweet spot — beyond that, accept rate drops faster than draft count grows.
- `--no-mmap`: required for KV-slot persistence + measured ~no perf hit on this rig.
- `-ctk q4_0 -ctv q4_0`: dense KV cache fits 262K context inside 24 GB without spilling.
- `--parallel 1`: MTP path currently only supports `n_parallel=1` upstream.

**What NOT to set:**
- `-ot` (expert offload) — defeats the GPU-resident speedup.
- `-ctk q8_0` at full 262K ctx — overflows VRAM during warmup.

## Performance (RTX 3090 Ti, 350 W power limit)

Measured 2026-05-06 at short-context inference, persistence + MTP on:

| Metric | Value |
|---|---|
| Decode tok/s (short ctx, no thinking) | **100.3** (live measured 2026-05-06, n=4 spec) |
| Decode tok/s (longer ramp 4K–256K ctx, mean) | 70–73 |
| Draft accept rate (n=4) | 86.6% |
| Speedup vs same trunk without MTP | 2.92× (33 → 97 t/s on identical workload) |
| KV slot restore (typical 50 K–200 K ctx) | 0.16–0.36 s |
| Cold load (model → ready) | ~5–6 s |

Memory footprint at 262 K ctx: ~17 GB (model) + ~5 GB (KV q4/q4) + scratch = ~22.5 GB used, ~1.5 GB headroom on a 24 GB card.

## Tokenizer

Inherits Qwen3.6 tokenizer (248,320 vocab, `qwen35` pre-tokenizer). Same chat template as upstream Qwen3.6-Instruct. If your runtime errors on `Jinja Exception: System message must be at the beginning`, use the loosened template at: https://huggingface.co/localweights/qwen36-loose-jinja (single line edit removing the strict-position assertion).

## License

Apache 2.0 (inherited from Qwen3.6).

## Provenance

Built on Crucible: 9950X / 96 GB DDR5 / RTX 3090 Ti. Same pipeline used for the sibling `Qwen3.6-35B-A3B-MTP-IQ4_XS-GGUF` repo.