---
license: apache-2.0
base_model: Qwen/Qwen3.6-27B
base_model_relation: quantized
library_name: transformers
tags:
  - qwen3_5
  - qwen3.6
  - nvfp4
  - quantized
  - modelopt
  - mtp
  - speculative-decoding
  - blackwell
  - text-only
pipeline_tag: text-generation
language:
  - en
  - zh
  - ja
  - ko
  - fr
  - de
  - es
  - it
  - pt
  - ru
  - ar
---

# Qwen3.6-27B-Text-NVFP4-MTP

NVFP4-quantized text-only sibling of [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B), with the **MTP (Multi-Token Prediction) head restored in bf16** so speculative decoding actually works.

## What's different from `sakamakismile/Qwen3.6-27B-NVFP4`

| | This repo (`-Text-NVFP4-MTP`) | `Qwen3.6-27B-NVFP4` |
|---|---|---|
| Quantization format | **`modelopt`** (vLLM SM120 native path) | `compressed-tensors` |
| MTP head | **Restored in bf16, working** | Dropped during export → 0% draft acceptance |
| Vision tower | **Stripped (text-only)** | Present (kept for VLM use) |
| Suggested launch | with `--speculative-config` | without speculation |

The original `Qwen3.6-27B-NVFP4` is left untouched so existing users (~15K downloads) are not disrupted. This is a focused text-only sibling for users who want maximum speed and don't need vision input.

## Why this exists

Two HF Discussion threads on the original repo prompted this:

- [#5 — slower than official FP8 on Blackwell](https://huggingface.co/sakamakismile/Qwen3.6-27B-NVFP4/discussions/5) — root cause is the `compressed-tensors` NVFP4 path being slower than `modelopt` on Blackwell SM120; this repo uses `modelopt` natively.
- [#7 — MTP not responding](https://huggingface.co/sakamakismile/Qwen3.6-27B-NVFP4/discussions/7) — `AutoModelForCausalLM.from_pretrained` does not load the MTP head, so it gets dropped during quantization, leading to 0% draft acceptance. This repo grafts the 15 `mtp.*` tensors (bf16) back into the quantized checkpoint and adds them to the quantization ignore list.

Recipe is adapted from [`osoleve/Qwen3.5-27B-Text-NVFP4-MTP`](https://huggingface.co/osoleve/Qwen3.5-27B-Text-NVFP4-MTP) — credit and thanks.

## Quantization details

- **Base**: `Qwen/Qwen3.6-27B` (bf16, 27.78B params, hybrid linear-attn + full-attn, 64 layers, 1 MTP layer)
- **Quantizer**: `nvidia-modelopt` 0.43.0 with `NVFP4_DEFAULT_CFG`
- **Calibration**: 20 samples from `neuralmagic/calibration` (LLM split), max_seq_len 8192
- **Ignored from quantization** (kept in bf16):
  - `lm_head`
  - All `model.visual.*` (vision tower) — then **physically deleted** in the text-only build
  - All `*linear_attn.conv1d*` (Mamba-style SSM convolutions, 48 of the 64 layers)
  - All `mtp.*` modules (the 1-layer MTP head: 15 tensors total, ~850 MB bf16)
  - Other defaults from `NVFP4_DEFAULT_CFG`: `*router*`, `*mlp.gate.*`, `*block_sparse_moe.gate*`, `*output_layer*`

## Usage with vLLM (Blackwell, SM120)

### With MTP speculative decoding (recommended)

```bash
vllm serve sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP \
    --trust-remote-code \
    --gpu-memory-utilization 0.85 \
    --max-model-len 8192 \
    --language-model-only \
    --quantization modelopt \
    --reasoning-parser qwen3 \
    --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'
```

`num_speculative_tokens: 3` is the **recommended setting** even though the model has a single MTP layer (`mtp_num_hidden_layers=1`): vLLM applies that one layer recursively three times per draft pass, and on Qwen3.5/3.6-family models the per-position acceptance rates stay high enough (typically ~87 / ~72 / ~61 % at positions 1 / 2 / 3) that mean accepted-length lands around 3.0 — which is what unlocks 100+ tok/s on a single Blackwell card. `num_speculative_tokens: 1` (the value originally documented for this family) is a safer fallback if you hit a draft-path bug. The `qwen3_5_mtp` method handler is what vLLM uses for the Qwen3.5/3.6 family (internal `model_type: qwen3_5`); plain `mtp` also works as a fallback.

### Without speculation

```bash
vllm serve sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP \
    --trust-remote-code \
    --gpu-memory-utilization 0.85 \
    --max-model-len 8192 \
    --language-model-only \
    --quantization modelopt
```

## Verified throughput

Single-request decode, T = 0, 9 runs across 3 prompt lengths on 1 × RTX PRO 6000 Blackwell, vLLM 0.19.1rc1:

| Prompt | Tokens | n=1 tok/s | **n=3 tok/s** |
|---|---|---|---|
| Short (50 tok) | 50 | ~71 | **132.5** |
| Medium (350 tok) | 350 | ~85 | **105.5** |
| Long-form (700 tok) | 700 | ~85 | **106.5** |

GPU memory at load: ~15 GB. Mean acceptance length 1.93 / 2.0 at n=1, ~3.0 / 4.0 at n=3 (per-position accept ~87 / 72 / 61 %, matches Pulsate1680's RTX PRO 4500 Blackwell result on this same checkpoint).

The `mtp.fc` weight is kept in **bf16** in the safetensors (not NVFP4) — equivalent to the Lorbus-style "dequantize the fusion layer in the file" trick, applied to NVFP4 instead of AutoRound. This is a side effect of the `*mtp*` ignore entry in the modelopt config, but it is the load-bearing detail behind the n=3 throughput.

## Hardware target

Built and tested on **NVIDIA RTX PRO 6000 Blackwell (SM120)**. Should also work on **RTX 5090** and other Blackwell consumer/workstation cards with sufficient VRAM (the model is roughly 14 GB after NVFP4 + ~850 MB of bf16 MTP/conv1d/lm_head).

## Acknowledgements

- [`osoleve`](https://huggingface.co/osoleve) — for the MTP-restoration recipe on Qwen3.5
- [`Qwen`](https://huggingface.co/Qwen) — for the base model
- [`nvidia-modelopt`](https://github.com/NVIDIA/TensorRT-Model-Optimizer) team
- The reporters of Discussions #5 and #7 — for catching this cleanly

## Support the Base Model Authors

If you find this model useful, please consider supporting:

- **Qwen Team** (original model): [Star the Qwen repo](https://huggingface.co/Qwen/Qwen3.6-27B)

## License

This model inherits the Apache 2.0 license.