Qwen3.6-35B-A3B-GPTQ-INT4-W4A16-LowLatency

An INT4 (W4A16) build of Qwen/Qwen3.6-35B-A3B made for spoken conversation — voice agents where every turn has to come back in well under half a second. Everything a decode step actually reads (routed and shared experts, Gated-DeltaNet and attention projections) is INT4, lm_head is INT8, the calibration data is spoken Chinese dialogue with thinking off, and the MTP head is kept in BF16.

TL;DR — 21.5 GB on disk, 18.6 GiB of weights in vLLM (the popular experts-only INT4 checkpoints: 24.5–25.5 GB / 22.2 GiB) · compressed-tensors INT4 sym g128 → vLLM Marlin MoE kernel on any SM 8.0+ GPU, zero-config (measured on vLLM 0.24.0 and 0.27.1) · one RTX 5090, single stream, ~1.5 k-token chat prompt, 20–30-token replies: TTFT 53 ms, 3.3 ms/token vs 4.6 ms/token for the experts-only INT4 checkpoint most people run — −29 % decode, same TTFT, same fidelity to the BF16 base on our harness; 5 concurrent streams 9.4 vs 12.2 ms/token.

Why another INT4. The community INT4 builds of this model (data-free RTN in AWQ format, true AWQ, GPTQ) quantize only the 256 routed experts and leave the GDN / attention / shared-expert projections and lm_head in BF16 — on a 3 B-active MoE those BF16 parts are most of the bytes a decode step reads, so they all land at ~4.6 ms/token on a 5090. This build quantizes them too; the speed comes from what is quantized, the calibration data decides where the rounding error goes.

Quick start (vLLM ≥ 0.17; measured on 0.24.0 / 0.27.1)

vllm serve Twu31/Qwen3.6-35B-A3B-GPTQ-INT4-W4A16-LowLatency \
  --served-model-name qwen36 --max-model-len 12288 --max-num-seqs 6 \
  --enable-prefix-caching --mamba-ssm-cache-dtype bfloat16 \
  --reasoning-parser qwen3 --default-chat-template-kwargs '{"enable_thinking": false}'

• --mamba-ssm-cache-dtype bfloat16 halves the hybrid model's prefix-cache block (1056 → 544 tokens); outputs were identical with and without it on our harness.
• Leave MTP off for chat — it loads and accepts (~53 % at k=2) but the turn gets slower (below). If you want it: --speculative-config '{"method":"mtp","num_speculative_tokens":2}'.
• Vision tower is BF16, images work out of the box (--mm-processor-kwargs '{"min_pixels":50176,"max_pixels":200704}' to bound image tokens). Fits a 24 GB card for text; on 32 GB at --gpu-memory-utilization 0.85 vLLM reports 298 k KV tokens. SGLang: untested.

Latency (measured)

Same harness for every row: a ~1.4 k-token fixed persona system prompt + 2 turns of history + 18 real spoken-Chinese user turns × 3, greedy, streaming, thinking off; medians of 54 requests; replies 20–27 tokens. One RTX 5090 32 GB, vLLM 0.27.1, flags as above. The anchor is the most-downloaded Qwen3.6-35B-A3B INT4 checkpoint (data-free RTN, experts only, AWQ format).

Checkpoint	Weights in vLLM	fp32 state, 1478-tok prompt — TTFT / turn / ms·tok	bf16 state, block-aligned 1693-tok prompt	+ MTP k=2 (acceptance)
anchor (experts-only INT4, data-free)	22.2 GiB	53 / 142 / 4.6	56 / 141 / 4.5	53 / 149 / 4.1 (56.9 %)
this build, INT4 body only	19.0 GiB	58 / 148 / 3.6	63 / 153 / 3.5	57 / 168 / 3.5 (54.5 %)
this build (main, + INT8 lm_head)	18.6 GiB	53 / 134 / 3.3	62 / 140 / 3.2	59 / 159 / 3.3 (52.7 %)

vLLM 0.24.0 on the same card: this build 3.2–3.3 ms/token, anchor 4.7. Five concurrent streams (MTP off): this build 329 ms/turn, 9.4 ms/token, 424 tok/s aggregate vs anchor 348 / 12.2 / 321.

MTP does not pay here: the draft head is a full BF16 MoE decoder layer, the INT4 main body is cheap, replies are 20–30 tokens and acceptance is 53–57 % — k=2 buys ~10 % on ms/token and gives it back in overhead, so the whole turn is slower (159 vs 140 ms). Re-measure if your replies are long.

Prefix caching on a hybrid GDN MoE reuses whole blocks only — 1056 tokens with the fp32 recurrent state, 544 with --mamba-ssm-cache-dtype bfloat16 — so pad a fixed system prompt to a block multiple if TTFT matters (on a 5090 the difference is small, 53–63 ms either way; on slower cards it is not).

Fidelity vs the BF16 base (same harness, n = 18)

Reference = greedy answers of the BF16 base to the same 18 prompts. NLL = mean per-token negative log-likelihood the quantized model assigns to the BF16 answer (lower = closer). Exact / first-sentence = greedy output identical to the BF16 answer / up to the first sentence break. Treat ±1–2 counts as noise.

Checkpoint	NLL mean / median ↓	exact / first-sent	long-form distinct-2	vision	format probes
anchor (experts-only INT4, data-free)	0.408 / 0.426	6 / 7	0.975	✓	2/3
experts-only INT4, our GPTQ + Chinese-chat calibration (structure control)	0.419 / 0.470	8 / 9	0.914	✓	2/3
this build (main)	0.400 / 0.425	5 / 8	0.922	✓	2/3
cyankiwi AWQ-4bit (experts only, g32)	0.419 / 0.437	6 / 10	0.935	✓	2/3
Intel int4-mixed AutoRound (experts + GDN + MTP INT4)	0.431 / 0.456	5 / 7	0.974	✓	2/3
palmfuture GPTQ-Int4 (experts only)	0.409 / 0.434	5 / 8	0.927	✓	2/3

All checkpoints sit in one NLL band (0.40–0.43); this build has the lowest. Persona tag compliance 54/54, empty replies 0, no repetition loops, for every checkpoint. Its greedy replies are a little longer than the anchor's (median 27 vs 20 tokens; BF16 base median 21.5) because on two or three prompts it takes a longer branch where the base stops after one sentence — same NLL either way, i.e. quantization noise flipping a near-tie.

What is quantized, what is not

Module	Precision	Why
Routed experts mlp.experts.*.{gate,up,down}_proj, shared expert, GDN in_proj_qkv / in_proj_z / out_proj, attention q/k/v/o_proj, incl. layer 0	INT4 sym g128 (GPTQ)	everything a decode step reads; the community builds keep all but the routed experts in BF16
GDN in_proj_a, in_proj_b; MoE router mlp.gate, shared_expert_gate; norms, conv1d, embeddings	BF16	tiny gates / routing decisions — 4-bit hurts, saves nothing
Vision tower model.visual.*	BF16	image quality; rarely on the decode path
MTP head mtp.*	BF16, re:.*mtp.* in quantization_config.ignore	the drafter must stay close to the target or it never accepts
lm_head	INT8 sym g128 (RTN, post-hoc)	0.5 B params read every token; lossless on our metric, −9 % decode

Format: compressed-tensors pack-quantized, group_0 = INT4 g128 sym actorder=weight on Linear, group_1 = INT8 g128 on re:.*lm_head$; per-expert tensors (experts.N.gate_proj.weight_packed, vLLM CompressedTensorsWNA16MarlinMoEMethod). 21.5 GB on disk (BF16 base 71.9 GB).

Calibration & reproduce

384 sequences ≥ 2048 tokens, rendered through the model's own chat template with enable_thinking=False: ~60 % multi-turn spoken-dialogue transcripts (a ≈1.4 k-token assistant-persona system prompt + 10–20 short colloquial turns, predominantly Chinese, real speech incl. disfluencies), ~20 % the same turns without a system prompt, ~20 % ShareGPT-GPT4 / UltraChat (zh/en). No benchmark test sets; the transcripts are not distributed and nothing from them is embedded in the weights. moe_calibrate_all_experts=True so rarely-routed experts still get a usable Hessian.

llm-compressor 0.13.0 · transformers 5.14.1 · compressed-tensors 0.18.0 · torch 2.13; one RTX 5090 + 120 GB RAM, 3 h wall-clock (GPTQ's per-module loop over 256 experts × 3 × 40 layers; independent of sample count).

import torch
from datasets import Dataset
from llmcompressor import oneshot
from llmcompressor.modeling.moe.linearize import load_quantizable_moe
from llmcompressor.modifiers.quantization import GPTQModifier
from transformers import AutoTokenizer, Qwen3_5MoeForConditionalGeneration

MODEL = "Qwen/Qwen3.6-35B-A3B"
with load_quantizable_moe(Qwen3_5MoeForConditionalGeneration):          # fused 3-D experts -> per-expert nn.Linear
    model = Qwen3_5MoeForConditionalGeneration.from_pretrained(MODEL, dtype=torch.bfloat16)
tok = AutoTokenizer.from_pretrained(MODEL)
ds = Dataset.from_list([{"text": t} for t in pre_rendered_texts])       # 384 x >=2048 tokens, chat template applied, thinking off
ignore = ["re:.*visual.*", "lm_head", r"re:.*linear_attn\.in_proj_a$", r"re:.*linear_attn\.in_proj_b$",
          r"re:.*mlp\.gate$", r"re:.*shared_expert_gate$"]
recipe = GPTQModifier(ignore=ignore, scheme="W4A16", targets=["Linear"], dampening_frac=0.01, actorder="weight")
oneshot(model=model, processor=tok, dataset=ds, recipe=recipe, max_seq_length=2048,
        num_calibration_samples=384, moe_calibrate_all_experts=True)
model.save_pretrained("out", save_compressed=True, max_shard_size="4GB"); tok.save_pretrained("out")

Three things the exporter does not do for you: (1) copy the BF16 mtp.* tensors from the base checkpoint into the export and add re:.*mtp.* to quantization_config.ignore — without both, vLLM's drafter loads packed weights it cannot use and MTP runs at 0 %; (2) apply the INT8 lm_head post-hoc (compressed-tensors packer, group_1, target re:.*lm_head$); (3) copy the upstream tokenizer / config files back verbatim (transformers 5.14 re-serialises tokenizer.json). Full scripts (quantization, export fix-up, INT8 head, evaluation harness) are in our repo, llm_quant/qwen36_35b_zhchat/.

Limitations

• Built and measured for short-reply, thinking-off, single/few-stream spoken chat. Long chain-of-thought, tool use, > 12 k context and academic benchmarks were not evaluated; fidelity numbers are relative to the BF16 base on 18 conversational prompts.
• Measured on one RTX 5090 only; the Marlin path is the same on Ampere/Ada but absolute numbers will differ.
• vLLM 0.27.1 on Python 3.10 needs from __future__ import annotations at the top of flashinfer/comm/fd_exchange.py; fresh machines need g++ and python3-dev for the JIT kernels.

Acknowledgements

Qwen team for Qwen3.6; the vLLM / llm-compressor / compressed-tensors maintainers.