Qwen3.5-122B-A10B-NVFP4-FP8Dense-GB10

Mixed-precision quantization of Qwen/Qwen3.5-122B-A10B — NVFP4 MoE experts, FP8 W8A8 dense path, BF16 lm_head — tuned for single-GPU serving on NVIDIA DGX Spark (GB10, SM121) and verified on datacenter GPUs (H200) with stock vLLM. Full multimodal — the vision encoder is preserved (BF16) and verified working.

Why this exists: scottgl's GB10 quant proved the mixed-precision idea (and measured FP8-GDN as quality-safe), but it ships the dense path as BF16 and quantizes it at runtime inside a custom SGLang fork — it doesn't serve on vLLM. This checkpoint bakes a comparable mixed layout into the weights as standard compressed-tensors, so it loads on stock vLLM with no fork and no runtime quantization. The second goal was quality: all-NVFP4 (W4A4) checkpoints of this model maximize decode speed but flatten output quality — the dense path's 4-bit activations and the FP4 lm_head are the prime suspects. So: NVFP4 only where it pays (the MoE experts, ~95% of weights), FP8 W8A8 for the dense path, BF16 lm_head.

What "FP8 dense" means

In a MoE model the dense path is everything outside the experts — the layers every token passes through regardless of routing: the standard attention projections (q/k/v/o, 12 layers) and the GDN linear-attention projections (36 layers). In this model that's only ~4B of the 122B parameters, but it runs on every single token, and in W4A4 quants its 4-bit activations are a prime quality suspect. Here the dense path is FP8 W8A8 dynamic: 8-bit float weights with per-channel scales, 8-bit float activations quantized per-token at runtime — near-lossless for these layers (cos_sim > 0.999 measured for FP8 GDN), at twice the bandwidth cost of FP4 on a small fraction of the weights. The experts (the other ~117B, of which ~4B activate per token) stay NVFP4, where the bandwidth savings actually matter.

Layout

Component	Format	Rationale
MoE routed experts (256/layer × 48)	NVFP4 (W4A4, group 16)	95% of weights; bandwidth dominates decode
MoE shared expert	NVFP4 (W4A4, group 16)	Same
Standard attention q/k/v/o (12 layers)	FP8 W8A8 dynamic (per-channel W, per-token A)	8-bit activations preserve attention fidelity
GDN linear-attention projections (36 layers × 5)	FP8 W8A8 dynamic	Measured-safe precedent (cos_sim > 0.999, SNR 31.5 dB — scottgl)
lm_head	BF16	Un-flattens the output distribution; ~0.76B params
Vision encoder + merger	BF16	Quality-critical, small
Router gates, embeddings, conv1d, norms	BF16	Routing/structural precision
MTP layer (785 tensors)	BF16 (raw copy)	Not loadable by vLLM; kept for speculative-decoding experiments elsewhere

Comparison with sibling quants of this model:

	This model	Full (all-FP4)	scottgl
MoE experts	NVFP4	NVFP4	NVFP4
Standard attention	FP8 W8A8 (baked)	NVFP4	BF16
GDN projections	FP8 W8A8 (baked)	NVFP4	BF16 stored, FP4/FP8 at SGLang runtime
lm_head	BF16	NVFP4	BF16 stored, FP8 at SGLang runtime
Vision	BF16 (preserved)	BF16 (preserved)	—
Serves on	stock vLLM	stock vLLM	custom SGLang fork
Checkpoint	74 GB	66 GB	—
Decode, GB10 single-stream	23–24 tok/s	33–35 tok/s	~46 (SGLang + NEXTN spec-dec)

Quantization details

• Method: llm-compressor oneshot() — must be installed from git main for this model (the 0.11.0 release pins transformers 4.x, which lacks qwen3_5_moe)
• Calibration: 512 samples, HuggingFaceH4/ultrachat_200k, seq_len 2048
• Hardware: 2× H200 (sequential pipeline uses one execution GPU + CPU offload; ~1.5–1.8 min/layer × 49 layers ≈ 75 min)
• MTP tensors restored into the checkpoint post-save via compressed_tensors.utils.save_mtp_tensors_to_checkpoint (transformers does not instantiate the MTP layer)
• recipe.yaml in this repo is the as-run record — see the warning below before reusing it

⚠️ If you re-quantize: the overlapping-targets trap

The as-run recipe used targets: [Linear] for the NVFP4 group, overlapping the FP8 regex group. llmcompressor resolves the overlap one way (group order → FP8 wins); vLLM resolves it the other way (NVFP4 wins) — it allocates [N, 192] group-scale buffers for FP8 layers and broadcasts the [N, 1] channel scales into them, producing garbage logits (every token decodes to !). The config.json in this repo is fixed: the NVFP4 group lists explicit, mutually-exclusive expert targets, and the result loads correctly on stock unpatched vLLM. If you write your own recipe, use explicit targets from the start. Also keep the lm_head ignore entry in regex form (re:.*lm_head$) — serialization expands it to the bare name lm_head, which older vLLM versions fail to match against language_model.lm_head.

Benchmarks

DGX Spark GB10 (vLLM 0.22.1, flashinfer_b12x MoE backend, bf16 KV)

Sequential decode, gen=500:

Context	TTFT	Decode tok/s
warmup	0.98s	24.1
9K	3.48s	23.7
9K (prefix-cached)	1.36s	23.7
18K	4.22s	23.3
27K	4.13s	22.9
27K (prefix-cached)	1.60s	22.9

Concurrent @ 32K context, gen=500:

Concurrency	Aggregate tok/s	Per-request tok/s
8×	72.8	10.5
16×	100.5	7.2
32×	129.0	4.8
64×	159.1	3.2

KV cache: 1,092,266 tokens at --gpu-memory-utilization 0.85 (4.17× the full 262K context).

Versus the all-FP4 Full checkpoint on the same box: −31% single-stream (pure weight-bandwidth: 8.0 vs 5.1 GB/token), but only −4…−9% at 8–64× concurrency — at batch the weight reads amortize and the dense-path FP8 cost nearly vanishes.

Quality

On a synthetic dashboard detail-extraction A/B (34 scoreable details, three degradation levels, temp 0), this model and the all-FP4 Full are indistinguishable — both ace it; the shared BF16 vision encoder is the limiting factor at marginal legibility. The layout's gain shows in text/reasoning fidelity (FP8 activations in the dense path + BF16 lm_head): subjectively, noticeably better comprehension in extended real-world use. If you need maximum tok/s and can tolerate the all-FP4 quality profile, use the Full checkpoint instead.

Serving

Datacenter GPUs (H100/H200/B200, SM90+) — stock vLLM

Verified on 1× H200 with stock vllm/vllm-openai:latest (≥ 0.22), no patches:

vllm serve demon-zombie/Qwen3.5-122B-A10B-NVFP4-FP8Dense-GB10 \
  --served-model-name Qwen3.5-122B-A10B \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

DGX Spark (GB10, SM121)

Two upstream bugs currently break the GB10-native FP4 MoE path; both are fixed by small patch scripts shipped in this repo under gb10-patches/:

1. patch_cutlass_dsl_global_dtors.py — nvidia-cutlass-dsl 4.5.2 emits an llvm.mlir.global_dtors op its own bundled MLIR verifier rejects (missing the newer mandatory data attribute); kills every CuTe-DSL tvm-ffi kernel JIT.
2. patch_b12x_gated_smem.py — flashinfer 0.6.11.post2's gated dynamic-MoE kernel under-budgets shared memory (its stage calculator counts ONE B/SFB buffer per pipeline stage; the gated kernel allocates TWO) → 118,784 B vs the 99 KiB SM12x ceiling → bare cudaErrorInvalidValue at launch. Affects any SM120/SM121 GPU with this model's shapes (k=3072 → 24 k-tiles keeps the over-budget stage count).

# one-time: extract the two files from the image and patch them
docker create --name tmp vllm/vllm-openai:latest
docker cp tmp:/usr/local/lib/python3.12/dist-packages/nvidia_cutlass_dsl/python_packages/cutlass/cutlass_dsl/tvm_ffi_provider.py .
docker cp tmp:/usr/local/lib/python3.12/dist-packages/flashinfer/fused_moe/cute_dsl/blackwell_sm12x/moe_dynamic_kernel.py .
docker rm tmp
python3 patch_cutlass_dsl_global_dtors.py tvm_ffi_provider.py
python3 patch_b12x_gated_smem.py moe_dynamic_kernel.py

docker run -d --name vllm --gpus all -p 8000:8000 --ipc host \
  -v /opt/vllm-cache:/root/.cache/huggingface \
  -v $PWD/tvm_ffi_provider.py:/usr/local/lib/python3.12/dist-packages/nvidia_cutlass_dsl/python_packages/cutlass/cutlass_dsl/tvm_ffi_provider.py:ro \
  -v $PWD/moe_dynamic_kernel.py:/usr/local/lib/python3.12/dist-packages/flashinfer/fused_moe/cute_dsl/blackwell_sm12x/moe_dynamic_kernel.py:ro \
  -e CUBLASLT_WORKSPACE_SIZE=33554432 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False \
  -e RUNAI_STREAMER_MEMORY_LIMIT=4294967296 \
  vllm/vllm-openai:latest \
  --model demon-zombie/Qwen3.5-122B-A10B-NVFP4-FP8Dense-GB10 \
  --served-model-name Qwen3.5-122B-A10B \
  --kernel-config '{"moe_backend": "flashinfer_b12x"}' \
  --load-format runai_streamer \
  --gpu-memory-utilization 0.85 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-batched-tokens 4176 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

GB10 unified-memory notes (all load-bearing, all measured):

• --load-format runai_streamer + RUNAI_STREAMER_MEMORY_LIMIT=4G: the default safetensors mmap path keeps roughly a full shard page-cache-resident on top of the 74 GB of materialized params and OOMs the 121 GB pool. The streamer loads 74 GB in ~40 s, bounded.
• expandable_segments:False: True prevents torch.cuda.empty_cache() from returning pages to the OS on this stack.
• --gpu-memory-utilization 0.85: 0.90 swap-storms the box with this model.
• moe_backend=flashinfer_b12x (explicit opt-in): the default Marlin MoE backend's repack allocates per-layer 256-expert transients that repeatedly OOM the unified pool during bring-up; b12x loads packed NVFP4 weights directly and uses the GB10's FP4 tensor cores.

Known limitations

1. vLLM cannot use the MTP weights (kept BF16 for SGLang/NEXTN-style speculative-decoding experiments).
2. The two GB10 patches are point fixes against vllm/vllm-openai:latest (vLLM 0.22.1, flashinfer 0.6.11.post2, nvidia-cutlass-dsl 4.5.2); re-check after image upgrades.
3. chat_template follows the base model; thinking output is separated into reasoning_content by vLLM's qwen3 reasoning parser.

Architecture

Qwen3.5-122B-A10B is a hybrid-attention MoE model: 48 layers (36 GDN/Gated-DeltaNet linear attention + 12 full attention, interval 4), 256 routed experts + 1 shared expert per layer (8 active), 3,072 hidden, 248,320 vocab, 262K context, plus a ViT vision encoder and an MTP head.

Credits

• Base model: Qwen/Qwen3.5-122B-A10B
• Layout informed by scottgl/Qwen3.5-122B-A10B-NVFP4-GB10's FP8-GDN quality measurements (cos_sim > 0.999, SNR 31.5 dB)
• Quantized with llm-compressor