hermia/glm-5.2-sm120-recipe

GLM-5.2-504B (NVFP4) on 4× RTX PRO 6000 — 250K context, stock vLLM

This is a recipe, not a model. No weights here. It documents how to serve 0xSero/GLM-5.2-504B-Nvidia at 250,000-token context on 4× RTX PRO 6000 Blackwell (sm_120) using stock vLLM 0.25.1 — no fork, no Docker, no CUDA 13.2, no b12x.

👉 Full recipe, patch, and scripts: https://github.com/hermia-ai/glm-5.2-sm120-stock-vllm

---

Measured results

Metric	Value
Context served	250,000 tokens
KV cache pool	554,759 tokens (vs 164,160 without the DCP patch — 3.37×)
Max concurrency @ 250K	2.22× (was 0.66× — could not fit even one full-context request)
Decode	~43 tok/s single-stream (incl. thinking tokens)
Weights	79.1 GB / GPU
Needle-in-haystack	PASS at 11,058 / 44,058 / 88,058 / 107,309 / 121,058 / 178,809 / 233,809 input tokens

All measured on real hardware, 2026-07-15. Nothing here is estimated.

The three walls between you and a working server

Each is documented in the GitHub repo with the exact error string and file:line cause:

1. OOM at 93.64 GiB/GPU. An upstream transformers bug: GlmMoeDsaConfig.__post_init__ unconditionally re-routes a legacy num_experts key over an explicit n_routed_experts. REAP-pruned checkpoints ship both keys, so vLLM builds FusedMoE for 256 experts that do not exist (log: Local/global number of experts: 64/256, correct is 42/168). Fix: delete the stale num_experts key from config.json.
2. TypeError: trtllm_batch_decode_with_kv_cache_mla() got an unexpected keyword argument 'kv_scale_format'. vLLM 0.25.1 pins flashinfer-python==0.6.13 but its own sm120 sparse-MLA call site passes a 0.6.14-only kwarg. Fix: install 0.6.14 + FLASHINFER_DISABLE_VERSION_CHECK=1.
3. AssertionError: Decode Context Parallelism (DCP) requires attention implementations to return the softmax LSE during decode, but FlashInferMLASparseSM120Impl does not. — and it tells you to pick another backend when that is the only backend on sm_120. Fix: a ~126-line patch (an independent reimplementation of vLLM PR #47779).

What does NOT work (documented so you don't repeat it)

• MTP / speculative decoding — the drafter eats ~6.96 of the 6.97 GiB KV budget → estimated maximum model length is 256. MTP and 250K context are mutually exclusive on 4×96 GB.
• 0xSero/GLM-5.2-504B-W4A16 (tempting: 57 GB smaller) — cannot load. indexer.wk ships int4-packed but vLLM hardcodes that layer quant_config=None → KeyError. Unreachable by any flag.
• SGLang — needs bf16 KV on sm_120, which doubles per-token bytes vs fp8. A regression.
• Raising --max-model-len without DCP — the KV pool is a fixed byte budget; it does not grow.

Credit

The DCP patch is an independent reimplementation of vLLM PR #47779, not an improvement on it — the design credit is upstream's. This work contributes validation on real sm_120 silicon.

Thanks to 0xSero for the checkpoint and for glm-5.2-sm120 — the complementary fork/Docker path, which reaches ~710K KV and MTP=3 at the cost of a multi-repo source build. If you want those numbers, use theirs; if you want to stay on stock vLLM, use this.

Also: DeepGEMM PR #324 (NVIDIA DevTech — sm120_fp4_mqa_logits), FlashInfer PR #3395 (SM120 sparse MLA), lukealonso/b12x.

Licensed Apache-2.0.