---
license: mit
language:
  - en
  - zh
pipeline_tag: text-generation
base_model: zai-org/GLM-5.3-Flash-BF16
base_model_relation: quantized
tags:
  - exl3
  - glm
  - moe
  - glm5_next
  - quantized
---

# GLM-5.3-Flash EXL3 K2

Community **EXL3 K2** pack of [zai-org/GLM-5.3-Flash-BF16](https://huggingface.co/zai-org/GLM-5.3-Flash-BF16): 2-bit MCG trellis on **routed MoE experts only**. Attn / shared / embed / head / vision stay source-native. **120** safetensors shards, **97,728,721,536 B (91.017 GiB)**.

> ### These weights need a specific vLLM build (K-pool fix shipped 2026-08-30; reinstall)
>
> **Stock vLLM cannot load this pack.** It has neither the `exl3` quantization
> method nor the `Glm5Next` architecture, and no flag turns them on. `pip install vllm`
> will appear to work and then fail *after* you have downloaded 91 GiB.
>
> Prebuilt wheels, no compiler, minutes to install:
> **[vcruz305/GLM-5.3-Flash-EXL3-K2-spark-vllm](https://huggingface.co/vcruz305/GLM-5.3-Flash-EXL3-K2-spark-vllm)**

| Where | What |
|---|---|
| **This repo** | the weights, 120 shards, 91.017 GiB |
| [**spark-vllm**](https://huggingface.co/vcruz305/GLM-5.3-Flash-EXL3-K2-spark-vllm) | prebuilt vLLM + ExLlamaV3 wheels for GB10 |
| [**recipe**](https://github.com/vcruz305/GLM-5.3-Flash-EXL3-K2-DGX-Spark-recipe) | install scripts, serve flags, full measurements |

| | |
|---|---|
| Format | EXL3, `bits=2`, codebook `mcg`, scope `glm53_routed_experts_only` |
| Architecture | `Glm5NextForConditionalGeneration` |
| Spark (TP=1) | native **MTP k=2**: **15.7–16.5 tok/s** @ 8k; **14.6–15.7 tok/s** @ 64k |
| Max ctx allocated | **131072** (KV 786,432 @ util 0.91); recommended serving ctx **65536**, see ladder |
| sixcat 0.5.1 | 120/120 think-on @ 64k — overall **84.17** on the fixed 2026-08-30 runtime (flagged: `mmlu:8` and `ifeval:1300` hit their budgets), same overall as the earlier container run |
| KLD vs BF16 (fidelity suite v1) | 512 contexts, 1,048,064 positions, full-vocab KL through the shared head — **token-mean 0.3346 nats** (95% CI 0.320–0.349), median 0.117, p99 3.33, **top-1 agreement 0.788**; the suite's FP8 anchor is 0.028 / 0.943; the [K2/K3 mix](https://huggingface.co/vcruz305/GLM-5.3-Flash-EXL3-K2K3-mix) scores 0.3121 / 0.795 on the same contexts. Method and scorer validation in the recipe's `docs/KLD.md` |
| long context | boots at **262,144** (KV pool 1,093,332 tokens, 4.17x full-256k concurrency); real-text needle ladder verified to **163,479 prompt tokens** (needle recalled verbatim, prefill ~590 tok/s, decode 17–20). Known runtime bug: prompts between ~163k and ~180k wedge the engine — keep prompts ≤163k until the fix lands; ≤131,072 contexts unaffected. Details in the recipe README |
| Also measured | 4× L40 TP=4 text `/v1` smoke (see below) |

This is a **quantized weight pack**. Pull it, then follow the GitHub recipe for vLLM flags. Weights are MIT (Z.AI).

## Quick start

```bash
# 1. the runtime first, so a bad environment fails in seconds instead of hours
git clone https://github.com/vcruz305/GLM-5.3-Flash-EXL3-K2-DGX-Spark-recipe
cd GLM-5.3-Flash-EXL3-K2-DGX-Spark-recipe
python scripts/preflight.py
bash scripts/install_prebuilt.sh

# 2. then the 91 GiB
hf download vcruz305/GLM-5.3-Flash-EXL3-K2 \
  --local-dir ~/models/GLM-5.3-Flash-EXL3-K2
```

`--local-dir` resumes. There is no `--resume-download` flag. Last path component must be the Hub basename.

One-Spark vLLM (winner flags). Full notes, ctx ladder, and scripts: **[GLM-5.3-Flash-EXL3-K2-DGX-Spark-recipe](https://github.com/vcruz305/GLM-5.3-Flash-EXL3-K2-DGX-Spark-recipe)**.

```bash
export EXL3_FUSED_MOE=1
vllm serve ~/models/GLM-5.3-Flash-EXL3-K2 \
  --served-model-name GLM-5.3-Flash-EXL3 \
  --quantization exl3 \
  --tensor-parallel-size 1 \
  --kv-cache-dtype fp8 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.87 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 2048 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
  --tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm45 \
  --skip-mm-profiling --limit-mm-per-prompt '{"image":4,"video":1}'
```

Load log must show fused `exl3_moe` at bits=2. Bits=2 loading is handled natively by the EXL3 plugin in the runtime wheels, so there is nothing to patch. Do not pass `--moe-backend marlin`. Native MTP; do not mix a DFlash sidecar on the same server.

Eval/serve at 64k: `MAX_MODEL_LEN=65536 GPU_MEM_UTIL=0.91` (same MTP k=2). Spec A/B was ranked at 8k so MLA page size stayed in one band.

## Spark measurements (2026-08-29)

Hardware: one NVIDIA GB10 (SM121), ~121 GiB unified. Engine: vLLM `--quantization exl3`, `EXL3_FUSED_MOE=1`. Bench: streamed `/v1`, thinking off, 128 gen, seqs=1.

**Spec ranking @ 8k**

| Config | Decode tok/s | Accept |
|---|---:|---|
| no spec | 9.6–9.8 | — |
| DFlash sidecar k=7 | 11.5 | mean ~1.8 / 7 |
| DFlash sidecar k=3 (older runtime) | 12.8 | mean ~1.8 / 3 |
| DFlash k=3 BF16, Triton draft attn | 15.6 | 26–79% by workload |
| MTP k=1 | 14.8 | 76–80% |
| **MTP k=2** | **15.7–16.5** | **~74/44%, mean ~2.2** |

**Context ladder** (KV from engine log; decode = same 128-token bench when run)

| max_model_len | spec | util | GPU KV | Decode |
|---:|---|---:|---:|---:|
| 8192 | none | 0.87 | 192k–208k | 9.6–9.8 |
| 8192 | DFlash k=7 | 0.87 | 15,281 | 11.5 |
| 16384 | DFlash | 0.91 | 45,095 | `/v1` pong |
| 32768 | DFlash | 0.91 | 90,035 | allocated (attn block 7168) |
| 8192 | MTP k=2 | 0.87 | 104,857 | **15.7–16.5** |
| **65536** | MTP k=2 | 0.91 | **786,432** | **14.6–15.7** |
| 131072 | MTP k=2 | 0.91 | 786,432 | allocates; 81,920-token prompt OK, 98,304 faults |

131072 allocates (KV 786,432, 6.00x) and an 81,920-token prompt completes, but a 98,304-token prompt faults in the EXL3 fused-MoE path and kills the engine, so **65536 stays the recommended serving ctx**. DFlash at 8k / util 0.87 cannot climb ctx until util 0.91 (draft KV collapse).

**sixcat 0.5.1** think-on, vendor `glm-5.x`, 20/category, host-guarded HumanEval, 64k serve: knowledge 65 · math **100** · truth 85 · instruct 75 · code 90 · tools 90. overall **84.2** with `trunc-in-think:instruct` — not a clean overall.

## What is in the shards

- Routed experts: EXL3 K2 (`suh`, `svh`, `trellis`, `(1,)` int32 `mcg`) on `gate_proj` / `up_proj` / `down_proj`
- 37,152 expert weights + source-native non-expert tensors
- `model.safetensors.index.json`, tokenizer, processor, generation config, chat template
- `quantization_config.json`: `quant_method=exl3`, `bits=2`, `codebook=mcg`, `scope=glm53_routed_experts_only`



## Also measured: 4× L40 TP4

Text `/v1` smoke on 2026-08-28, 4× NVIDIA L40 (SM89), TP=4, 8k, fused `exl3_moe`, no BF16 expert reconstruct. `/health` 200; `17 * 6` → `102`. Build context under `runtime/l40-vllm/` in this repo (CUDA 8.9, dense MLA fallback on L40). That gate is **L40 TP4 text @ 8k**. One-Spark GB10 flags and ctx live in the GitHub recipe.

## Links

- Recipe and scripts: https://github.com/vcruz305/GLM-5.3-Flash-EXL3-K2-DGX-Spark-recipe
- Prebuilt runtime wheels: https://huggingface.co/vcruz305/GLM-5.3-Flash-EXL3-K2-spark-vllm
- BF16 source: https://huggingface.co/zai-org/GLM-5.3-Flash-BF16
- GLM-5.3-Flash: https://huggingface.co/zai-org/GLM-5.3-Flash

License: MIT (Z.AI), same as the BF16 source. Community quant; not an official Z.ai release.