README.md

---
language:
- en
- zh
license: mit
library_name: transformers
pipeline_tag: text-generation
base_model:
- malaiwah/GLM-5.2-EXL3-TR3-MTP78
- zai-org/GLM-5.2
base_model_relation: quantized
tags:
- glm
- exl3
- trellis
- vllm
- blackwell
- mixture-of-experts
inference: false
---

# GLM-5.2 EXL3 TR3 3.0 bpw

This is a TP4, rank-sliced EXL3 build of
[zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2), optimized for
four NVIDIA Blackwell workstation GPUs. Routed MoE experts in layers 3-77 use
EXL3 Trellis weights targeting 3.0 bits per weight. Accuracy-sensitive and
dense components remain in BF16.

The repository payload is 332.19 GB (309.37 GiB). This format requires the
custom vLLM + Sparkinfer runtime below; it is not a drop-in Transformers model.
The `config.json` retains ModelOpt/NVFP4 compatibility metadata used by the
conversion pipeline, but the routed weights are EXL3 and the required launch
flag is `--quantization exl3`. NVFP4 in the supplied runtime refers to the KV
cache, not the routed-expert weight format.

## Quantization layout

| Component | Storage |
| --- | --- |
| Routed MoE experts, layers 3-77 | EXL3 Trellis, TP4 rank-sliced, 3.0 bpw target |
| Dense MLP layers 0-2 | BF16 |
| Shared-expert MLPs | BF16 |
| Attention and sparse indexer | BF16 |
| Embeddings and LM head | BF16 |
| Norms, router gates, and e-score correction bias | BF16/FP32 source precision |
| MTP layer 78 | EXL3 routed experts from the MTP-78 lineage; remaining MTP tensors retain source precision |

The calibration manifest is included as `calibration_manifest.json`. It records
12,228 owner-corpus samples across general, legal, coding/agentic, and
reasoning/termination axes. The model was calibrated at TP8 and packed for TP4.

## Supported runtime

The tested runtime is published at
[verdictai/glm52-exl3-sparkinfer](https://hub.docker.com/r/verdictai/glm52-exl3-sparkinfer):

```text
verdictai/glm52-exl3-sparkinfer:v31-gg-v20-sic3828fd-vllm0c79e41-cu132-sm120a
```

The supplied scripts pin the immutable registry manifest:

```text
verdictai/glm52-exl3-sparkinfer:v31-gg-v20-sic3828fd-vllm0c79e41-cu132-sm120a@sha256:0433ae94665b769b78dd301f952d907508a3ba80bce47a1630ec20ade8812dff
```

It pins:

- Gilded Gnosis v20 canonical vLLM `6722c1d` (`dev/gilded-gnosis`) + EXL3 Trellis (rebased PR #139)
- Sparkinfer v20 canonical `1a88b389` (`master`) + EXL3 Trellis fused arm (rebased PR #49)
- CUDA 13.2, PyTorch 2.12, CUTLASS DSL 4.6.0, FlashInfer 801d57a, NCCL 2.30.4, SM120a
- EXL3 Trellis MoE, B12X sparse MLA, DCP A2A, and MTP speculative decoding
- NVFP4 DeepSeek-MLA KV cache using calibrated outer scales

### v20 changes (Gilded Gnosis v20 base)

This release rebases the EXL3 Trellis backend onto the Gilded Gnosis **v20**
canonical heads (vLLM `6722c1d`, Sparkinfer `1a88b389`). The v20 base supplies
the upstreamed MTP/DCP correctness fixes — head-major cross-rank BMM (vLLM #147),
MTP verifier-decode dispatch (vLLM #164), and graph-resource isolation (vLLM
#149). On top of that base this image adds (vLLM PR #139 / Sparkinfer PR #49,
both rebased onto v20):

- **EXL3 Trellis MoE backend** for rank-sliced GLM/DeepSeek routed experts.
- **MTP tool-call + DSA-crash fix** — the structured-output grammar advances
  from the authoritative step delta so tool calling engages under speculative
  decoding, and `has_indexer` is derived from `index_k` so MTP draft steps that
  skip top-k no longer trip the fused-norm-rope assertion.
- **Dual-plan Trellis prefill** — prefill batches route through the planned
  Trellis MoE.
- **SM120 + B12X flattening fix** — MTP `next_n>2` uses the native `(B, next_n)`
  sparse-indexer path instead of the DeepGEMM `next_n<=2` flatten fallback, which
  had corrupted MTP-3 code generation.

Validated on 4x RTX PRO 6000 Blackwell 96 GB (TP4/DCP4, MTP-3 greedy, NVFP4 KV,
FULL_AND_PIECEWISE cudagraphs):

- **Code generation** — the previously reported ~50% syntax-error rate under
  MTP-3 is eliminated; generated Python/HTML is 94–100% syntactically valid
  across runs. The rare remaining edge is fp8-KV/DCP floating-point
  nondeterminism, not the prior systematic corruption.
- **LAVD** long-context retrieval (r10/c5, temp 0): **10/10 (6 exact, 4 near,
  0 fail)** — up from 8/10 (2 fails) on the prior base. The v20 canonical MTP
  fixes plus the flattening fix drive the fail count from 2 to 0.

Asynchronous scheduling remains disabled as a correctness guard for this
DCP4/MTP path; do not enable it on this release.

## Quick start

Install Docker Engine, Docker Compose v2, the NVIDIA Container Toolkit, and the
Hugging Face CLI. Then download the model and start the server:

```bash
hf download brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw \
  --local-dir "$HOME/models/GLM-5.2-EXL3-TR3-3.0bpw"

cd "$HOME/models/GLM-5.2-EXL3-TR3-3.0bpw"
chmod +x server.sh
./server.sh start
./server.sh logs
```

The OpenAI-compatible endpoint is available locally at
`http://localhost:8000/v1`. Here, `localhost` always means the machine on
which the downloader starts this model; it does not refer to the model
publisher's machine.

```bash
curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "GLM-5.2-EXL3-TR3-3.0bpw",
    "messages": [{"role": "user", "content": "What is 2 + 2?"}],
    "temperature": 0,
    "max_tokens": 128
  }'
```

Useful controls:

```bash
./server.sh status
./server.sh logs
./server.sh restart
./server.sh stop
```

## Direct Compose use

`server.sh` exports the defaults and invokes the included
`docker-compose.yml`. To run Compose directly, set the model and cache paths:

```bash
export MODEL_DIR="$HOME/models/GLM-5.2-EXL3-TR3-3.0bpw"
export CACHE_DIR="$HOME/.cache/glm52-exl3-sparkinfer"
docker compose pull
docker compose up -d
docker compose logs --tail 100 -f glm52
```

The defaults can be overridden without editing the files:

| Variable | Default | Purpose |
| --- | --- | --- |
| `MODEL_DIR` | Directory containing `server.sh` | Model mount |
| `CACHE_DIR` | `~/.cache/glm52-exl3-sparkinfer` | Persistent JIT cache |
| `PORT` | `8000` | Host API port |
| `BIND_ADDRESS` | `127.0.0.1` | Local-only host binding |
| `CUDA_VISIBLE_DEVICES` | `3,1,2,0` | Physical GPU to TP-rank order |
| `GPU_MEMORY_UTILIZATION` | `0.96` | vLLM memory reservation |
| `MAX_MODEL_LEN` | `262144` | Per-request context cap |
| `MTP_TOKENS` | `3` | Validated speculative-token count (MTP-3) |
| `ENABLE_ASYNC_SCHEDULING` | `0` | Required correctness guard |
| `NUM_GPU_BLOCKS_OVERRIDE` | `1024` | 262,144 logical KV tokens at DCP4 |

The tested GPU order intentionally keeps physical GPU 3 away from TP rank 3.
On another host, set `CUDA_VISIBLE_DEVICES=0,1,2,3` or use the order appropriate
for that machine.

The supplied Compose file binds only to loopback by default, so it does not
publish the API to the LAN or internet.

## Runtime validation

The exact published image completed all 81 model-shard loads, EXL3
initialization, Sparkinfer PCIe collective initialization, NVFP4 KV allocation,
and full plus piecewise CUDA graph capture. The v20 base has a larger runtime
footprint, so the release preset allocates 262,144 logical KV-cache tokens:
1,024 blocks x 64 tokens x DCP4, with 65,536 tokens local to each DCP rank. The
configured per-request context cap is also 262,144. A GPU 0 with no other
resident processes supports a larger KV pool and context (raise
`NUM_GPU_BLOCKS_OVERRIDE` / `MAX_MODEL_LEN` within the KV capacity reported at
startup).

Quality was validated with the release serving configuration (MTP-3, TP4/DCP4,
concurrency 5):

| Evaluation | v20 result | Prior base |
| --- | ---: | ---: |
| LAVD r10/c5 | 10/10 (6 exact, 4 near, 0 fail) | 8/10 (2 fail) |
| Code generation (Python/HTML, temp 0) | 94–100% valid | ~50% (reported bug) |

The LAVD fail count dropping 2 → 0 reflects the v20 canonical MTP fixes plus the
SM120+B12X flattening fix. Exact/near split varies run to run (fp8-KV/DCP
nondeterminism); the 0-fail result is the stable signal.

The following prefill, decode, and KLD tables are indicative reference measured
on the prior base; the v20 image is validated for correctness by the results
above.

Cold standalone prefill results:

| Requested context | Prompt tokens | TTFT | Client tok/s | Server tok/s |
| ---: | ---: | ---: | ---: | ---: |
| 8K | 8,201 | 5.46 s | 1,502 | 1,507 |
| 64K | 64,512 | 51.64 s | 1,249 | 1,252 |
| 128K | 128,881 | 109.00 s | 1,182 | 1,185 |

Sustained decode used 20-second steady-state cells after warmup, zero input
context, and continuous OpenAI stream usage:

| Concurrency | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| Aggregate tok/s | 48.9 | 112.9 | 154.0 | 188.2 | 218.2 | 239.4 | 253.9 | 266.8 |
| Per-request tok/s | 48.9 | 56.4 | 51.3 | 47.0 | 43.6 | 39.9 | 36.3 | 33.3 |

A separate 30-second C1 run measured 48.5 tok/s. No decode cell was underfilled,
capacity-limited, or errored.

Five-run, 2,047-position DCP4 KLD against the same verified BF16 reference:

| KV cache | Mean KLD | Sample SD | Min | Max |
| --- | ---: | ---: | ---: | ---: |
| NVFP4 DeepSeek MLA | 0.1124021 | 0.0025948 | 0.1086084 | 0.1156108 |
| FP8 | 0.1036666 | 0.0018374 | 0.1016535 | 0.1066077 |

Full methodology, raw JSON, and copyable Rich TUI logs are in
[benchmarks/2026-07-22](benchmarks/2026-07-22/README.md).

## Source

- [vLLM EXL3 integration PR](https://github.com/local-inference-lab/vllm/pull/139)
- [Sparkinfer EXL3 Trellis PR](https://github.com/local-inference-lab/b12x/pull/49)
- [Upstream GLM-5.2 model](https://huggingface.co/zai-org/GLM-5.2)
- [GLM-5 technical report](https://arxiv.org/abs/2602.15763)

## License

The model and this derivative are released under the MIT license. See
`LICENSE` and the upstream model card for attribution and usage terms.

## Credits and lineage

- [Z.ai](https://z.ai/) for
  [GLM-5.2](https://huggingface.co/zai-org/GLM-5.2).
- [Brandon Music](https://huggingface.co/brandonmusic) for the EXL3/TR3
  encoder work, owner corpus, derivative construction, runtime PR work,
  validation, and publication.
- [turboderp](https://github.com/turboderp-org) for
  [ExLlamaV3 v0.0.43 and EXL3](https://github.com/turboderp-org/exllamav3/tree/v0.0.43),
  including the MCG trellis implementation used by this checkpoint.
- [malaiwah](https://huggingface.co/malaiwah) for the
  [MTP-78 overlay](https://huggingface.co/malaiwah/GLM-5.2-EXL3-TR3-MTP78)
  and [calibration capture](https://huggingface.co/datasets/malaiwah/GLM-5.2-MTP78-calibration-capture).
- [Josh Cartu](https://github.com/jcartu) for the MTP-78 recipe and
  rank-sliced runtime work.
- [Luke Alonso](https://github.com/lukealonso) for
  [B12X](https://github.com/local-inference-lab/b12x) and the associated
  Blackwell kernel/runtime work.
- [Martin Vit](https://github.com/voipmonitor) and
  [yatesdr](https://github.com/yatesdr) for the Infernal Invocation / RC2 image
  engineering credited by the upstream MTP-78 overlay.
- Special thanks to [local-inference-lab](https://github.com/local-inference-lab)
  for its [vLLM fork](https://github.com/local-inference-lab/vllm), image,
  review, testing, and release engineering.

Canonical project attribution is recorded in the
[TR3 quantization provenance](https://github.com/local-inference-lab/rtx6kpro/blob/master/models/glm5.2/glm52-exl3-tr3-quantization-2026-07-26.md#credits).