---
license: mit
base_model:
- madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid
- baseten/GLM-5.2-Vision-NVFP4
- zai-org/GLM-5.2
- lukealonso/GLM-5.2-NVFP4
base_model_relation: quantized
pipeline_tag: image-text-to-text
tags:
- glm
- moe
- nvfp4
- nf3
- mxfp8
- vision
- multimodal
- quantization
---

# GLM-5.2-MXFP8-NVFP4-NF3-Hybrid-Vision

> ⚠️ **Usable — read the hazards.** Loads ONLY through the custom serving image below
> (in-house NF3 3-bit kernel + a small vision plugin — no official vLLM/inference stack
> runs it). Serves on 4× 96GB sm120 GPUs with a **509,467-token FP8 KV pool @ 250k max
> context** at DCP4 (measured live 2026-07-23) — 2.04× concurrency at full length.
> NOT the official model's 1M context, and not officially supported.

**The full GLM-5.2 — 753B, all 256 experts, no pruning — with eyes, on 4× 96GB GPUs.**

This is [madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid](https://huggingface.co/madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid)
(v3.6 text checkpoint, carried over byte-exact) with the GLM-5.2-Vision tower and
projector grafted on from
[baseten/GLM-5.2-Vision-NVFP4](https://huggingface.co/baseten/GLM-5.2-Vision-NVFP4)
(revision `f6eab611`, ≈0.93 GB of vision payload: MoonViT-3d tower + multimodal
projector, served through vLLM's Kimi-K2.5 vision stack via the bundled plugin).

The text tier is exactly the hybrid recipe from the base card:
- **NVFP4** for the 64 damage-ranked experts per layer (byte-exact from
  [lukealonso/GLM-5.2-NVFP4](https://huggingface.co/lukealonso/GLM-5.2-NVFP4) — credit
  to Luke for the best-in-class low-KLD 4-bit tier);
- **NF3** — madeby561's in-house 3-bit expert format with a matching custom sm120
  kernel — for the long tail (192 experts/layer);
- non-expert layers **stored true BF16, served as MXFP8** (single BF16→mxfp8 hop at
  load), which is what buys the large KV pool.

≈342 GB total (184 × ≤2 GB shards + vision payload). The only GLM-5.2 *with vision*
that fits 4 cards with a 500k-class KV pool.

## Accuracy

**Text tier:** identical weights to the base model, so its measured numbers carry
over unchanged — GPQA-Diamond **88.89** (vs 89.39 full NVFP4 on 8 GPUs, vs 86.87 for
the REAP-594B prune), top-1 agreement ≈99.98% vs FP8, KLD 0.126 in the community
vs-BF16 frame (Luke NVFP4 anchor: 0.060). Full tables, protocol, and the
vs-FP8 / vs-BF16 frame warnings are on the
[base model card](https://huggingface.co/madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid) —
they were measured on the text checkpoint this repo carries, not re-run here.

**Vision tier:** the tower/projector ride NVFP4 exactly as shipped by baseten; no
additional quantization was applied here. Validation so far is smoke-level (text
generation `TEXT_OK`, image captioning verified on a COCO bus photo, clean post-start
error scan). No vision benchmark suite has been run on this artifact yet — treat
vision quality as inherited-but-unverified.

## Serving

Everything is baked into one image — NF3 kernel, hybrid loader, native mxfp8 tier,
and the `glm52-vision-vllm` plugin that registers the `Glm5vForConditionalGeneration`
architecture:

```text
ghcr.io/0xsero/glm52-nf3-vision:v3-attn
```

The `docker-compose.yml` in this repo is the exact measured configuration:

```bash
hf download 0xSero/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid-Vision --local-dir ./glm52-vision
MODEL_DIR=./glm52-vision docker compose up
```

Needs 4× 96GB sm120 GPUs + ≈64 GB RAM. Serves **DCP4, FP8 KV** with a
**509,467-token KV pool @ 250,000 max context** (util 0.974; ≈85–88 GiB weights per
GPU). Boots in ≈2–3 min (fastsafetensors, ≤2 GB shards). Single-stream decode
≈24–37 t/s on a PCIe-only (no-P2P) 4-card box. The compose mounts
`mxfp8_tier_nokvb.json` — the measured overlay convention from the base card
(kv_b_proj + shared_experts stay true BF16).

Vision requests use standard OpenAI `image_url` content parts against
`/v1/chat/completions`; the image token budget comes out of the same context pool.

### Differences from the text-only base serving

- **MTP is off.** The five-token MTP draft head generates drafts on this graft but
  the accepted-token counter stays at zero, so speculation is disabled rather than
  shipped broken.
- **FP8 KV only.** `nvfp4_ds_mla` KV is unsupported by the MLA attention backend for
  sm120 in this runtime.
- `config.json` carries the EOS fix applied 2026-07-22 (generation previously would
  not stop on the vision template).

### Build it yourself

The image is `madeby561/vllm-glm52-nvfp4-nf3-hybrid:v3` (public, Docker Hub) plus a
pip install of the small plugin in [`serving/`](./serving/):

```bash
cd serving
docker build -t glm52-nf3-vision:v3-attn .
```

*Text hybrid by [madeby561](https://huggingface.co/madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid);
NVFP4 expert tier by [Luke Alonso](https://huggingface.co/lukealonso/GLM-5.2-NVFP4);
vision tower NVFP4 by [baseten](https://huggingface.co/baseten/GLM-5.2-Vision-NVFP4);
vision graft + serving plugin by 0xSero. Built to answer the follow-up question: can
the 4-card full-model box also see? Smoke says yes — benchmarks pending.*