REPRODUCING.md

Giving a 753B Model Eyes: Reproducing GLM-5.2 Vision on 4× 96 GB GPUs

How GLM-5.2-MXFP8-NVFP4-NF3-Hybrid-Vision was made: grafting the GLM-5.2-Vision tower onto madeby561's 4-card NF3 hybrid, and serving 250k context with vision on a single workstation.

---

The starting point is madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid — the full 753B GLM-5.2, all 256 experts per layer, no pruning, compressed to ≈341 GB with a three-tier scheme (NVFP4 for the 64 damage-ranked experts per layer, an in-house NF3 3-bit format for the other 192, MXFP8-served BF16 for everything else). It runs on four 96 GB sm120 GPUs and scores within one GPQA-Diamond question of full precision.

But it's blind. The official vision variant exists as baseten/GLM-5.2-Vision-NVFP4 — 435 GB-class, needs 8 GPUs.

The observation that makes this work: the vision tower and projector are tiny and quantization-independent of the text backbone. GLM-5.2-Vision's eyes are a MoonViT-3d tower plus a multimodal projector — ≈0.93 GB in NVFP4. Nothing about them cares whether the text experts underneath are FP8, NVFP4, or NF3. So you can lift them off baseten's checkpoint and bolt them onto the 4-card hybrid, byte-exact on both sides.

The five-minute path

If you just want to run it:

hf download 0xSero/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid-Vision --local-dir ./glm52-vision
MODEL_DIR=./glm52-vision docker compose up   # compose file ships in the repo

Serving image: ghcr.io/0xsero/glm52-nf3-vision:v3-attn. Needs 4× 96 GB sm120 GPUs. The rest of this article is how that artifact was produced, so you can rebuild it from primary sources — or repeat the trick on the next checkpoint pair.

Step 1 — Graft the checkpoint

Everything in this step is in the model repo under graft/.

SOURCE_DIR=/models/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid \
TARGET_DIR=/models/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid-Vision \
./stage.sh

stage.sh downloads exactly eight files from baseten's repo at a pinned revision (f6eab611…): the vision tower and projector weights, the vision processor code (kimi_k25_*.py, media_utils.py, preprocessor_config.json), the vision config.json, and the vision chat template. prepare_checkpoint.py then:

1. Verifies every downloaded file against a sha256 manifest (asset-manifest.json) — a moved revision or truncated download fails loudly before any assembly starts.
2. Hardlinks the 184 text shards from the source checkpoint into the target — the graft costs ≈1 GB of disk, not 342 GB, and the text weights are byte-identical by construction, so every accuracy number measured on the base model carries over.
3. Performs the config surgery: baseten's vision config becomes the outer config.json with "architectures": ["Glm5vForConditionalGeneration"] and "model_type": "glm5v"; the hybrid's entire config is embedded as text_config; the hybrid's quantization_config is hoisted to the top level so the loader still sees the NF3/NVFP4 tiering.
4. Merges the safetensors index: the tower and projector tensors are appended to model.safetensors.index.json, and total_parameters/total_size are updated from the actual safetensors headers.
5. Writes VISION_PROVENANCE.json — source hashes, vision repo + revision, parameter and byte counts — and assembles everything in a .partial directory that is atomically renamed at the end, so a crashed run never leaves a half-checkpoint.

One post-graft fix that cost us a debugging session: the vision chat template changes the stop behavior, and generation initially would not terminate. The shipped config.json carries the EOS fix — if you re-derive the config yourself, compare against the shipped one.

Step 2 — The serving plugin

vLLM has no Glm5vForConditionalGeneration. The trick is that it does have a complete implementation of the same vision architecture under a different name: Kimi-K2.5's MoonViT stack. The serving/ folder contains glm52-vision-vllm, a ~200-line package that bridges the two:

• registers Glm5vForConditionalGeneration in vLLM's model registry via the standard vllm.general_plugins entry point (no vLLM source edits — pip install and it's active);
• reuses vLLM's KimiK25ForConditionalGeneration / MoonViT / projector classes for the vision path, resolves GLM's <|image|> placeholder token, and instantiates the text backbone from the embedded text_config;
• propagates the sparse-indexer attention overrides (use_index_cache, the 78-character index_topk_pattern) from the outer config into the text config, and patches vLLM's SpeculativeConfig so MTP setup reads the text config rather than choking on the multimodal wrapper.

Step 3 — Build the image

The Dockerfile is four lines on top of madeby561's public serving image (which contains the NF3 kernel and hybrid loader):

ARG BASE_IMAGE=madeby561/vllm-glm52-nvfp4-nf3-hybrid:v3
FROM ${BASE_IMAGE}
COPY pyproject.toml /opt/glm52-vision/pyproject.toml
COPY src /opt/glm52-vision/src
RUN /opt/venv/bin/pip install --no-deps /opt/glm52-vision

cd serving && docker build -t glm52-nf3-vision:v3-attn .

The prebuilt result is ghcr.io/0xsero/glm52-nf3-vision:v3-attn.

Step 4 — Serve

The repo's docker-compose.yml is the exact measured configuration. The load-bearing choices:

Flag	Value	Why
--tensor-parallel-size / --decode-context-parallel-size	4 / 4	DCP4 shards the KV cache across all four GPUs — this is what buys the big pool
--kv-cache-dtype	fp8	nvfp4_ds_mla KV is unsupported by the MLA backend for sm120 in this runtime — and costs 30–50% decode where it does run
--gpu-memory-utilization	0.974	weights take ≈85–88 GiB/GPU; this leaves the pool below
--max-model-len	250000	fits the measured pool with 2× concurrency headroom
--attention-backend / --moe-backend	B12X_MLA_SPARSE / b12x	the custom sparse-MLA + NF3/NVFP4 MoE kernels
speculative config	absent	see gotchas

Boot takes ≈2–3 min (fastsafetensors, ≤2 GB shards). The line to look for:

GPU KV cache size: 509,467 tokens
Maximum concurrency for 250,000 tokens per request: 2.04x

Step 5 — Validate

Text, then vision, against the OpenAI-compatible endpoint:

curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "GLM-5.2-Vision",
  "messages": [{"role":"user","content":[
    {"type":"image_url","image_url":{"url":"https://ultralytics.com/images/bus.jpg"}},
    {"type":"text","text":"What vehicle is in this image? One word."}]}]
}'

Expected: bus. Image tokens come out of the same 250k context pool as text.

Gotchas, honestly

• MTP speculation is off. The five-token MTP draft head generates drafts on the graft, but the accepted-token counter stays at zero. Rather than ship speculation that silently never accepts, the compose omits --speculative-config. Single-stream decode is therefore ≈24–37 tok/s on a PCIe-only (no-P2P) 4-card box — if your platform has working P2P, expect meaningfully more.
• Vision quality is inherited, not re-verified. The tower rides byte-exact NVFP4 from baseten; the text tier is byte-exact madeby561 v3.6 (GPQA-Diamond 88.89 carries over). But no vision benchmark suite has been run on the combined artifact — validation so far is smoke-level.
• Trust the manifest, not the tag. Every external input is pinned: the baseten revision, the sha256 of all eight vision assets, the base image tag, and the published image digest. If any hash fails, stop — a silently different tower is the kind of bug you find weeks later.

Credits

Text hybrid, NF3 format + kernel, and serving image by madeby561. NVFP4 expert tier by Luke Alonso. Vision tower NVFP4 by baseten. MoonViT serving stack from vLLM's Kimi-K2.5 implementation. Graft + plugin: 0xSero.