REPRODUCING.md

# Giving a 753B Model Eyes: Reproducing GLM-5.2 Vision on 4× 96 GB GPUs

*How [GLM-5.2-MXFP8-NVFP4-NF3-Hybrid-Vision](https://huggingface.co/0xSero/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid-Vision) was made: grafting the GLM-5.2-Vision tower onto madeby561's 4-card NF3 hybrid, and serving 250k context with vision on a single workstation.*

---

The starting point is [madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid](https://huggingface.co/madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid) — the full 753B GLM-5.2, all 256 experts per layer, no pruning, compressed to ≈341 GB with a three-tier scheme (NVFP4 for the 64 damage-ranked experts per layer, an in-house NF3 3-bit format for the other 192, MXFP8-served BF16 for everything else). It runs on four 96 GB sm120 GPUs and scores within one GPQA-Diamond question of full precision.

But it's blind. The official vision variant exists as [baseten/GLM-5.2-Vision-NVFP4](https://huggingface.co/baseten/GLM-5.2-Vision-NVFP4) — 435 GB-class, needs 8 GPUs.

The observation that makes this work: **the vision tower and projector are tiny and quantization-independent of the text backbone.** GLM-5.2-Vision's eyes are a MoonViT-3d tower plus a multimodal projector — ≈0.93 GB in NVFP4. Nothing about them cares whether the text experts underneath are FP8, NVFP4, or NF3. So you can lift them off baseten's checkpoint and bolt them onto the 4-card hybrid, byte-exact on both sides.

## The five-minute path

If you just want to run it:

```bash
hf download 0xSero/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid-Vision --local-dir ./glm52-vision
MODEL_DIR=./glm52-vision docker compose up   # compose file ships in the repo
```

Serving image: `ghcr.io/0xsero/glm52-nf3-vision:v3-attn`. Needs 4× 96 GB sm120 GPUs. The rest of this article is how that artifact was produced, so you can rebuild it from primary sources — or repeat the trick on the next checkpoint pair.

## Step 1 — Graft the checkpoint

Everything in this step is in the model repo under [`graft/`](https://huggingface.co/0xSero/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid-Vision/tree/main/graft).

```bash
SOURCE_DIR=/models/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid \
TARGET_DIR=/models/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid-Vision \
./stage.sh
```

`stage.sh` downloads exactly eight files from baseten's repo at a **pinned revision** (`f6eab611…`): the vision tower and projector weights, the vision processor code (`kimi_k25_*.py`, `media_utils.py`, `preprocessor_config.json`), the vision `config.json`, and the vision chat template. `prepare_checkpoint.py` then:

1. **Verifies every downloaded file against a sha256 manifest** (`asset-manifest.json`) — a moved revision or truncated download fails loudly before any assembly starts.
2. **Hardlinks the 184 text shards** from the source checkpoint into the target — the graft costs ≈1 GB of disk, not 342 GB, and the text weights are byte-identical by construction, so every accuracy number measured on the base model carries over.
3. **Performs the config surgery**: baseten's vision config becomes the outer `config.json` with `"architectures": ["Glm5vForConditionalGeneration"]` and `"model_type": "glm5v"`; the hybrid's entire config is embedded as `text_config`; the hybrid's `quantization_config` is hoisted to the top level so the loader still sees the NF3/NVFP4 tiering.
4. **Merges the safetensors index**: the tower and projector tensors are appended to `model.safetensors.index.json`, and `total_parameters`/`total_size` are updated from the actual safetensors headers.
5. **Writes `VISION_PROVENANCE.json`** — source hashes, vision repo + revision, parameter and byte counts — and assembles everything in a `.partial` directory that is atomically renamed at the end, so a crashed run never leaves a half-checkpoint.

One post-graft fix that cost us a debugging session: the vision chat template changes the stop behavior, and generation initially would not terminate. The shipped `config.json` carries the EOS fix — if you re-derive the config yourself, compare against the shipped one.

## Step 2 — The serving plugin

vLLM has no `Glm5vForConditionalGeneration`. The trick is that it *does* have a complete implementation of the same vision architecture under a different name: Kimi-K2.5's MoonViT stack. The [`serving/`](https://huggingface.co/0xSero/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid-Vision/tree/main/serving) folder contains `glm52-vision-vllm`, a ~200-line package that bridges the two:

- registers `Glm5vForConditionalGeneration` in vLLM's model registry via the standard `vllm.general_plugins` entry point (no vLLM source edits — `pip install` and it's active);
- reuses vLLM's `KimiK25ForConditionalGeneration` / MoonViT / projector classes for the vision path, resolves GLM's `<|image|>` placeholder token, and instantiates the text backbone from the embedded `text_config`;
- propagates the sparse-indexer attention overrides (`use_index_cache`, the 78-character `index_topk_pattern`) from the outer config into the text config, and patches vLLM's `SpeculativeConfig` so MTP setup reads the text config rather than choking on the multimodal wrapper.

## Step 3 — Build the image

The Dockerfile is four lines on top of madeby561's public serving image (which contains the NF3 kernel and hybrid loader):

```dockerfile
ARG BASE_IMAGE=madeby561/vllm-glm52-nvfp4-nf3-hybrid:v3
FROM ${BASE_IMAGE}
COPY pyproject.toml /opt/glm52-vision/pyproject.toml
COPY src /opt/glm52-vision/src
RUN /opt/venv/bin/pip install --no-deps /opt/glm52-vision
```

```bash
cd serving && docker build -t glm52-nf3-vision:v3-attn .
```

The prebuilt result is `ghcr.io/0xsero/glm52-nf3-vision:v3-attn`.

## Step 4 — Serve

The repo's `docker-compose.yml` is the exact measured configuration. The load-bearing choices:

| Flag | Value | Why |
|---|---|---|
| `--tensor-parallel-size` / `--decode-context-parallel-size` | 4 / 4 | DCP4 shards the KV cache across all four GPUs — this is what buys the big pool |
| `--kv-cache-dtype` | `fp8` | `nvfp4_ds_mla` KV is unsupported by the MLA backend for sm120 in this runtime — and costs 30–50% decode where it does run |
| `--gpu-memory-utilization` | 0.974 | weights take ≈85–88 GiB/GPU; this leaves the pool below |
| `--max-model-len` | 250000 | fits the measured pool with 2× concurrency headroom |
| `--attention-backend` / `--moe-backend` | `B12X_MLA_SPARSE` / `b12x` | the custom sparse-MLA + NF3/NVFP4 MoE kernels |
| speculative config | **absent** | see gotchas |

Boot takes ≈2–3 min (fastsafetensors, ≤2 GB shards). The line to look for:

```text
GPU KV cache size: 509,467 tokens
Maximum concurrency for 250,000 tokens per request: 2.04x
```

## Step 5 — Validate

Text, then vision, against the OpenAI-compatible endpoint:

```bash
curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "GLM-5.2-Vision",
  "messages": [{"role":"user","content":[
    {"type":"image_url","image_url":{"url":"https://ultralytics.com/images/bus.jpg"}},
    {"type":"text","text":"What vehicle is in this image? One word."}]}]
}'
```

Expected: `bus`. Image tokens come out of the same 250k context pool as text.

## Gotchas, honestly

- **MTP speculation is off.** The five-token MTP draft head generates drafts on the graft, but the accepted-token counter stays at zero. Rather than ship speculation that silently never accepts, the compose omits `--speculative-config`. Single-stream decode is therefore ≈24–37 tok/s on a PCIe-only (no-P2P) 4-card box — if your platform has working P2P, expect meaningfully more.
- **Vision quality is inherited, not re-verified.** The tower rides byte-exact NVFP4 from baseten; the text tier is byte-exact madeby561 v3.6 (GPQA-Diamond 88.89 carries over). But no vision benchmark suite has been run on the combined artifact — validation so far is smoke-level.
- **Trust the manifest, not the tag.** Every external input is pinned: the baseten revision, the sha256 of all eight vision assets, the base image tag, and the published image digest. If any hash fails, stop — a silently different tower is the kind of bug you find weeks later.

## Credits

Text hybrid, NF3 format + kernel, and serving image by [madeby561](https://huggingface.co/madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid). NVFP4 expert tier by [Luke Alonso](https://huggingface.co/lukealonso/GLM-5.2-NVFP4). Vision tower NVFP4 by [baseten](https://huggingface.co/baseten/GLM-5.2-Vision-NVFP4). MoonViT serving stack from vLLM's Kimi-K2.5 implementation. Graft + plugin: [0xSero](https://huggingface.co/0xSero).