Add reproduction article + graft bundle (stage.sh, prepare_checkpoint.py, sha256 manifest)

REPRODUCING.md

@@ -0,0 +1,113 @@
+# Giving a 753B Model Eyes: Reproducing GLM-5.2 Vision on 4× 96 GB GPUs
+
+*How [GLM-5.2-MXFP8-NVFP4-NF3-Hybrid-Vision](https://huggingface.co/0xSero/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid-Vision) was made: grafting the GLM-5.2-Vision tower onto madeby561's 4-card NF3 hybrid, and serving 250k context with vision on a single workstation.*
+
+---
+
+The starting point is [madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid](https://huggingface.co/madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid) — the full 753B GLM-5.2, all 256 experts per layer, no pruning, compressed to ≈341 GB with a three-tier scheme (NVFP4 for the 64 damage-ranked experts per layer, an in-house NF3 3-bit format for the other 192, MXFP8-served BF16 for everything else). It runs on four 96 GB sm120 GPUs and scores within one GPQA-Diamond question of full precision.
+
+But it's blind. The official vision variant exists as [baseten/GLM-5.2-Vision-NVFP4](https://huggingface.co/baseten/GLM-5.2-Vision-NVFP4) — 435 GB-class, needs 8 GPUs.
+
+The observation that makes this work: **the vision tower and projector are tiny and quantization-independent of the text backbone.** GLM-5.2-Vision's eyes are a MoonViT-3d tower plus a multimodal projector — ≈0.93 GB in NVFP4. Nothing about them cares whether the text experts underneath are FP8, NVFP4, or NF3. So you can lift them off baseten's checkpoint and bolt them onto the 4-card hybrid, byte-exact on both sides.
+
+## The five-minute path
+
+If you just want to run it:
+
+```bash
+hf download 0xSero/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid-Vision --local-dir ./glm52-vision
+MODEL_DIR=./glm52-vision docker compose up   # compose file ships in the repo
+```
+
+Serving image: `ghcr.io/0xsero/glm52-nf3-vision:v3-attn`. Needs 4× 96 GB sm120 GPUs. The rest of this article is how that artifact was produced, so you can rebuild it from primary sources — or repeat the trick on the next checkpoint pair.
+
+## Step 1 — Graft the checkpoint
+
+Everything in this step is in the model repo under [`graft/`](https://huggingface.co/0xSero/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid-Vision/tree/main/graft).
+
+```bash
+SOURCE_DIR=/models/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid \
+TARGET_DIR=/models/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid-Vision \
+./stage.sh
+```
+
+`stage.sh` downloads exactly eight files from baseten's repo at a **pinned revision** (`f6eab611…`): the vision tower and projector weights, the vision processor code (`kimi_k25_*.py`, `media_utils.py`, `preprocessor_config.json`), the vision `config.json`, and the vision chat template. `prepare_checkpoint.py` then:
+
+1. **Verifies every downloaded file against a sha256 manifest** (`asset-manifest.json`) — a moved revision or truncated download fails loudly before any assembly starts.
+2. **Hardlinks the 184 text shards** from the source checkpoint into the target — the graft costs ≈1 GB of disk, not 342 GB, and the text weights are byte-identical by construction, so every accuracy number measured on the base model carries over.
+3. **Performs the config surgery**: baseten's vision config becomes the outer `config.json` with `"architectures": ["Glm5vForConditionalGeneration"]` and `"model_type": "glm5v"`; the hybrid's entire config is embedded as `text_config`; the hybrid's `quantization_config` is hoisted to the top level so the loader still sees the NF3/NVFP4 tiering.
+4. **Merges the safetensors index**: the tower and projector tensors are appended to `model.safetensors.index.json`, and `total_parameters`/`total_size` are updated from the actual safetensors headers.
+5. **Writes `VISION_PROVENANCE.json`** — source hashes, vision repo + revision, parameter and byte counts — and assembles everything in a `.partial` directory that is atomically renamed at the end, so a crashed run never leaves a half-checkpoint.
+
+One post-graft fix that cost us a debugging session: the vision chat template changes the stop behavior, and generation initially would not terminate. The shipped `config.json` carries the EOS fix — if you re-derive the config yourself, compare against the shipped one.
+
+## Step 2 — The serving plugin
+
+vLLM has no `Glm5vForConditionalGeneration`. The trick is that it *does* have a complete implementation of the same vision architecture under a different name: Kimi-K2.5's MoonViT stack. The [`serving/`](https://huggingface.co/0xSero/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid-Vision/tree/main/serving) folder contains `glm52-vision-vllm`, a ~200-line package that bridges the two:
+
+- registers `Glm5vForConditionalGeneration` in vLLM's model registry via the standard `vllm.general_plugins` entry point (no vLLM source edits — `pip install` and it's active);
+- reuses vLLM's `KimiK25ForConditionalGeneration` / MoonViT / projector classes for the vision path, resolves GLM's `<|image|>` placeholder token, and instantiates the text backbone from the embedded `text_config`;
+- propagates the sparse-indexer attention overrides (`use_index_cache`, the 78-character `index_topk_pattern`) from the outer config into the text config, and patches vLLM's `SpeculativeConfig` so MTP setup reads the text config rather than choking on the multimodal wrapper.
+
+## Step 3 — Build the image
+
+The Dockerfile is four lines on top of madeby561's public serving image (which contains the NF3 kernel and hybrid loader):
+
+```dockerfile
+ARG BASE_IMAGE=madeby561/vllm-glm52-nvfp4-nf3-hybrid:v3
+FROM ${BASE_IMAGE}
+COPY pyproject.toml /opt/glm52-vision/pyproject.toml
+COPY src /opt/glm52-vision/src
+RUN /opt/venv/bin/pip install --no-deps /opt/glm52-vision
+```
+
+```bash
+cd serving && docker build -t glm52-nf3-vision:v3-attn .
+```
+
+The prebuilt result is `ghcr.io/0xsero/glm52-nf3-vision:v3-attn`.
+
+## Step 4 — Serve
+
+The repo's `docker-compose.yml` is the exact measured configuration. The load-bearing choices:
+
+| Flag | Value | Why |
+|---|---|---|
+| `--tensor-parallel-size` / `--decode-context-parallel-size` | 4 / 4 | DCP4 shards the KV cache across all four GPUs — this is what buys the big pool |
+| `--kv-cache-dtype` | `fp8` | `nvfp4_ds_mla` KV is unsupported by the MLA backend for sm120 in this runtime — and costs 30–50% decode where it does run |
+| `--gpu-memory-utilization` | 0.974 | weights take ≈85–88 GiB/GPU; this leaves the pool below |
+| `--max-model-len` | 250000 | fits the measured pool with 2× concurrency headroom |
+| `--attention-backend` / `--moe-backend` | `B12X_MLA_SPARSE` / `b12x` | the custom sparse-MLA + NF3/NVFP4 MoE kernels |
+| speculative config | **absent** | see gotchas |
+
+Boot takes ≈2–3 min (fastsafetensors, ≤2 GB shards). The line to look for:
+
+```text
+GPU KV cache size: 509,467 tokens
+Maximum concurrency for 250,000 tokens per request: 2.04x
+```
+
+## Step 5 — Validate
+
+Text, then vision, against the OpenAI-compatible endpoint:
+
+```bash
+curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
+  "model": "GLM-5.2-Vision",
+  "messages": [{"role":"user","content":[
+    {"type":"image_url","image_url":{"url":"https://ultralytics.com/images/bus.jpg"}},
+    {"type":"text","text":"What vehicle is in this image? One word."}]}]
+}'
+```
+
+Expected: `bus`. Image tokens come out of the same 250k context pool as text.
+
+## Gotchas, honestly
+
+- **MTP speculation is off.** The five-token MTP draft head generates drafts on the graft, but the accepted-token counter stays at zero. Rather than ship speculation that silently never accepts, the compose omits `--speculative-config`. Single-stream decode is therefore ≈24–37 tok/s on a PCIe-only (no-P2P) 4-card box — if your platform has working P2P, expect meaningfully more.
+- **Vision quality is inherited, not re-verified.** The tower rides byte-exact NVFP4 from baseten; the text tier is byte-exact madeby561 v3.6 (GPQA-Diamond 88.89 carries over). But no vision benchmark suite has been run on the combined artifact — validation so far is smoke-level.
+- **Trust the manifest, not the tag.** Every external input is pinned: the baseten revision, the sha256 of all eight vision assets, the base image tag, and the published image digest. If any hash fails, stop — a silently different tower is the kind of bug you find weeks later.
+
+## Credits
+
+Text hybrid, NF3 format + kernel, and serving image by [madeby561](https://huggingface.co/madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid). NVFP4 expert tier by [Luke Alonso](https://huggingface.co/lukealonso/GLM-5.2-NVFP4). Vision tower NVFP4 by [baseten](https://huggingface.co/baseten/GLM-5.2-Vision-NVFP4). MoonViT serving stack from vLLM's Kimi-K2.5 implementation. Graft + plugin: [0xSero](https://huggingface.co/0xSero).

graft/asset-manifest.json

@@ -0,0 +1,33 @@
+{
+  "revision": "f6eab6117386a0c69152fdf272dc65bfd0254f9f",
+  "repository": "baseten/GLM-5.2-Vision-NVFP4",
+  "files": {
+    "chat_template.jinja": {
+      "sha256": "6d172cae09cb494bd825a6b14d07e7b32a2c26988afe43d8bc2bbda91aab99c8"
+    },
+    "config.json": {
+      "sha256": "5a4c1c2e388e28373dcdd9d38c968246a23a261f7ad49908d94823f3a4616f65"
+    },
+    "kimi_k25_processor.py": {
+      "sha256": "cbfe0d8ca280568605011927fc80a515451ec6d1c01519ff89bd6c09dec702c4"
+    },
+    "kimi_k25_vision_processing.py": {
+      "sha256": "498e20753c9aa6b6224ddbcd05b04deb7e987b76595bf5c28ee8583f33ebc375"
+    },
+    "media_utils.py": {
+      "sha256": "77a6b1fed3580b0562443092928a0240c2989e3950377e93893952123ae48e3d"
+    },
+    "mm_projector.safetensors": {
+      "sha256": "e7c6ce8c27424f292e708e7bbb48ade57ea9f1aaddd28bd6a1020a860d9db80c",
+      "size": 99117136
+    },
+    "preprocessor_config.json": {
+      "sha256": "b938c8d816d50579bd8d9bba2ff0ffb738c528b9b05c6966c186d30a1a065041"
+    },
+    "vision_tower.safetensors": {
+      "sha256": "af1c927919bd073c2ab893bf2c6d9826100a06828c3b40f0128fabb4660a610d",
+      "size": 833769904
+    }
+  }
+}
+

graft/configuration_glm5v.py

@@ -0,0 +1,120 @@
+from transformers import AutoConfig
+from transformers.configuration_utils import PretrainedConfig
+
+
+class Glm5vVisionConfig(PretrainedConfig):
+    model_type = "glm5v_vision"
+
+    def __init__(
+        self,
+        patch_size: int = 14,
+        init_pos_emb_height: int = 64,
+        init_pos_emb_width: int = 64,
+        init_pos_emb_time: int = 4,
+        pos_emb_type: str = "divided_fixed",
+        num_attention_heads: int = 16,
+        num_hidden_layers: int = 27,
+        hidden_size: int = 1152,
+        intermediate_size: int = 4304,
+        merge_kernel_size=(2, 2),
+        video_attn_type: str = "spatial_temporal",
+        merge_type: str = "sd2_tpool",
+        mm_projector_type: str = "patchmerger",
+        mm_hidden_size: int = 6144,
+        vt_hidden_size: int | None = None,
+        projector_hidden_act: str = "gelu",
+        projector_ln_eps: float = 1e-5,
+        text_hidden_size: int = 6144,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.patch_size = patch_size
+        self.init_pos_emb_height = init_pos_emb_height
+        self.init_pos_emb_width = init_pos_emb_width
+        self.init_pos_emb_time = init_pos_emb_time
+        self.pos_emb_type = pos_emb_type
+        self.num_attention_heads = num_attention_heads
+        self.num_hidden_layers = num_hidden_layers
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.merge_kernel_size = merge_kernel_size
+        self.video_attn_type = video_attn_type
+        self.merge_type = merge_type
+        self.mm_projector_type = mm_projector_type
+        self.mm_hidden_size = mm_hidden_size
+        self.vt_hidden_size = vt_hidden_size if vt_hidden_size is not None else hidden_size
+        self.projector_hidden_act = projector_hidden_act
+        self.projector_ln_eps = projector_ln_eps
+        self.text_hidden_size = text_hidden_size
+
+    def __getattr__(self, name):
+        if name.startswith("vt_"):
+            values = object.__getattribute__(self, "__dict__")
+            base_name = name[3:]
+            if base_name in values:
+                return values[base_name]
+        raise AttributeError(name)
+
+
+class Glm5vConfig(PretrainedConfig):
+    model_type = "glm5v"
+
+    def __init__(
+        self,
+        text_config=None,
+        vision_config=None,
+        ignore_index: int = -100,
+        media_placeholder_token_id: int = 154854,
+        pad_token_id: int = 154820,
+        use_unified_vision_chunk: bool = True,
+        video_placeholder: str = "<|glm5v_video_placeholder|>",
+        encoder_only: bool = False,
+        language_only: bool = False,
+        **kwargs,
+    ):
+        if vision_config is None:
+            self.vision_config = Glm5vVisionConfig()
+        elif isinstance(vision_config, dict):
+            self.vision_config = Glm5vVisionConfig(**vision_config)
+        else:
+            self.vision_config = vision_config
+        raw_text_config = dict(text_config) if isinstance(text_config, dict) else None
+        if text_config is None:
+            self.text_config = AutoConfig.for_model("glm_moe_dsa")
+        elif isinstance(text_config, dict):
+            normalized_text_config = dict(text_config)
+            normalized_text_config.setdefault("model_type", "glm_moe_dsa")
+            normalized_text_config.pop("layer_types", None)
+            self.text_config = AutoConfig.for_model(**normalized_text_config)
+        else:
+            self.text_config = text_config
+        if raw_text_config is not None:
+            for key in ("qk_rope_head_dim", "index_topk_freq"):
+                if key in raw_text_config:
+                    setattr(self.text_config, key, raw_text_config[key])
+            if hasattr(self.text_config, "qk_nope_head_dim") and hasattr(
+                self.text_config,
+                "qk_rope_head_dim",
+            ):
+                self.text_config.qk_head_dim = (
+                    self.text_config.qk_nope_head_dim
+                    + self.text_config.qk_rope_head_dim
+                )
+        self.ignore_index = ignore_index
+        self.media_placeholder_token_id = media_placeholder_token_id
+        self.use_unified_vision_chunk = use_unified_vision_chunk
+        self.video_placeholder = video_placeholder
+        self.encoder_only = encoder_only
+        self.language_only = language_only
+        if getattr(self.text_config, "quantization_config", None) is not None:
+            self.quantization_config = self.text_config.quantization_config
+        super().__init__(pad_token_id=pad_token_id, **kwargs)
+
+    @property
+    def hidden_size(self) -> int:
+        return self.text_config.hidden_size
+
+    @property
+    def vocab_size(self) -> int:
+        return self.text_config.vocab_size
+

graft/prepare_checkpoint.py

@@ -0,0 +1,152 @@
+import argparse
+import hashlib
+import json
+import os
+import shutil
+import struct
+from pathlib import Path
+
+
+REPLACED_FILES = {
+    "chat_template.jinja",
+    "config.json",
+    "docker-compose.yml",
+    "model.safetensors.index.json",
+    "README.md",
+}
+
+
+def sha256(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as handle:
+        while chunk := handle.read(8 * 1024 * 1024):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def safetensors_header(path: Path) -> dict:
+    with path.open("rb") as handle:
+        header_size = struct.unpack("<Q", handle.read(8))[0]
+        return json.loads(handle.read(header_size))
+
+
+def verify_assets(assets: Path, manifest_path: Path) -> dict:
+    manifest = json.loads(manifest_path.read_text())
+    for name, expected in manifest["files"].items():
+        path = assets / name
+        if not path.is_file():
+            raise FileNotFoundError(path)
+        if expected.get("size") is not None and path.stat().st_size != expected["size"]:
+            raise ValueError(f"Unexpected size for {path}")
+        if sha256(path) != expected["sha256"]:
+            raise ValueError(f"Unexpected hash for {path}")
+    return manifest
+
+
+def add_weight_file(index: dict, path: Path) -> tuple[int, int]:
+    header = safetensors_header(path)
+    parameters = 0
+    payload_bytes = 0
+    for name, tensor in header.items():
+        if name == "__metadata__":
+            continue
+        index["weight_map"][name] = path.name
+        tensor_parameters = 1
+        for dimension in tensor["shape"]:
+            tensor_parameters *= dimension
+        parameters += tensor_parameters
+        payload_bytes += tensor["data_offsets"][1] - tensor["data_offsets"][0]
+    return parameters, payload_bytes
+
+
+def build_config(source: Path, assets: Path) -> dict:
+    source_config = json.loads((source / "config.json").read_text())
+    baseten_config = json.loads((assets / "config.json").read_text())
+    config = dict(baseten_config)
+    config["architectures"] = ["Glm5vForConditionalGeneration"]
+    config["auto_map"] = {"AutoConfig": "configuration_glm5v.Glm5vConfig"}
+    config["model_type"] = "glm5v"
+    config["text_config"] = source_config
+    config["quantization_config"] = source_config["quantization_config"]
+    config["vision_config"] = dict(baseten_config["vision_config"])
+    config["vision_config"]["mm_hidden_size"] = source_config["hidden_size"]
+    config["vision_config"]["text_hidden_size"] = source_config["hidden_size"]
+    return config
+
+
+def assemble(source: Path, target: Path, assets: Path, bundle: Path) -> None:
+    if not source.is_dir():
+        raise FileNotFoundError(source)
+    if target.exists():
+        raise FileExistsError(target)
+    partial = target.with_name(f".{target.name}.partial-{os.getpid()}")
+    if partial.exists():
+        raise FileExistsError(partial)
+    partial.mkdir(parents=True)
+    try:
+        for source_path in source.iterdir():
+            if not source_path.is_file() or source_path.name in REPLACED_FILES:
+                continue
+            destination = partial / source_path.name
+            if source_path.name.startswith("model-") and source_path.suffix == ".safetensors":
+                os.link(source_path, destination)
+            else:
+                shutil.copy2(source_path, destination)
+        for name in (
+            "chat_template.jinja",
+            "kimi_k25_processor.py",
+            "kimi_k25_vision_processing.py",
+            "media_utils.py",
+            "preprocessor_config.json",
+        ):
+            shutil.copy2(assets / name, partial / name)
+        for name in ("vision_tower.safetensors", "mm_projector.safetensors"):
+            os.link(assets / name, partial / name)
+        shutil.copy2(bundle / "configuration_glm5v.py", partial / "configuration_glm5v.py")
+        config = build_config(source, assets)
+        (partial / "config.json").write_text(json.dumps(config, indent=2) + "\n")
+        index = json.loads((source / "model.safetensors.index.json").read_text())
+        total_parameters = 0
+        total_size = 0
+        for name in ("vision_tower.safetensors", "mm_projector.safetensors"):
+            parameters, payload_bytes = add_weight_file(index, partial / name)
+            total_parameters += parameters
+            total_size += payload_bytes
+        metadata = index.setdefault("metadata", {})
+        if "total_parameters" in metadata:
+            metadata["total_parameters"] = int(metadata["total_parameters"]) + total_parameters
+        metadata["total_size"] = int(metadata.get("total_size", 0)) + total_size
+        (partial / "model.safetensors.index.json").write_text(
+            json.dumps(index, indent=2, sort_keys=True) + "\n"
+        )
+        provenance = {
+            "base_checkpoint": str(source),
+            "base_config_sha256": sha256(source / "config.json"),
+            "base_index_sha256": sha256(source / "model.safetensors.index.json"),
+            "vision_repository": "baseten/GLM-5.2-Vision-NVFP4",
+            "vision_revision": "f6eab6117386a0c69152fdf272dc65bfd0254f9f",
+            "vision_parameters": total_parameters,
+            "vision_payload_bytes": total_size,
+        }
+        (partial / "VISION_PROVENANCE.json").write_text(
+            json.dumps(provenance, indent=2, sort_keys=True) + "\n"
+        )
+        partial.rename(target)
+    except BaseException:
+        shutil.rmtree(partial, ignore_errors=True)
+        raise
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--source", type=Path, required=True)
+    parser.add_argument("--target", type=Path, required=True)
+    parser.add_argument("--assets", type=Path, required=True)
+    parser.add_argument("--bundle", type=Path, required=True)
+    args = parser.parse_args()
+    verify_assets(args.assets, args.bundle / "asset-manifest.json")
+    assemble(args.source.resolve(), args.target.resolve(), args.assets.resolve(), args.bundle.resolve())
+
+
+if __name__ == "__main__":
+    main()

graft/stage.sh

@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+BUNDLE_DIR="$(cd "$(dirname "$0")" && pwd)"
+SOURCE_DIR="${SOURCE_DIR:-/mnt/llm_models/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid}"
+TARGET_DIR="${TARGET_DIR:-/mnt/llm_models/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid-Vision}"
+ASSET_DIR="${ASSET_DIR:-/mnt/llm_models/.glm52-vision-assets/f6eab6117386a0c69152fdf272dc65bfd0254f9f}"
+REVISION="f6eab6117386a0c69152fdf272dc65bfd0254f9f"
+
+mkdir -p "$ASSET_DIR"
+hf download baseten/GLM-5.2-Vision-NVFP4 \
+  --revision "$REVISION" \
+  --include \
+  config.json \
+  chat_template.jinja \
+  preprocessor_config.json \
+  kimi_k25_processor.py \
+  kimi_k25_vision_processing.py \
+  media_utils.py \
+  vision_tower.safetensors \
+  mm_projector.safetensors \
+  --local-dir "$ASSET_DIR"
+python3 "$BUNDLE_DIR/prepare_checkpoint.py" \
+  --source "$SOURCE_DIR" \
+  --target "$TARGET_DIR" \
+  --assets "$ASSET_DIR" \
+  --bundle "$BUNDLE_DIR"