| --- |
| license: mit |
| base_model: zai-org/GLM-5.2-FP8 |
| base_model_relation: quantized |
| tags: |
| - glm |
| - mixture-of-experts |
| - int4 |
| - apple-silicon |
| - ssd-streaming |
| --- |
| |
| # GLM-5.2 — int4 container for iliria (full 744B, SSD-streamed) |
|
|
| [Explore the model guide](https://huggingface.co/spaces/philipjohnbasile/local-ai-guide) · [All public work](https://huggingface.co/philipjohnbasile) |
|
|
| ## Release at a glance |
|
|
| | | This artifact | |
| |---|---| |
| | Purpose | The unpruned GLM 5.2 expert set in an int4 container with an int8 MTP head for the iliria engine. | |
| | Runtime | iliria streams experts from SSD. The documented setup is an Apple Silicon 128 GB machine with roughly 384 GB of weight storage. | |
| | Status | Recorded runtime; see the evidence and limits below. | |
| | Tensor download | 383.94 GB (357.57 GiB) of root `.safetensors` files, including any root sidecars. This is a file-size total, not peak RAM. | |
| | Read first | The on-disk payload is not resident RAM. These files are not interchangeable with the pruned MLX build; consult iliria for runtime limits. | |
|
|
| The **full** GLM-5.2 (744B total / 39B active) quantized to int4 with an int8 MTP head, packaged |
| as the container that [**iliria**](https://github.com/PhilipJohnBasile/iliria) serves directly. |
|
|
| iliria runs this model on a single 128 GB Apple Silicon machine by **streaming experts from the |
| SSD** rather than holding all 744B parameters in memory. This repo exists so you don't have to |
| run that conversion yourself — otherwise the on-ramp is downloading ~372 GB of FP8 weights and |
| converting them locally for hours. |
|
|
| ## This is not the pruned build |
|
|
| If you came from |
| [GLM-5.2-Demolition-q4a4-soul-MLX](https://huggingface.co/philipjohnbasile/GLM-5.2-Demolition-q4a4-soul-MLX), |
| note these are different artifacts and **not interchangeable**: |
|
|
| | | this repo | the demolition build | |
| |---|---|---| |
| | model | full 744B, unpruned | expert-pruned + LoRA-healed | |
| | root tensor size | ~384 GB (int4) | ~105 GB / 98 GiB | |
| | format | iliria container (`out-*.safetensors`) | MLX | |
| | runs on | iliria | MLX / mlx-lm | |
|
|
| ## Requirements |
|
|
| - Apple Silicon **M-Max / M-Ultra class with 128 GB** unified memory |
| - **~384 GB free SSD** (the weights are streamed from disk, not loaded into RAM) |
| - [iliria](https://github.com/PhilipJohnBasile/iliria) |
|
|
| ## Use |
|
|
| ```bash |
| hf download philipjohnbasile/GLM-5.2-colibri-int4-with-int8-mtp --local-dir ./glm52-int4 |
| # then, from iliria: |
| ./illi serve --model ./glm52-int4 --ram 60 |
| ``` |
|
|
| That exposes an OpenAI-compatible endpoint. See the iliria README for the engine's flags, |
| context/RAM tuning, and measured throughput — expect deep-tier speeds, not chat speeds: this is |
| a 744B model reading experts off an SSD for every token. |
|
|
| ## Contents |
|
|
| 164 `out-*.safetensors` shards (int4 experts + int8 MTP head) plus `config.json`, |
| `generation_config.json`, and the tokenizer. Nothing else — no runtime state, no caches. |
|
|
| ## Provenance |
|
|
| Quantized from [`zai-org/GLM-5.2-FP8`](https://huggingface.co/zai-org/GLM-5.2-FP8) (MIT). |
| Quantization only — no pruning, no retraining, no distillation; the full expert set is intact. |
| Produced by iliria's `tools/convert_fp8_to_int4.py`. Released under MIT, matching the base model. |
|
|
