upload checkpoint

README.md

@@ -0,0 +1,136 @@
+---
+language:
+- en
+library_name: transformers
+tags:
+- glm
+- MOE
+- pruning
+- compression
+license: mit
+name: cerebras/GLM-4.6-REAP-218B-A32B-FP8
+description: >
+  This model was obtained by uniformly pruning 40% of experts in GLM-4.6-FP8 using the REAP method.
+readme: >
+  https://huggingface.co/cerebras/GLM-4.6-REAP-218B-A32B-FP8/main/README.md
+license_link: https://huggingface.co/zai-org/GLM-4.6-FP8/blob/main/LICENSE
+pipeline_tag: text-generation
+base_model:
+- zai-org/GLM-4.6-FP8
+---
+
+<p align="center">
+  <em>𓌳 <strong>REAP</strong>𓌳  the Experts: Why Pruning Prevails for One-Shot MoE Compression</em><br>
+  <img src="https://i.imgur.com/rmzG3gg.png" alt="REAP" width="75%">
+</p>
+
+# GLM-4.6-REAP-218B-A32B-FP8
+
+## ✨ Highlights
+
+Introducing **GLM-4.6-REAP-218B-A32B-FP8**, a **memory-efficient compressed variant** of GLM-4.6-FP8 that maintains near-identical performance while being **40% lighter**.
+
+**Note: this is a BF16 version for more accurate downstream low-bit quantization. An [FP8 version](https://huggingface.co/cerebras/GLM-4.6-REAP-218B-A32B-FP8) is also available on HF.**
+
+This model was created using **REAP (Router-weighted Expert Activation Pruning)**, a novel expert pruning method that selectively removes redundant experts while preserving the router's independent control over remaining experts. Key features include:
+
+- **Near-Lossless Performance**: Maintains almost identical accuracy on code generation, agentic coding, and function calling tasks compared to the full 355B model
+- **40% Memory Reduction**: Compressed from 355B to 218B parameters, significantly lowering deployment costs and memory requirements
+- **Preserved Capabilities**: Retains all core functionalities including code generation, agentic workflows, repository-scale understanding, and function calling
+- **Drop-in Compatibility**: Works with vanilla vLLM - no source modifications or custom patches required
+- **Optimized for Real-World Use**: Particularly effective for resource-constrained environments, local deployments, and academic research
+---
+## 📋 Model Overview
+
+**GLM-4.6-REAP-218B-A32B-FP8** has the following specifications:
+
+- **Base Model**: GLM-4.6-FP8
+- **Compression Method**: REAP (Router-weighted Expert Activation Pruning)
+- **Compression Ratio**: 40% expert pruning
+- **Type**: Sparse Mixture-of-Experts (SMoE) Causal Language Model
+- **Number of Parameters**: 218B total, 32B activated per token
+- **Number of Layers**: 92
+- **Number of Attention Heads (GQA)**: 96 for Q and 8 for KV
+- **Number of Experts**: 96 (uniformly pruned from 160)
+- **Number of Activated Experts**: 8 per token
+- **Context Length**: 202,752 tokens
+- **License**: MIT
+
+---
+
+## 📊 Evaluations
+
+TBD for BF16 model. [Evalulation results available for the FP8 variant](https://huggingface.co/cerebras/GLM-4.6-REAP-218B-A32B-FP8#%F0%9F%93%8A-evaluations).
+
+For more details on the evaluation setup, refer to the [REAP arXiv preprint](https://arxiv.org/abs/2510.13999).
+
+---
+
+## 🚀 Deployment
+
+You can deploy the model directly using the **latest vLLM** (v0.11.0), no source modifications or custom patches required.
+
+```bash
+vllm serve cerebras/GLM-4.6-REAP-218B-A32B-FP8 \
+    --tensor-parallel-size 8 \
+    --tool-call-parser glm45 \
+    --enable-auto-tool-choice \
+    --enable-expert-parallel
+```
+
+If you encounter insufficient memory when running this model, you might need to set a lower value for `--max-num-seqs` flag (e.g. set to 64).
+
+
+## 🧩 Model Creation
+
+This checkpoint was created by applying the **REAP (Router-weighted Expert Activation Pruning)** method uniformly across all Mixture-of-Experts (MoE) blocks of **GLM-4.6-FP8**, with a **40% pruning rate**.
+
+### How REAP Works
+
+REAP selects experts to prune based on a novel **saliency criterion** that considers both:
+- **Router gate values**: How frequently and strongly the router activates each expert
+- **Expert activation norms**: The magnitude of each expert's output contributions
+
+This dual consideration ensures that experts contributing minimally to the layer's output are pruned, while preserving those that play critical roles in the model's computations.
+
+### Key Advantages
+
+- **One-Shot Compression**: No fine-tuning required after pruning - the model is immediately ready for deployment
+- **Preserved Router Control**: Unlike expert merging methods, REAP maintains the router's independent, input-dependent control over remaining experts, avoiding "functional subspace collapse"
+- **Generative Task Superiority**: REAP significantly outperforms expert merging approaches on generative benchmarks (code generation, creative writing, mathematical reasoning) while maintaining competitive performance on discriminative tasks
+
+### Calibration
+
+The model was calibrated using a diverse mixture of domain-specific datasets including:
+- Code generation samples ([evol-codealpaca](https://huggingface.co/datasets/theblackcat102/evol-codealpaca-v1))
+- Function calling examples ([xlam-function-calling](https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k))
+- Agentic multi-turn trajectories ([SWE-smith-trajectories](https://huggingface.co/datasets/SWE-bench/SWE-smith-trajectories))
+
+📚 For more details, refer to the following resources:
+
+- [🧾 arXiv Preprint](https://arxiv.org/abs/2510.13999)
+- [🧾 REAP Blog](https://www.cerebras.ai/blog/reap)
+- [💻 REAP Codebase (GitHub)](https://github.com/CerebrasResearch/reap)
+
+---
+
+## ⚖️ License
+
+This model is derived from
+**[`zai-org/GLM-4.6-FP8`](https://huggingface.co/zai-org/GLM-4.6-FP8)**
+and distributed under the **MIT license**.
+
+---
+
+## 🧾 Citation
+
+If you use this checkpoint, please cite the REAP paper:
+
+```bibtex
+@article{lasby-reap,
+  title={REAP the Experts: Why Pruning Prevails for One-Shot MoE compression},
+  author={Lasby, Mike and Lazarevich, Ivan and Sinnadurai, Nish and Lie, Sean and Ioannou, Yani and Thangarasa, Vithursan},
+  journal={arXiv preprint arXiv:2510.13999},
+  year={2025}
+}
+```

chat_template.jinja

@@ -0,0 +1,103 @@
+[gMASK]<sop>
+{%- if tools -%}
+<|system|>
+# Tools
+
+You may call one or more functions to assist with the user query.
+
+You are provided with function signatures within <tools></tools> XML tags:
+<tools>
+{% for tool in tools %}
+{{ tool | tojson(ensure_ascii=False) }}
+{% endfor %}
+</tools>
+
+For each function call, output the function name and arguments within the following XML format:
+<tool_call>{function-name}
+<arg_key>{arg-key-1}</arg_key>
+<arg_value>{arg-value-1}</arg_value>
+<arg_key>{arg-key-2}</arg_key>
+<arg_value>{arg-value-2}</arg_value>
+...
+</tool_call>{%- endif -%}
+{%- macro visible_text(content) -%}
+    {%- if content is string -%}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping -%}
+        {%- for item in content -%}
+            {%- if item is mapping and item.type == 'text' -%}
+                {{- item.text }}
+            {%- elif item is string -%}
+                {{- item }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{- content }}
+    {%- endif -%}
+{%- endmacro -%}
+{%- set ns = namespace(last_user_index=-1) %}
+{%- for m in messages %}
+    {%- if m.role == 'user' %}
+        {% set ns.last_user_index = loop.index0 -%}
+    {%- endif %}
+{%- endfor %}
+{% for m in messages %}
+{%- if m.role == 'user' -%}<|user|>
+{{ visible_text(m.content) }}
+{{- '/nothink' if (enable_thinking is defined and not enable_thinking and not visible_text(m.content).endswith("/nothink")) else '' -}}
+{%- elif m.role == 'assistant' -%}
+<|assistant|>
+{%- set reasoning_content = '' %}
+{%- set content = visible_text(m.content) %}
+{%- if m.reasoning_content is string %}
+    {%- set reasoning_content = m.reasoning_content %}
+{%- else %}
+    {%- if '</think>' in content %}
+        {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+        {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+    {%- endif %}
+{%- endif %}
+{%- if loop.index0 > ns.last_user_index and reasoning_content -%}
+{{ '\n<think>' + reasoning_content.strip() +  '</think>'}}
+{%- else -%}
+{{ '\n<think></think>' }}
+{%- endif -%}
+{%- if content.strip() -%}
+{{ '\n' + content.strip() }}
+{%- endif -%}
+{% if m.tool_calls %}
+{% for tc in m.tool_calls %}
+{%- if tc.function %}
+    {%- set tc = tc.function %}
+{%- endif %}
+{{ '\n<tool_call>' + tc.name }}
+{% set _args = tc.arguments %}
+{% for k, v in _args.items() %}
+<arg_key>{{ k }}</arg_key>
+<arg_value>{{ v | tojson(ensure_ascii=False) if v is not string else v }}</arg_value>
+{% endfor %}
+</tool_call>{% endfor %}
+{% endif %}
+{%- elif m.role == 'tool' -%}
+{%- if m.content is string -%}
+{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+    {{- '<|observation|>' }}
+{%- endif %}
+{{- '\n<tool_response>\n' }}
+{{- m.content }}
+{{- '\n</tool_response>' }}
+{%- else -%}
+<|observation|>{% for tr in m.content %}
+
+<tool_response>
+{{ tr.output if tr.output is defined else tr }}
+</tool_response>{% endfor -%}
+{% endif -%}
+{%- elif m.role == 'system' -%}
+<|system|>
+{{ visible_text(m.content) }}
+{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    <|assistant|>{{- '\n<think></think>' if (enable_thinking is defined and not enable_thinking) else '' -}}
+{%- endif -%}

config.json

@@ -0,0 +1,43 @@
+{
+  "architectures": [
+    "Glm4MoeForCausalLM"
+  ],
+  "attention_bias": true,
+  "attention_dropout": 0.0,
+  "eos_token_id": [
+    151329,
+    151336,
+    151338
+  ],
+  "first_k_dense_replace": 3,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 12288,
+  "max_position_embeddings": 202752,
+  "model_type": "glm4_moe",
+  "moe_intermediate_size": 1536,
+  "n_group": 1,
+  "n_routed_experts": 96,
+  "n_shared_experts": 1,
+  "norm_topk_prob": true,
+  "num_attention_heads": 96,
+  "num_experts_per_tok": 8,
+  "num_hidden_layers": 92,
+  "num_key_value_heads": 8,
+  "num_nextn_predict_layers": 0,
+  "pad_token_id": 151329,
+  "partial_rotary_factor": 0.5,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 1000000,
+  "routed_scaling_factor": 2.5,
+  "tie_word_embeddings": false,
+  "topk_group": 1,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.55.0",
+  "use_cache": true,
+  "use_qk_norm": true,
+  "vocab_size": 151552
+}

generation_config.json

@@ -0,0 +1,10 @@
+{
+  "_from_model_config": true,
+  "eos_token_id": [
+    151329,
+    151336,
+    151338
+  ],
+  "pad_token_id": 151329,
+  "transformers_version": "4.55.0"
+}

model-00001-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f213cb52e38e58b4df71754baf8e79463d4732e3bd3f9e193b6f360accf3206e
+size 4986172552

model-00002-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3967c803a41450bd06b6e89bfc1de39f5a329a69f8fa6beab4170072f85a41e0
+size 4992293480

model-00003-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b7b804b069cee3e5816a7a645b28057d6b1d2a80986d9e59d18ded802ad440d0
+size 4992293480

model-00004-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f93ccc1ae9e84c2cfa143fa72f0c75bb66b95877d88f1ace178da02f96cda078
+size 4992293480

model-00005-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8e1adfaa0d65ee00726d606ec247e68a366e029f73ecd8252e6dc7d4663c72d8
+size 4992293480

model-00006-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e27a78820dbf46e3bd4fecb4211ffa084fe2010801285429dfa29b6d9bbf5b73
+size 4992293480

model-00007-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6665027edc55b86bfd431ea6cea1b097e1b1205ace0ed728b3b288901e9031ab
+size 4992293480

model-00008-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3a6269274922c643b21799bebe33bfb0b6575f43a498bae4ec3b93739f6997c4
+size 4992293640

model-00009-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c991949a091038d04a95171128d75453576051ef68aed6c82fe094df8616f88a
+size 4992293792

model-00010-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c2bbd267acc7a5655741d9951d7d2b5cee22c8a8265dfb7cfa94ba92afc3749
+size 4992293792

model-00011-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:13e01bfb105bbc52e7ee077103ee0fa13aacfe26f17fde3bc9f77694b96f6ea2
+size 4992293792

model-00012-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b2008f34d590922db4f557d8ce8426f35a1513303a6aaaa1391fd5fff99f7f31
+size 4992293792

model-00013-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f1b5b1a419a917a80d76e1edd6f71e53834ee835be3652fcb43603b9e0eb43e4
+size 4992293792

model-00014-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4a9103e86f9916abab2d7f5bd791d7bfe137242068ca5a6e8f01a7123a0cfe20
+size 4992293792

model-00015-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e9cebf42037176c056b09eabebd81b1eaf1143414e4cd8a12b7e2fc8b2d01caf
+size 4992293792

model-00016-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:acbaad1879ebaedb156a7f37ab8ea6abc0e4b8694880e4c88db4a872ef359e49
+size 4992293792

model-00017-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b81d419097e8ae6272f8b3f04d3130dec68c72597fc9fa81c0754587359df57
+size 4992293792

model-00018-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff581dcedcdc0435eb9b8e2e103f54fafed4c90781bb12989cc0e9379a22d7a9
+size 4992293792

model-00019-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b08644d1a29b0a94433e1072d063a46cfb060f601c559e49a5962092511259f
+size 4992293792

model-00020-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ffe695d19fd52c544eaf05ddcd39bd63bafd3418820adae251590217f1b0973
+size 4992293792

model-00021-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:68c11b4bab1e82683f10b26a59ade004c9d00f5119c5a8723d3f677d58b9aa14
+size 4992293792

model-00022-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:501e96e832463df6cac1fed2c9f3a7b478d2f4538d660c6757d2479e61ca0260
+size 4992293792

model-00023-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7086a8f45e58a736746bf850b9bbe565873bcea99d067adae7888199f8090c25
+size 4992293792

model-00024-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4444ddf6719f671447a337e35aaebb87c1a37dc8a5456a58b235898316859f14
+size 4992293792

model-00025-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:47c917a252bd745880e9bde73dcc6f2d5b28835a0a1a53c6b3bfb31fc101fe8c
+size 4961840616

model-00026-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f0e21bd9a493d1b48fb883c18ced7c5e237ddd2d92b2325ac318c2f14712b102
+size 4997564904

model-00027-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:076f7df82cb2a2b07c6fb6ae5f9a9d81120716c6293c96c84be5371b7b605cb6
+size 4992293648

model-00028-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:214ee7b9cbf2b814186b8485e5c0a6dd679286e1dedbb1986ee29cbd243a3729
+size 4992293784

model-00029-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:672a4625ab1d3eb13d67fdc44e1ef694ed968f6846a9297f8264f3582f57f5a7
+size 4992293784

model-00030-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b6fcca6e532ff06f16b643263618801f7d5fcdbcb6a1a2c2f47b425503ea7e3c
+size 4992293784

model-00031-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:300677074feb61f932309518db12bf616e5ab531f560d382f148bcc705d029e2
+size 4992293792

model-00032-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ed488b0edcbacdcd690430843068ce20c066dd820c66e75931c1a2161fc790bc
+size 4992293792

model-00033-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:74183f9ef9b234c322d4cc0fa5785883e1709957be32b8c13c165fa0f1607f07
+size 4992293792

model-00034-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8827b19e6efc63650f8d6bd8adfbb5391ce8ed01d5380c55dfc0518878ef4e50
+size 4992293792

model-00035-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:922416ce9661049e00b5feec19d1d11faf6f25cde56c47d6d008f82e2e828b52
+size 4992293792

model-00036-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8f8292739cafc758cdb755d6bb4a14f77436580cce3d01391af37cefc7c7b9e7
+size 4992293792

model-00037-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2e4faa56cd877b62e41e10c9e7be27fdcee67280ab75053d83a0f4a47ad8b2ba
+size 4992293792

model-00038-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:15cdaec9523176d1d06b3daf8638f4c38b3701c593f38455321f93e8a84a81d0
+size 4992293792

model-00039-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c70f49e68d5f76da7de5a373f66f3d430bbc2134d116e90a67450a16f09a2614
+size 4992293792

model-00040-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:86a45cf9a15093f168136d69461b90cd7ef0c625c3bbe6eb3b498b98f2b52c1c
+size 4992293792

model-00041-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:09edce6fe6009c31bd6fc2db9e71f835fba8e6ecc0966cb620d47c6e431fbaef
+size 4992293792

model-00042-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8a0a0327319e646fe6c2ebba06ef968e03e55ebeaacfe46be8a5bbf7f685eb28
+size 4992293792

model-00043-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cb0d235a1847cd1fbb7b9532e69c6d1d3e5942bf4713bd01326be8597a2e553f
+size 4992293792

model-00044-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:daf3b00726c8b74313c309066422852d7bdb0e1620be77cfe9c12cff74a110db
+size 4992293792

model-00045-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4d05f7e1819825fdc69bca19143f11304d64984f8f8d190da27cc8f15c57d235
+size 4992293792

model-00046-of-00088.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5bd5a0b3fe1430b775829635f941fd4f9a9fd13a25ed3e4bad5c3eedf5bbf54b
+size 4992293792