upload checkpoint

README.md

@@ -0,0 +1,184 @@
+---
+language:
+- en
+library_name: transformers
+tags:
+- glm
+- MOE
+- pruning
+- compression
+license: mit
+name: cerebras/GLM-4.7-REAP-218B-A32B-FP8
+description: >
+  This model was obtained by uniformly pruning 40% of experts in GLM-4.7-FP8 using the REAP method.
+readme: >
+  https://huggingface.co/cerebras/GLM-4.7-REAP-218B-A32B-FP8/main/README.md
+license_link: https://huggingface.co/zai-org/GLM-4.7-FP8/blob/main/LICENSE
+pipeline_tag: text-generation
+base_model:
+- zai-org/GLM-4.7-FP8
+---
+
+<p align="center">
+  <em>𓌳 <strong>REAP</strong>𓌳  the Experts: Why Pruning Prevails for One-Shot MoE Compression</em><br>
+  <img src="https://i.imgur.com/rmzG3gg.png" alt="REAP" width="75%">
+</p>
+
+# GLM-4.7-REAP-218B-A32B-FP8
+
+## ✨ Highlights
+
+Introducing **GLM-4.7-REAP-218B-A32B-FP8**, a **memory-efficient compressed variant** of GLM-4.7-FP8 that maintains near-identical performance while being **40% lighter**.
+
+This model was created using **REAP (Router-weighted Expert Activation Pruning)**, a novel expert pruning method that selectively removes redundant experts while preserving the router's independent control over remaining experts. Key features include:
+
+- **Near-Lossless Performance**: Maintains almost identical accuracy on code generation, agentic coding, and function calling tasks compared to the full 355B model
+- **40% Memory Reduction**: Compressed from 355B to 218B parameters, significantly lowering deployment costs and memory requirements
+- **Preserved Capabilities**: Retains all core functionalities including code generation, agentic workflows, repository-scale understanding, and function calling
+- **Drop-in Compatibility**: Works with vanilla vLLM - no source modifications or custom patches required
+- **Optimized for Real-World Use**: Particularly effective for resource-constrained environments, local deployments, and academic research
+
+**For downstream low-bit quantization, we suggest using the [BF16 variant](https://huggingface.co/cerebras/GLM-4.7-REAP-218B-A32B).**
+
+---
+## 📋 Model Overview
+
+**GLM-4.7-REAP-218B-A32B-FP8** has the following specifications:
+
+- **Base Model**: GLM-4.7-FP8
+- **Compression Method**: REAP (Router-weighted Expert Activation Pruning)
+- **Compression Ratio**: 40% expert pruning
+- **Type**: Sparse Mixture-of-Experts (SMoE) Causal Language Model
+- **Number of Parameters**: 218B total, 32B activated per token
+- **Number of Layers**: 92
+- **Number of Attention Heads (GQA)**: 96 for Q and 8 for KV
+- **Number of Experts**: 96 (uniformly pruned from 160)
+- **Number of Activated Experts**: 8 per token
+- **Context Length**: 202,752 tokens
+- **License**: MIT
+
+---
+
+## 📊 Evaluations
+
+<table>
+  <thead>
+    <tr>
+      <th align="left">Benchmark</th>
+      <th align="center">GLM-4.7-FP8</th>
+      <th align="center"><a href="https://huggingface.co/cerebras/GLM-4.7-REAP-268B-A32B-FP8">GLM-4.7-REAP-268B-A32B-FP8</a></th>
+      <th align="center"><a href="https://huggingface.co/cerebras/GLM-4.7-REAP-218B-A32B-FP8">GLM-4.7-REAP-218B-A32B-FP8</a></th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><strong>Compression</strong></td>
+      <td align="center">—</td>
+      <td align="center">25%</td>
+      <td align="center">40%</td>
+    </tr>
+    <tr>
+      <td colspan="5" align="center"><strong>Coding</strong></td>
+    </tr>
+    <tr>
+      <td><strong>HumanEval</strong></td>
+      <td align="center">95.7</td>
+      <td align="center">97.0</td>
+      <td align="center">95.1</td>
+    </tr>
+    <tr>
+      <td><strong>HumanEval+</strong></td>
+      <td align="center">91.5</td>
+      <td align="center">90.2</td>
+      <td align="center">87.8</td>
+    </tr>
+    <tr>
+      <td><strong>MBPP</strong></td>
+      <td align="center">91.0</td>
+      <td align="center">91.0</td>
+      <td align="center">91.5</td>
+    </tr>
+    <tr>
+      <td><strong>MBPP+</strong></td>
+      <td align="center">78.3</td>
+      <td align="center">78.3</td>
+      <td align="center">77.2</td>
+    </tr>
+</table>
+
+
+🟩 *This checkpoint maintains almost identical performance while being 40% lighter.*
+
+For more details on the evaluation setup, refer to the [REAP arXiv preprint](https://arxiv.org/abs/2510.13999).
+
+---
+
+## 🚀 Deployment
+
+You can deploy the model directly using the **latest vLLM** (v0.11.0), no source modifications or custom patches required.
+
+```bash
+vllm serve cerebras/GLM-4.7-REAP-218B-A32B-FP8 \
+    --tensor-parallel-size 8 \
+    --reasoning-parser glm45 \
+    --tool-call-parser glm47 \
+    --enable-auto-tool-choice \
+    --enable-expert-parallel
+```
+
+If you encounter insufficient memory when running this model, you might need to set a lower value for `--max-num-seqs` flag (e.g. set to 64).
+
+
+## 🧩 Model Creation
+
+This checkpoint was created by applying the **REAP (Router-weighted Expert Activation Pruning)** method uniformly across all Mixture-of-Experts (MoE) blocks of **GLM-4.7-FP8**, with a **40% pruning rate**.
+
+### How REAP Works
+
+REAP selects experts to prune based on a novel **saliency criterion** that considers both:
+- **Router gate values**: How frequently and strongly the router activates each expert
+- **Expert activation norms**: The magnitude of each expert's output contributions
+
+This dual consideration ensures that experts contributing minimally to the layer's output are pruned, while preserving those that play critical roles in the model's computations.
+
+### Key Advantages
+
+- **One-Shot Compression**: No fine-tuning required after pruning - the model is immediately ready for deployment
+- **Preserved Router Control**: Unlike expert merging methods, REAP maintains the router's independent, input-dependent control over remaining experts, avoiding "functional subspace collapse"
+- **Generative Task Superiority**: REAP significantly outperforms expert merging approaches on generative benchmarks (code generation, creative writing, mathematical reasoning) while maintaining competitive performance on discriminative tasks
+
+### Calibration
+
+The model was calibrated using a diverse mixture of domain-specific datasets including:
+- Code generation samples ([evol-codealpaca](https://huggingface.co/datasets/theblackcat102/evol-codealpaca-v1))
+- Function calling examples ([xlam-function-calling](https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k))
+- Agentic multi-turn trajectories ([SWE-smith-trajectories](https://huggingface.co/datasets/SWE-bench/SWE-smith-trajectories))
+
+📚 For more details, refer to the following resources:
+
+- [🧾 arXiv Preprint](https://arxiv.org/abs/2510.13999)
+- [🧾 REAP Blog](https://www.cerebras.ai/blog/reap)
+- [💻 REAP Codebase (GitHub)](https://github.com/CerebrasResearch/reap)
+
+---
+
+## ⚖️ License
+
+This model is derived from
+**[`zai-org/GLM-4.7-FP8`](https://huggingface.co/zai-org/GLM-4.7-FP8)**
+and distributed under the **MIT license**.
+
+---
+
+## 🧾 Citation
+
+If you use this checkpoint, please cite the REAP paper:
+
+```bibtex
+@article{lasby-reap,
+  title={REAP the Experts: Why Pruning Prevails for One-Shot MoE compression},
+  author={Lasby, Mike and Lazarevich, Ivan and Sinnadurai, Nish and Lie, Sean and Ioannou, Yani and Thangarasa, Vithursan},
+  journal={arXiv preprint arXiv:2510.13999},
+  year={2025}
+}
+```

chat_template.jinja

@@ -0,0 +1,86 @@
+[gMASK]<sop>
+{%- if tools -%}
+<|system|>
+# Tools
+
+You may call one or more functions to assist with the user query.
+
+You are provided with function signatures within <tools></tools> XML tags:
+<tools>
+{% for tool in tools %}
+{{ tool | tojson(ensure_ascii=False) }}
+{% endfor %}
+</tools>
+
+For each function call, output the function name and arguments within the following XML format:
+<tool_call>{function-name}<arg_key>{arg-key-1}</arg_key><arg_value>{arg-value-1}</arg_value><arg_key>{arg-key-2}</arg_key><arg_value>{arg-value-2}</arg_value>...</tool_call>{%- endif -%}
+{%- macro visible_text(content) -%}
+    {%- if content is string -%}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping -%}
+        {%- for item in content -%}
+            {%- if item is mapping and item.type == 'text' -%}
+                {{- item.text }}
+            {%- elif item is string -%}
+                {{- item }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{- content }}
+    {%- endif -%}
+{%- endmacro -%}
+{%- set ns = namespace(last_user_index=-1) %}
+{%- for m in messages %}
+    {%- if m.role == 'user' %}
+        {% set ns.last_user_index = loop.index0 -%}
+    {%- endif %}
+{%- endfor %}
+{% for m in messages %}
+{%- if m.role == 'user' -%}<|user|>{{ visible_text(m.content) }}
+{%- elif m.role == 'assistant' -%}
+<|assistant|>
+{%- set reasoning_content = '' %}
+{%- set content = visible_text(m.content) %}
+{%- if m.reasoning_content is string %}
+    {%- set reasoning_content = m.reasoning_content %}
+{%- else %}
+    {%- if '</think>' in content %}
+        {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+        {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+    {%- endif %}
+{%- endif %}
+{%- if ((clear_thinking is defined and not clear_thinking) or loop.index0 > ns.last_user_index) and reasoning_content -%}
+{{ '<think>' + reasoning_content.strip() +  '</think>'}}
+{%- else -%}
+{{ '</think>' }}
+{%- endif -%}
+{%- if content.strip() -%}
+{{ content.strip() }}
+{%- endif -%}
+{% if m.tool_calls %}
+{% for tc in m.tool_calls %}
+{%- if tc.function %}
+    {%- set tc = tc.function %}
+{%- endif %}
+{{- '<tool_call>' + tc.name -}}
+{% set _args = tc.arguments %}{% for k, v in _args.items() %}<arg_key>{{ k }}</arg_key><arg_value>{{ v | tojson(ensure_ascii=False) if v is not string else v }}</arg_value>{% endfor %}</tool_call>{% endfor %}
+{% endif %}
+{%- elif m.role == 'tool' -%}
+{%- if m.content is string -%}
+{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+    {{- '<|observation|>' }}
+{%- endif %}
+{{- '<tool_response>' }}
+{{- m.content }}
+{{- '</tool_response>' }}
+{%- else -%}
+<|observation|>{% for tr in m.content %}
+<tool_response>{{ tr.output if tr.output is defined else tr }}</tool_response>{% endfor -%}
+{% endif -%}
+{%- elif m.role == 'system' -%}
+<|system|>{{ visible_text(m.content) }}
+{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    <|assistant|>{{- '</think>' if (enable_thinking is defined and not enable_thinking) else '<think>' -}}
+{%- endif -%}

config.json

@@ -0,0 +1,927 @@
+{
+  "architectures": [
+    "Glm4MoeForCausalLM"
+  ],
+  "attention_bias": true,
+  "attention_dropout": 0.0,
+  "eos_token_id": [
+    151329,
+    151336,
+    151338
+  ],
+  "first_k_dense_replace": 3,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 12288,
+  "max_position_embeddings": 202752,
+  "model_type": "glm4_moe",
+  "moe_intermediate_size": 1536,
+  "moe_router_dtype": "float32",
+  "n_group": 1,
+  "n_routed_experts": 96,
+  "n_shared_experts": 1,
+  "norm_topk_prob": true,
+  "num_attention_heads": 96,
+  "num_experts_per_tok": 8,
+  "num_hidden_layers": 92,
+  "num_key_value_heads": 8,
+  "num_nextn_predict_layers": 0,
+  "pad_token_id": 151329,
+  "partial_rotary_factor": 0.5,
+  "quantization_config": {
+    "config_groups": {
+      "group_0": {
+        "input_activations": {
+          "actorder": null,
+          "block_structure": null,
+          "dynamic": true,
+          "group_size": null,
+          "num_bits": 8,
+          "observer": null,
+          "observer_kwargs": {},
+          "strategy": "token",
+          "symmetric": true,
+          "type": "float"
+        },
+        "output_activations": null,
+        "targets": [
+          "Linear"
+        ],
+        "weights": {
+          "actorder": null,
+          "block_structure": null,
+          "dynamic": false,
+          "group_size": null,
+          "num_bits": 8,
+          "observer": "minmax",
+          "observer_kwargs": {},
+          "strategy": "channel",
+          "symmetric": true,
+          "type": "float"
+        }
+      }
+    },
+    "format": "float-quantized",
+    "global_compression_ratio": null,
+    "ignore": [
+      "model.layers.46.post_attention_layernorm",
+      "model.layers.16.mlp.gate",
+      "model.layers.36.self_attn.k_proj.bias",
+      "model.layers.88.self_attn.k_norm",
+      "model.layers.17.self_attn.q_proj.bias",
+      "model.layers.26.self_attn.v_proj.bias",
+      "model.layers.5.mlp.gate",
+      "model.layers.59.input_layernorm",
+      "model.layers.24.self_attn.k_proj.bias",
+      "model.layers.40.self_attn.v_proj.bias",
+      "model.layers.55.mlp.gate.e_score_correction_bias",
+      "model.layers.72.self_attn.v_proj.bias",
+      "model.layers.62.self_attn.q_norm",
+      "model.layers.24.self_attn.q_norm",
+      "model.layers.7.self_attn.q_norm",
+      "model.layers.78.self_attn.k_proj.bias",
+      "model.layers.75.self_attn.q_norm",
+      "model.layers.45.post_attention_layernorm",
+      "model.layers.50.self_attn.k_norm",
+      "model.layers.22.self_attn.v_proj.bias",
+      "model.layers.15.post_attention_layernorm",
+      "model.layers.24.input_layernorm",
+      "model.layers.24.mlp.gate.e_score_correction_bias",
+      "model.layers.34.self_attn.q_norm",
+      "model.layers.21.post_attention_layernorm",
+      "model.layers.15.self_attn.k_norm",
+      "model.layers.82.self_attn.v_proj.bias",
+      "model.layers.60.self_attn.q_proj.bias",
+      "model.layers.86.self_attn.k_norm",
+      "model.layers.91.input_layernorm",
+      "model.layers.67.self_attn.q_proj.bias",
+      "model.layers.27.self_attn.k_norm",
+      "model.layers.76.self_attn.q_proj.bias",
+      "model.layers.53.self_attn.v_proj.bias",
+      "model.layers.8.self_attn.k_proj.bias",
+      "model.layers.81.self_attn.k_proj.bias",
+      "model.layers.20.self_attn.k_norm",
+      "model.layers.24.self_attn.v_proj.bias",
+      "model.layers.81.input_layernorm",
+      "model.layers.23.mlp.gate",
+      "model.layers.42.input_layernorm",
+      "model.layers.48.mlp.gate.e_score_correction_bias",
+      "model.layers.91.self_attn.q_proj.bias",
+      "model.layers.32.self_attn.q_norm",
+      "model.layers.23.input_layernorm",
+      "model.layers.49.mlp.gate",
+      "model.layers.54.self_attn.v_proj.bias",
+      "model.layers.62.self_attn.k_proj.bias",
+      "model.layers.90.self_attn.k_norm",
+      "model.layers.72.self_attn.k_proj.bias",
+      "model.layers.39.self_attn.q_norm",
+      "model.layers.28.self_attn.v_proj.bias",
+      "model.layers.63.self_attn.k_proj.bias",
+      "model.layers.17.self_attn.v_proj.bias",
+      "model.layers.72.post_attention_layernorm",
+      "model.layers.26.input_layernorm",
+      "model.layers.77.mlp.gate.e_score_correction_bias",
+      "model.layers.33.mlp.gate.e_score_correction_bias",
+      "model.layers.54.self_attn.k_proj.bias",
+      "model.layers.72.self_attn.k_norm",
+      "model.layers.87.self_attn.k_proj.bias",
+      "model.layers.22.input_layernorm",
+      "model.layers.90.mlp.gate",
+      "model.layers.38.self_attn.k_proj.bias",
+      "model.layers.20.input_layernorm",
+      "model.layers.4.self_attn.k_proj.bias",
+      "model.layers.43.mlp.gate",
+      "model.layers.37.post_attention_layernorm",
+      "model.layers.14.mlp.gate",
+      "model.layers.79.post_attention_layernorm",
+      "model.layers.84.self_attn.k_norm",
+      "model.layers.71.self_attn.q_norm",
+      "model.layers.56.mlp.gate.e_score_correction_bias",
+      "model.layers.6.self_attn.k_proj.bias",
+      "model.layers.70.self_attn.k_proj.bias",
+      "model.layers.54.post_attention_layernorm",
+      "model.layers.55.self_attn.q_norm",
+      "model.layers.4.mlp.gate",
+      "model.layers.81.mlp.gate",
+      "model.layers.7.mlp.gate",
+      "model.layers.8.self_attn.q_norm",
+      "model.layers.57.self_attn.v_proj.bias",
+      "model.layers.65.self_attn.q_norm",
+      "model.layers.88.post_attention_layernorm",
+      "model.layers.19.input_layernorm",
+      "model.layers.6.self_attn.v_proj.bias",
+      "model.layers.4.input_layernorm",
+      "model.layers.71.self_attn.k_norm",
+      "model.layers.32.self_attn.k_norm",
+      "model.layers.22.post_attention_layernorm",
+      "model.layers.6.self_attn.q_proj.bias",
+      "model.layers.61.self_attn.q_norm",
+      "model.layers.3.input_layernorm",
+      "model.layers.13.input_layernorm",
+      "model.layers.74.self_attn.k_norm",
+      "model.layers.59.self_attn.v_proj.bias",
+      "model.layers.60.self_attn.v_proj.bias",
+      "model.layers.35.self_attn.q_proj.bias",
+      "model.layers.18.post_attention_layernorm",
+      "model.layers.44.self_attn.k_proj.bias",
+      "model.layers.64.mlp.gate",
+      "model.layers.61.post_attention_layernorm",
+      "model.layers.46.self_attn.k_norm",
+      "model.layers.19.post_attention_layernorm",
+      "model.layers.66.input_layernorm",
+      "model.layers.40.mlp.gate.e_score_correction_bias",
+      "model.layers.18.self_attn.k_norm",
+      "model.layers.30.input_layernorm",
+      "model.layers.56.self_attn.v_proj.bias",
+      "model.layers.49.self_attn.v_proj.bias",
+      "model.layers.3.self_attn.v_proj.bias",
+      "model.layers.72.mlp.gate.e_score_correction_bias",
+      "model.layers.79.self_attn.k_norm",
+      "model.layers.92.self_attn.k_norm",
+      "model.layers.79.self_attn.v_proj.bias",
+      "model.layers.32.mlp.gate.e_score_correction_bias",
+      "model.layers.26.mlp.gate.e_score_correction_bias",
+      "model.embed_tokens",
+      "model.layers.18.mlp.gate",
+      "model.layers.62.self_attn.q_proj.bias",
+      "model.layers.44.post_attention_layernorm",
+      "model.layers.55.post_attention_layernorm",
+      "model.layers.18.input_layernorm",
+      "model.layers.46.mlp.gate.e_score_correction_bias",
+      "model.layers.75.mlp.gate",
+      "model.layers.63.self_attn.q_norm",
+      "model.layers.36.mlp.gate.e_score_correction_bias",
+      "model.layers.66.self_attn.q_norm",
+      "model.layers.45.mlp.gate",
+      "model.layers.24.post_attention_layernorm",
+      "model.layers.45.self_attn.k_norm",
+      "model.layers.0.self_attn.q_proj.bias",
+      "model.layers.42.mlp.gate.e_score_correction_bias",
+      "model.layers.64.self_attn.q_norm",
+      "model.layers.8.input_layernorm",
+      "model.layers.26.self_attn.q_proj.bias",
+      "model.layers.32.post_attention_layernorm",
+      "model.layers.40.self_attn.k_norm",
+      "model.layers.91.self_attn.q_norm",
+      "model.layers.69.post_attention_layernorm",
+      "model.layers.27.self_attn.v_proj.bias",
+      "model.layers.9.self_attn.k_norm",
+      "model.layers.62.self_attn.k_norm",
+      "model.layers.92.input_layernorm",
+      "model.layers.14.self_attn.q_proj.bias",
+      "model.layers.84.input_layernorm",
+      "model.layers.88.mlp.gate",
+      "model.layers.7.self_attn.v_proj.bias",
+      "model.layers.14.self_attn.q_norm",
+      "model.layers.17.self_attn.q_norm",
+      "model.layers.85.mlp.gate",
+      "model.layers.63.mlp.gate",
+      "model.layers.66.self_attn.q_proj.bias",
+      "model.layers.76.self_attn.k_proj.bias",
+      "model.layers.18.self_attn.q_norm",
+      "model.layers.77.post_attention_layernorm",
+      "model.layers.55.self_attn.k_norm",
+      "model.layers.37.self_attn.q_norm",
+      "model.layers.91.post_attention_layernorm",
+      "model.layers.53.self_attn.q_norm",
+      "model.layers.76.input_layernorm",
+      "model.layers.1.self_attn.v_proj.bias",
+      "model.layers.38.self_attn.v_proj.bias",
+      "model.layers.14.mlp.gate.e_score_correction_bias",
+      "model.layers.9.self_attn.v_proj.bias",
+      "model.layers.75.post_attention_layernorm",
+      "model.layers.9.self_attn.q_norm",
+      "model.layers.57.mlp.gate.e_score_correction_bias",
+      "model.layers.82.self_attn.k_norm",
+      "model.layers.7.post_attention_layernorm",
+      "model.layers.43.self_attn.q_norm",
+      "model.layers.91.self_attn.k_proj.bias",
+      "model.layers.29.self_attn.k_norm",
+      "model.layers.11.self_attn.k_proj.bias",
+      "model.layers.6.mlp.gate",
+      "model.layers.7.self_attn.q_proj.bias",
+      "model.layers.63.post_attention_layernorm",
+      "model.layers.11.self_attn.q_proj.bias",
+      "model.layers.25.input_layernorm",
+      "model.layers.21.input_layernorm",
+      "model.layers.61.input_layernorm",
+      "model.layers.37.mlp.gate.e_score_correction_bias",
+      "model.layers.65.mlp.gate",
+      "model.layers.20.self_attn.q_norm",
+      "model.layers.60.post_attention_layernorm",
+      "model.layers.35.self_attn.k_proj.bias",
+      "model.layers.70.mlp.gate.e_score_correction_bias",
+      "model.layers.85.self_attn.k_norm",
+      "model.layers.58.input_layernorm",
+      "model.layers.66.self_attn.v_proj.bias",
+      "model.layers.80.self_attn.q_norm",
+      "model.layers.59.self_attn.k_norm",
+      "model.layers.47.self_attn.k_proj.bias",
+      "model.layers.46.mlp.gate",
+      "model.layers.27.self_attn.k_proj.bias",
+      "model.layers.71.self_attn.k_proj.bias",
+      "model.layers.76.self_attn.q_norm",
+      "model.layers.16.input_layernorm",
+      "model.layers.56.self_attn.k_proj.bias",
+      "model.layers.30.self_attn.q_proj.bias",
+      "model.layers.89.self_attn.q_norm",
+      "model.layers.80.self_attn.k_norm",
+      "model.layers.40.mlp.gate",
+      "model.layers.10.mlp.gate.e_score_correction_bias",
+      "model.layers.89.self_attn.k_proj.bias",
+      "model.layers.41.mlp.gate.e_score_correction_bias",
+      "model.layers.79.self_attn.q_norm",
+      "model.layers.49.self_attn.k_proj.bias",
+      "model.layers.43.self_attn.k_norm",
+      "model.layers.21.self_attn.q_proj.bias",
+      "model.layers.69.self_attn.v_proj.bias",
+      "model.layers.70.self_attn.k_norm",
+      "model.layers.65.self_attn.k_norm",
+      "model.layers.59.mlp.gate.e_score_correction_bias",
+      "model.layers.92.post_attention_layernorm",
+      "model.layers.81.self_attn.v_proj.bias",
+      "model.layers.51.input_layernorm",
+      "model.layers.3.post_attention_layernorm",
+      "model.layers.28.post_attention_layernorm",
+      "model.layers.1.self_attn.q_proj.bias",
+      "model.layers.0.self_attn.v_proj.bias",
+      "model.layers.10.self_attn.q_norm",
+      "model.layers.34.mlp.gate.e_score_correction_bias",
+      "model.layers.52.self_attn.v_proj.bias",
+      "model.layers.1.self_attn.q_norm",
+      "model.layers.89.self_attn.k_norm",
+      "model.layers.27.mlp.gate.e_score_correction_bias",
+      "model.layers.53.self_attn.k_norm",
+      "model.layers.61.self_attn.v_proj.bias",
+      "model.layers.78.input_layernorm",
+      "model.layers.2.self_attn.v_proj.bias",
+      "model.layers.76.post_attention_layernorm",
+      "model.layers.36.self_attn.q_norm",
+      "model.layers.68.self_attn.q_proj.bias",
+      "model.layers.33.self_attn.k_proj.bias",
+      "model.layers.78.mlp.gate",
+      "model.layers.36.mlp.gate",
+      "model.layers.44.mlp.gate",
+      "model.layers.54.mlp.gate.e_score_correction_bias",
+      "model.layers.73.mlp.gate.e_score_correction_bias",
+      "model.layers.41.self_attn.k_proj.bias",
+      "model.layers.63.self_attn.k_norm",
+      "model.layers.29.self_attn.q_proj.bias",
+      "model.layers.22.self_attn.q_norm",
+      "model.layers.74.self_attn.q_proj.bias",
+      "model.layers.47.post_attention_layernorm",
+      "model.layers.64.input_layernorm",
+      "model.layers.77.self_attn.k_norm",
+      "model.layers.27.mlp.gate",
+      "model.layers.33.self_attn.q_proj.bias",
+      "model.layers.77.self_attn.v_proj.bias",
+      "model.layers.87.input_layernorm",
+      "model.layers.5.input_layernorm",
+      "model.layers.14.self_attn.k_proj.bias",
+      "model.layers.32.input_layernorm",
+      "model.layers.35.input_layernorm",
+      "model.layers.44.self_attn.q_norm",
+      "model.layers.51.self_attn.k_norm",
+      "model.layers.80.self_attn.v_proj.bias",
+      "model.layers.63.self_attn.v_proj.bias",
+      "model.layers.4.self_attn.k_norm",
+      "model.layers.89.self_attn.q_proj.bias",
+      "model.layers.61.self_attn.q_proj.bias",
+      "model.layers.9.post_attention_layernorm",
+      "model.layers.29.self_attn.v_proj.bias",
+      "model.layers.92.hnorm",
+      "model.layers.53.self_attn.q_proj.bias",
+      "model.layers.73.self_attn.k_norm",
+      "model.layers.91.mlp.gate.e_score_correction_bias",
+      "model.layers.75.self_attn.q_proj.bias",
+      "model.layers.47.self_attn.q_norm",
+      "model.layers.77.mlp.gate",
+      "model.layers.21.mlp.gate.e_score_correction_bias",
+      "model.layers.90.self_attn.v_proj.bias",
+      "model.layers.44.mlp.gate.e_score_correction_bias",
+      "model.layers.37.self_attn.k_proj.bias",
+      "model.layers.92.mlp.gate",
+      "model.layers.32.mlp.gate",
+      "model.layers.54.input_layernorm",
+      "model.layers.13.self_attn.q_proj.bias",
+      "model.layers.0.post_attention_layernorm",
+      "model.layers.84.mlp.gate.e_score_correction_bias",
+      "model.layers.31.mlp.gate.e_score_correction_bias",
+      "model.layers.57.self_attn.q_norm",
+      "model.layers.83.post_attention_layernorm",
+      "model.layers.61.mlp.gate.e_score_correction_bias",
+      "model.layers.49.input_layernorm",
+      "model.layers.50.mlp.gate.e_score_correction_bias",
+      "model.layers.48.self_attn.k_norm",
+      "model.layers.18.self_attn.v_proj.bias",
+      "model.layers.54.self_attn.q_proj.bias",
+      "model.layers.8.mlp.gate.e_score_correction_bias",
+      "model.layers.14.self_attn.v_proj.bias",
+      "model.layers.5.self_attn.q_norm",
+      "model.layers.90.mlp.gate.e_score_correction_bias",
+      "model.layers.83.mlp.gate.e_score_correction_bias",
+      "model.layers.69.input_layernorm",
+      "model.layers.82.self_attn.k_proj.bias",
+      "model.layers.86.self_attn.q_norm",
+      "model.layers.65.input_layernorm",
+      "model.layers.16.mlp.gate.e_score_correction_bias",
+      "model.layers.22.mlp.gate.e_score_correction_bias",
+      "model.layers.80.mlp.gate.e_score_correction_bias",
+      "model.layers.8.self_attn.q_proj.bias",
+      "model.layers.23.self_attn.q_norm",
+      "model.layers.11.self_attn.k_norm",
+      "model.layers.53.mlp.gate.e_score_correction_bias",
+      "model.layers.65.self_attn.v_proj.bias",
+      "model.layers.47.mlp.gate",
+      "model.layers.70.mlp.gate",
+      "model.layers.12.mlp.gate.e_score_correction_bias",
+      "model.layers.38.mlp.gate.e_score_correction_bias",
+      "model.layers.46.self_attn.k_proj.bias",
+      "model.layers.64.self_attn.k_norm",
+      "model.layers.89.mlp.gate",
+      "model.layers.44.self_attn.v_proj.bias",
+      "model.layers.33.self_attn.k_norm",
+      "model.layers.8.self_attn.k_norm",
+      "model.layers.1.self_attn.k_proj.bias",
+      "model.layers.51.self_attn.q_proj.bias",
+      "lm_head",
+      "model.layers.42.self_attn.k_norm",
+      "model.layers.60.input_layernorm",
+      "model.layers.59.self_attn.q_proj.bias",
+      "model.layers.52.self_attn.q_norm",
+      "model.layers.84.self_attn.q_proj.bias",
+      "model.layers.51.mlp.gate",
+      "model.layers.81.self_attn.k_norm",
+      "model.layers.1.input_layernorm",
+      "model.layers.33.self_attn.q_norm",
+      "model.layers.78.mlp.gate.e_score_correction_bias",
+      "model.layers.17.post_attention_layernorm",
+      "model.layers.50.self_attn.k_proj.bias",
+      "model.layers.73.input_layernorm",
+      "model.layers.16.self_attn.q_proj.bias",
+      "model.layers.70.self_attn.q_proj.bias",
+      "model.layers.26.post_attention_layernorm",
+      "model.layers.27.input_layernorm",
+      "model.layers.86.self_attn.q_proj.bias",
+      "model.layers.33.self_attn.v_proj.bias",
+      "model.layers.50.mlp.gate",
+      "model.layers.64.post_attention_layernorm",
+      "model.layers.73.self_attn.q_proj.bias",
+      "model.layers.72.input_layernorm",
+      "model.layers.66.self_attn.k_norm",
+      "model.layers.50.self_attn.v_proj.bias",
+      "model.layers.24.self_attn.k_norm",
+      "model.layers.69.self_attn.k_norm",
+      "model.layers.75.self_attn.k_norm",
+      "model.layers.36.input_layernorm",
+      "model.layers.40.self_attn.q_norm",
+      "model.layers.33.input_layernorm",
+      "model.layers.92.enorm",
+      "model.layers.23.self_attn.k_norm",
+      "model.layers.25.mlp.gate.e_score_correction_bias",
+      "model.layers.74.self_attn.q_norm",
+      "model.layers.13.self_attn.q_norm",
+      "model.layers.75.self_attn.v_proj.bias",
+      "model.layers.30.self_attn.v_proj.bias",
+      "model.layers.42.self_attn.v_proj.bias",
+      "model.layers.29.input_layernorm",
+      "model.layers.91.self_attn.k_norm",
+      "model.layers.92.self_attn.k_proj.bias",
+      "model.layers.12.self_attn.v_proj.bias",
+      "model.layers.19.self_attn.v_proj.bias",
+      "model.layers.70.self_attn.q_norm",
+      "model.layers.75.self_attn.k_proj.bias",
+      "model.layers.49.post_attention_layernorm",
+      "model.layers.50.self_attn.q_proj.bias",
+      "model.layers.84.post_attention_layernorm",
+      "model.layers.7.mlp.gate.e_score_correction_bias",
+      "model.layers.59.post_attention_layernorm",
+      "model.layers.86.input_layernorm",
+      "model.layers.22.self_attn.k_proj.bias",
+      "model.layers.31.self_attn.q_proj.bias",
+      "model.layers.30.post_attention_layernorm",
+      "model.layers.79.input_layernorm",
+      "model.layers.3.self_attn.k_proj.bias",
+      "model.layers.66.self_attn.k_proj.bias",
+      "model.layers.81.mlp.gate.e_score_correction_bias",
+      "model.layers.88.mlp.gate.e_score_correction_bias",
+      "model.layers.29.self_attn.k_proj.bias",
+      "model.layers.4.self_attn.q_proj.bias",
+      "model.layers.79.self_attn.q_proj.bias",
+      "model.layers.35.post_attention_layernorm",
+      "model.layers.63.input_layernorm",
+      "model.layers.12.self_attn.q_proj.bias",
+      "model.layers.0.self_attn.q_norm",
+      "model.layers.13.post_attention_layernorm",
+      "model.layers.58.self_attn.q_proj.bias",
+      "model.layers.90.self_attn.k_proj.bias",
+      "model.layers.45.self_attn.v_proj.bias",
+      "model.layers.86.self_attn.k_proj.bias",
+      "model.layers.38.input_layernorm",
+      "model.layers.70.post_attention_layernorm",
+      "model.layers.69.self_attn.k_proj.bias",
+      "model.layers.88.input_layernorm",
+      "model.layers.75.input_layernorm",
+      "model.layers.76.mlp.gate.e_score_correction_bias",
+      "model.layers.76.mlp.gate",
+      "model.layers.8.post_attention_layernorm",
+      "model.layers.30.mlp.gate",
+      "model.layers.38.self_attn.q_proj.bias",
+      "model.layers.19.self_attn.q_norm",
+      "model.layers.76.self_attn.k_norm",
+      "model.layers.79.mlp.gate",
+      "model.layers.23.self_attn.v_proj.bias",
+      "model.layers.27.self_attn.q_proj.bias",
+      "model.layers.57.post_attention_layernorm",
+      "model.layers.56.mlp.gate",
+      "model.layers.45.mlp.gate.e_score_correction_bias",
+      "model.layers.76.self_attn.v_proj.bias",
+      "model.layers.26.self_attn.q_norm",
+      "model.layers.17.self_attn.k_norm",
+      "model.layers.19.mlp.gate.e_score_correction_bias",
+      "model.layers.6.post_attention_layernorm",
+      "model.layers.12.post_attention_layernorm",
+      "model.layers.71.input_layernorm",
+      "model.layers.5.post_attention_layernorm",
+      "model.layers.25.self_attn.k_norm",
+      "model.layers.37.mlp.gate",
+      "model.layers.20.self_attn.k_proj.bias",
+      "model.layers.20.self_attn.v_proj.bias",
+      "model.layers.52.input_layernorm",
+      "model.layers.43.post_attention_layernorm",
+      "model.layers.51.post_attention_layernorm",
+      "model.layers.12.mlp.gate",
+      "model.layers.47.input_layernorm",
+      "model.layers.49.self_attn.q_norm",
+      "model.layers.78.self_attn.q_proj.bias",
+      "model.layers.53.input_layernorm",
+      "model.layers.11.mlp.gate",
+      "model.layers.88.self_attn.q_norm",
+      "model.layers.36.self_attn.q_proj.bias",
+      "model.layers.58.self_attn.q_norm",
+      "model.layers.77.input_layernorm",
+      "model.layers.41.post_attention_layernorm",
+      "model.layers.74.self_attn.v_proj.bias",
+      "model.layers.2.self_attn.q_norm",
+      "model.layers.5.self_attn.k_proj.bias",
+      "model.layers.92.self_attn.v_proj.bias",
+      "model.layers.58.self_attn.k_norm",
+      "model.layers.5.mlp.gate.e_score_correction_bias",
+      "model.layers.71.mlp.gate.e_score_correction_bias",
+      "model.layers.87.mlp.gate.e_score_correction_bias",
+      "model.layers.46.input_layernorm",
+      "model.layers.66.mlp.gate.e_score_correction_bias",
+      "model.layers.74.post_attention_layernorm",
+      "model.layers.80.self_attn.k_proj.bias",
+      "model.layers.25.self_attn.v_proj.bias",
+      "model.layers.14.self_attn.k_norm",
+      "model.layers.3.mlp.gate.e_score_correction_bias",
+      "model.layers.57.self_attn.q_proj.bias",
+      "model.layers.19.self_attn.q_proj.bias",
+      "model.layers.68.mlp.gate",
+      "model.layers.28.input_layernorm",
+      "model.layers.41.self_attn.q_norm",
+      "model.layers.4.mlp.gate.e_score_correction_bias",
+      "model.layers.37.input_layernorm",
+      "model.layers.91.mlp.gate",
+      "model.layers.35.mlp.gate.e_score_correction_bias",
+      "model.layers.39.mlp.gate",
+      "model.layers.30.self_attn.q_norm",
+      "model.layers.87.post_attention_layernorm",
+      "model.layers.52.self_attn.k_proj.bias",
+      "model.layers.11.self_attn.q_norm",
+      "model.layers.59.self_attn.k_proj.bias",
+      "model.layers.86.mlp.gate.e_score_correction_bias",
+      "model.layers.85.post_attention_layernorm",
+      "model.layers.83.mlp.gate",
+      "model.layers.70.input_layernorm",
+      "model.layers.69.self_attn.q_proj.bias",
+      "model.layers.92.eh_proj",
+      "model.layers.83.self_attn.k_norm",
+      "model.layers.85.self_attn.v_proj.bias",
+      "model.layers.34.input_layernorm",
+      "model.layers.70.self_attn.v_proj.bias",
+      "model.layers.34.self_attn.k_norm",
+      "model.layers.67.mlp.gate",
+      "model.layers.85.mlp.gate.e_score_correction_bias",
+      "model.layers.11.self_attn.v_proj.bias",
+      "model.layers.21.mlp.gate",
+      "model.layers.0.self_attn.k_proj.bias",
+      "model.layers.20.mlp.gate.e_score_correction_bias",
+      "model.layers.55.self_attn.k_proj.bias",
+      "model.layers.73.self_attn.k_proj.bias",
+      "model.layers.29.self_attn.q_norm",
+      "model.layers.41.self_attn.k_norm",
+      "model.layers.65.post_attention_layernorm",
+      "model.layers.46.self_attn.q_proj.bias",
+      "model.layers.83.self_attn.v_proj.bias",
+      "model.layers.73.self_attn.v_proj.bias",
+      "model.layers.82.post_attention_layernorm",
+      "model.layers.11.mlp.gate.e_score_correction_bias",
+      "model.layers.87.mlp.gate",
+      "model.layers.36.self_attn.v_proj.bias",
+      "model.layers.40.self_attn.k_proj.bias",
+      "model.layers.17.self_attn.k_proj.bias",
+      "model.layers.60.mlp.gate.e_score_correction_bias",
+      "model.layers.39.post_attention_layernorm",
+      "model.layers.61.self_attn.k_proj.bias",
+      "model.layers.91.self_attn.v_proj.bias",
+      "model.layers.49.mlp.gate.e_score_correction_bias",
+      "model.layers.19.self_attn.k_proj.bias",
+      "model.layers.32.self_attn.v_proj.bias",
+      "model.layers.77.self_attn.q_proj.bias",
+      "model.layers.33.post_attention_layernorm",
+      "model.layers.68.self_attn.k_norm",
+      "model.layers.73.self_attn.q_norm",
+      "model.layers.37.self_attn.v_proj.bias",
+      "model.layers.86.self_attn.v_proj.bias",
+      "model.layers.52.self_attn.k_norm",
+      "model.layers.6.self_attn.q_norm",
+      "model.layers.86.post_attention_layernorm",
+      "model.layers.47.self_attn.v_proj.bias",
+      "model.layers.15.self_attn.k_proj.bias",
+      "model.layers.21.self_attn.k_norm",
+      "model.layers.48.post_attention_layernorm",
+      "model.layers.82.mlp.gate",
+      "model.layers.88.self_attn.v_proj.bias",
+      "model.layers.85.self_attn.k_proj.bias",
+      "model.layers.74.mlp.gate.e_score_correction_bias",
+      "model.layers.43.mlp.gate.e_score_correction_bias",
+      "model.layers.74.mlp.gate",
+      "model.layers.84.self_attn.k_proj.bias",
+      "model.layers.29.post_attention_layernorm",
+      "model.layers.38.self_attn.k_norm",
+      "model.layers.7.self_attn.k_norm",
+      "model.layers.28.self_attn.k_proj.bias",
+      "model.layers.25.post_attention_layernorm",
+      "model.layers.35.self_attn.q_norm",
+      "model.layers.34.mlp.gate",
+      "model.layers.25.mlp.gate",
+      "model.layers.31.mlp.gate",
+      "model.layers.55.mlp.gate",
+      "model.layers.49.self_attn.q_proj.bias",
+      "model.layers.83.self_attn.q_proj.bias",
+      "model.layers.88.self_attn.k_proj.bias",
+      "model.layers.50.input_layernorm",
+      "model.layers.67.input_layernorm",
+      "model.layers.78.post_attention_layernorm",
+      "model.layers.13.mlp.gate.e_score_correction_bias",
+      "model.layers.60.self_attn.k_norm",
+      "model.layers.15.self_attn.q_norm",
+      "model.layers.72.mlp.gate",
+      "model.layers.1.self_attn.k_norm",
+      "model.layers.92.self_attn.q_norm",
+      "model.layers.20.post_attention_layernorm",
+      "model.layers.42.self_attn.k_proj.bias",
+      "model.layers.80.mlp.gate",
+      "model.layers.29.mlp.gate.e_score_correction_bias",
+      "model.layers.4.self_attn.q_norm",
+      "model.layers.60.self_attn.q_norm",
+      "model.layers.52.post_attention_layernorm",
+      "model.layers.48.self_attn.k_proj.bias",
+      "model.layers.37.self_attn.q_proj.bias",
+      "model.layers.65.self_attn.q_proj.bias",
+      "model.layers.31.post_attention_layernorm",
+      "model.layers.4.self_attn.v_proj.bias",
+      "model.layers.28.self_attn.k_norm",
+      "model.layers.38.self_attn.q_norm",
+      "model.layers.3.mlp.gate",
+      "model.layers.89.mlp.gate.e_score_correction_bias",
+      "model.layers.81.self_attn.q_proj.bias",
+      "model.layers.15.mlp.gate",
+      "model.layers.55.self_attn.v_proj.bias",
+      "model.layers.69.self_attn.q_norm",
+      "model.layers.77.self_attn.q_norm",
+      "model.layers.5.self_attn.k_norm",
+      "model.layers.9.input_layernorm",
+      "model.layers.62.mlp.gate.e_score_correction_bias",
+      "model.layers.23.post_attention_layernorm",
+      "model.layers.83.self_attn.k_proj.bias",
+      "model.layers.34.self_attn.q_proj.bias",
+      "model.layers.82.mlp.gate.e_score_correction_bias",
+      "model.layers.89.self_attn.v_proj.bias",
+      "model.layers.10.post_attention_layernorm",
+      "model.layers.42.self_attn.q_proj.bias",
+      "model.layers.2.input_layernorm",
+      "model.layers.66.mlp.gate",
+      "model.layers.87.self_attn.q_proj.bias",
+      "model.layers.46.self_attn.v_proj.bias",
+      "model.layers.17.mlp.gate.e_score_correction_bias",
+      "model.layers.16.self_attn.k_proj.bias",
+      "model.layers.2.post_attention_layernorm",
+      "model.layers.2.self_attn.k_norm",
+      "model.layers.90.self_attn.q_norm",
+      "model.layers.42.self_attn.q_norm",
+      "model.layers.37.self_attn.k_norm",
+      "model.layers.54.self_attn.k_norm",
+      "model.layers.68.input_layernorm",
+      "model.layers.3.self_attn.k_norm",
+      "model.layers.31.self_attn.k_proj.bias",
+      "model.layers.79.self_attn.k_proj.bias",
+      "model.layers.33.mlp.gate",
+      "model.layers.87.self_attn.q_norm",
+      "model.layers.68.post_attention_layernorm",
+      "model.layers.87.self_attn.k_norm",
+      "model.layers.0.self_attn.k_norm",
+      "model.layers.4.post_attention_layernorm",
+      "model.layers.26.self_attn.k_proj.bias",
+      "model.layers.39.self_attn.q_proj.bias",
+      "model.layers.71.mlp.gate",
+      "model.layers.43.self_attn.k_proj.bias",
+      "model.layers.50.post_attention_layernorm",
+      "model.layers.41.self_attn.v_proj.bias",
+      "model.layers.9.mlp.gate",
+      "model.layers.89.input_layernorm",
+      "model.layers.73.mlp.gate",
+      "model.layers.7.self_attn.k_proj.bias",
+      "model.layers.10.self_attn.v_proj.bias",
+      "model.layers.3.self_attn.q_norm",
+      "model.layers.12.self_attn.k_norm",
+      "model.layers.40.input_layernorm",
+      "model.layers.21.self_attn.v_proj.bias",
+      "model.layers.22.self_attn.k_norm",
+      "model.layers.16.post_attention_layernorm",
+      "model.layers.81.post_attention_layernorm",
+      "model.layers.56.self_attn.q_norm",
+      "model.layers.10.self_attn.k_norm",
+      "model.layers.35.mlp.gate",
+      "model.layers.65.self_attn.k_proj.bias",
+      "model.layers.41.input_layernorm",
+      "model.layers.80.input_layernorm",
+      "model.layers.31.input_layernorm",
+      "model.layers.30.self_attn.k_proj.bias",
+      "model.layers.49.self_attn.k_norm",
+      "model.layers.56.self_attn.k_norm",
+      "model.layers.92.embed_tokens",
+      "model.layers.42.mlp.gate",
+      "model.layers.84.self_attn.q_norm",
+      "model.layers.13.self_attn.v_proj.bias",
+      "model.layers.21.self_attn.k_proj.bias",
+      "model.layers.50.self_attn.q_norm",
+      "model.layers.44.self_attn.k_norm",
+      "model.layers.10.self_attn.k_proj.bias",
+      "model.layers.80.post_attention_layernorm",
+      "model.layers.12.input_layernorm",
+      "model.layers.57.self_attn.k_proj.bias",
+      "model.layers.27.post_attention_layernorm",
+      "model.layers.73.post_attention_layernorm",
+      "model.layers.90.input_layernorm",
+      "model.layers.68.self_attn.v_proj.bias",
+      "model.layers.19.mlp.gate",
+      "model.layers.9.self_attn.k_proj.bias",
+      "model.layers.54.mlp.gate",
+      "model.layers.78.self_attn.k_norm",
+      "model.layers.32.self_attn.q_proj.bias",
+      "model.layers.15.self_attn.q_proj.bias",
+      "model.layers.25.self_attn.q_norm",
+      "model.layers.23.self_attn.k_proj.bias",
+      "model.layers.57.self_attn.k_norm",
+      "model.layers.12.self_attn.q_norm",
+      "model.layers.56.self_attn.q_proj.bias",
+      "model.layers.48.input_layernorm",
+      "model.layers.44.self_attn.q_proj.bias",
+      "model.layers.45.self_attn.q_norm",
+      "model.layers.83.self_attn.q_norm",
+      "model.layers.39.self_attn.k_proj.bias",
+      "model.layers.40.self_attn.q_proj.bias",
+      "model.layers.84.self_attn.v_proj.bias",
+      "model.layers.48.mlp.gate",
+      "model.layers.15.mlp.gate.e_score_correction_bias",
+      "model.layers.67.self_attn.k_proj.bias",
+      "model.layers.62.mlp.gate",
+      "model.layers.28.mlp.gate.e_score_correction_bias",
+      "model.layers.18.self_attn.q_proj.bias",
+      "model.layers.9.self_attn.q_proj.bias",
+      "model.layers.26.self_attn.k_norm",
+      "model.layers.65.mlp.gate.e_score_correction_bias",
+      "model.layers.22.mlp.gate",
+      "model.layers.68.self_attn.k_proj.bias",
+      "model.layers.32.self_attn.k_proj.bias",
+      "model.layers.56.post_attention_layernorm",
+      "model.layers.54.self_attn.q_norm",
+      "model.layers.59.self_attn.q_norm",
+      "model.layers.71.post_attention_layernorm",
+      "model.layers.1.post_attention_layernorm",
+      "model.layers.64.self_attn.q_proj.bias",
+      "model.layers.68.mlp.gate.e_score_correction_bias",
+      "model.layers.41.mlp.gate",
+      "model.layers.22.self_attn.q_proj.bias",
+      "model.layers.31.self_attn.v_proj.bias",
+      "model.layers.74.input_layernorm",
+      "model.layers.27.self_attn.q_norm",
+      "model.layers.34.self_attn.v_proj.bias",
+      "model.layers.45.input_layernorm",
+      "model.layers.39.self_attn.v_proj.bias",
+      "model.layers.16.self_attn.k_norm",
+      "model.layers.62.input_layernorm",
+      "model.layers.24.self_attn.q_proj.bias",
+      "model.layers.17.input_layernorm",
+      "model.layers.17.mlp.gate",
+      "model.layers.29.mlp.gate",
+      "model.layers.51.self_attn.v_proj.bias",
+      "model.layers.14.post_attention_layernorm",
+      "model.layers.13.self_attn.k_norm",
+      "model.layers.43.self_attn.v_proj.bias",
+      "model.layers.56.input_layernorm",
+      "model.layers.77.self_attn.k_proj.bias",
+      "model.layers.79.mlp.gate.e_score_correction_bias",
+      "model.layers.82.input_layernorm",
+      "model.layers.26.mlp.gate",
+      "model.layers.78.self_attn.v_proj.bias",
+      "model.layers.5.self_attn.q_proj.bias",
+      "model.layers.39.mlp.gate.e_score_correction_bias",
+      "model.layers.67.self_attn.v_proj.bias",
+      "model.layers.51.self_attn.k_proj.bias",
+      "model.layers.35.self_attn.v_proj.bias",
+      "model.layers.67.self_attn.k_norm",
+      "model.layers.69.mlp.gate.e_score_correction_bias",
+      "model.layers.39.input_layernorm",
+      "model.layers.86.mlp.gate",
+      "model.layers.88.self_attn.q_proj.bias",
+      "model.layers.2.self_attn.k_proj.bias",
+      "model.layers.23.mlp.gate.e_score_correction_bias",
+      "model.layers.18.self_attn.k_proj.bias",
+      "model.layers.38.mlp.gate",
+      "model.layers.59.mlp.gate",
+      "model.layers.58.mlp.gate",
+      "model.layers.25.self_attn.q_proj.bias",
+      "model.layers.72.self_attn.q_proj.bias",
+      "model.layers.64.self_attn.k_proj.bias",
+      "model.layers.71.self_attn.v_proj.bias",
+      "model.layers.92.shared_head.norm",
+      "model.layers.43.input_layernorm",
+      "model.layers.30.self_attn.k_norm",
+      "model.layers.53.mlp.gate",
+      "model.layers.7.input_layernorm",
+      "model.layers.62.self_attn.v_proj.bias",
+      "model.layers.47.self_attn.k_norm",
+      "model.layers.53.post_attention_layernorm",
+      "model.layers.60.mlp.gate",
+      "model.layers.85.self_attn.q_proj.bias",
+      "model.layers.53.self_attn.k_proj.bias",
+      "model.layers.48.self_attn.q_proj.bias",
+      "model.layers.15.self_attn.v_proj.bias",
+      "model.layers.71.self_attn.q_proj.bias",
+      "model.layers.84.mlp.gate",
+      "model.layers.20.self_attn.q_proj.bias",
+      "model.layers.8.self_attn.v_proj.bias",
+      "model.layers.20.mlp.gate",
+      "model.layers.39.self_attn.k_norm",
+      "model.layers.57.mlp.gate",
+      "model.layers.40.post_attention_layernorm",
+      "model.layers.62.post_attention_layernorm",
+      "model.layers.68.self_attn.q_norm",
+      "model.layers.8.mlp.gate",
+      "model.layers.55.self_attn.q_proj.bias",
+      "model.layers.52.mlp.gate.e_score_correction_bias",
+      "model.layers.60.self_attn.k_proj.bias",
+      "model.layers.47.self_attn.q_proj.bias",
+      "model.layers.0.input_layernorm",
+      "model.layers.12.self_attn.k_proj.bias",
+      "model.layers.21.self_attn.q_norm",
+      "model.layers.51.self_attn.q_norm",
+      "model.layers.80.self_attn.q_proj.bias",
+      "model.layers.15.input_layernorm",
+      "model.layers.18.mlp.gate.e_score_correction_bias",
+      "model.layers.2.self_attn.q_proj.bias",
+      "model.layers.58.self_attn.k_proj.bias",
+      "model.layers.58.post_attention_layernorm",
+      "model.layers.5.self_attn.v_proj.bias",
+      "model.layers.3.self_attn.q_proj.bias",
+      "model.layers.75.mlp.gate.e_score_correction_bias",
+      "model.layers.45.self_attn.q_proj.bias",
+      "model.layers.43.self_attn.q_proj.bias",
+      "model.layers.28.self_attn.q_proj.bias",
+      "model.layers.10.mlp.gate",
+      "model.layers.36.post_attention_layernorm",
+      "model.layers.19.self_attn.k_norm",
+      "model.layers.66.post_attention_layernorm",
+      "model.layers.90.post_attention_layernorm",
+      "model.layers.23.self_attn.q_proj.bias",
+      "model.layers.87.self_attn.v_proj.bias",
+      "model.layers.36.self_attn.k_norm",
+      "model.layers.35.self_attn.k_norm",
+      "model.layers.44.input_layernorm",
+      "model.layers.24.mlp.gate",
+      "model.layers.34.post_attention_layernorm",
+      "model.layers.11.post_attention_layernorm",
+      "model.layers.13.mlp.gate",
+      "model.layers.85.input_layernorm",
+      "model.layers.52.mlp.gate",
+      "model.norm",
+      "model.layers.51.mlp.gate.e_score_correction_bias",
+      "model.layers.25.self_attn.k_proj.bias",
+      "model.layers.64.self_attn.v_proj.bias",
+      "model.layers.45.self_attn.k_proj.bias",
+      "model.layers.82.self_attn.q_proj.bias",
+      "model.layers.30.mlp.gate.e_score_correction_bias",
+      "model.layers.31.self_attn.k_norm",
+      "model.layers.67.mlp.gate.e_score_correction_bias",
+      "model.layers.52.self_attn.q_proj.bias",
+      "model.layers.57.input_layernorm",
+      "model.layers.69.mlp.gate",
+      "model.layers.85.self_attn.q_norm",
+      "model.layers.78.self_attn.q_norm",
+      "model.layers.10.self_attn.q_proj.bias",
+      "model.layers.48.self_attn.q_norm",
+      "model.layers.92.mlp.gate.e_score_correction_bias",
+      "model.layers.67.post_attention_layernorm",
+      "model.layers.28.self_attn.q_norm",
+      "model.layers.74.self_attn.k_proj.bias",
+      "model.layers.58.mlp.gate.e_score_correction_bias",
+      "model.layers.67.self_attn.q_norm",
+      "model.layers.46.self_attn.q_norm",
+      "model.layers.82.self_attn.q_norm",
+      "model.layers.9.mlp.gate.e_score_correction_bias",
+      "model.layers.16.self_attn.q_norm",
+      "model.layers.92.self_attn.q_proj.bias",
+      "model.layers.63.mlp.gate.e_score_correction_bias",
+      "model.layers.38.post_attention_layernorm",
+      "model.layers.72.self_attn.q_norm",
+      "model.layers.81.self_attn.q_norm",
+      "model.layers.63.self_attn.q_proj.bias",
+      "model.layers.48.self_attn.v_proj.bias",
+      "model.layers.6.input_layernorm",
+      "model.layers.61.mlp.gate",
+      "model.layers.14.input_layernorm",
+      "model.layers.31.self_attn.q_norm",
+      "model.layers.61.self_attn.k_norm",
+      "model.layers.83.input_layernorm",
+      "model.layers.89.post_attention_layernorm",
+      "model.layers.28.mlp.gate",
+      "model.layers.16.self_attn.v_proj.bias",
+      "model.layers.10.input_layernorm",
+      "model.layers.6.mlp.gate.e_score_correction_bias",
+      "model.layers.6.self_attn.k_norm",
+      "model.layers.11.input_layernorm",
+      "model.layers.34.self_attn.k_proj.bias",
+      "model.layers.41.self_attn.q_proj.bias",
+      "model.layers.58.self_attn.v_proj.bias",
+      "model.layers.90.self_attn.q_proj.bias",
+      "model.layers.55.input_layernorm",
+      "model.layers.13.self_attn.k_proj.bias",
+      "model.layers.64.mlp.gate.e_score_correction_bias",
+      "model.layers.47.mlp.gate.e_score_correction_bias",
+      "model.layers.42.post_attention_layernorm"
+    ],
+    "kv_cache_scheme": null,
+    "quant_method": "compressed-tensors",
+    "quantization_status": "compressed",
+    "sparsity_config": {}
+  },
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 1000000,
+  "routed_scaling_factor": 2.5,
+  "scoring_func": "sigmoid",
+  "tie_word_embeddings": false,
+  "topk_group": 1,
+  "topk_method": "noaux_tc",
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.55.0",
+  "use_cache": true,
+  "use_qk_norm": true,
+  "vocab_size": 151552
+}

model-00001-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:46fd95a5a4d3132e89c6a0eb5d7ce0d115db88f324c28bfc355215aaa835fe6e
+size 5366929936

model-00002-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19d4e3430706fd06192dc1fd08f4eec4a33249b9a8bd1ce8266a07f267c213f9
+size 5366681096

model-00003-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8d9c731c3b3b8043b191c86c353eb4117ca2d6f9b4af3979b3d8dd7ce6cbec37
+size 5366688264

model-00004-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:12c380514aa2dfb9b75d5d7f58bd722e12223c1dc94d7e49a7bab9876af0f011
+size 5366688728

model-00005-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c8a1f4ad5381a150242bedea0f56a9b9f5a774b767a0cb6bd8ce6e2bf97f8d9a
+size 5336255280

model-00006-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6086040dfc311b6850acd2c4d0b6164f448349db49d4aba11e9f98a5fb8aa9c0
+size 5361367224

model-00007-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1e427e30dce96e9d1d1f8e1bf274f88eaca82259c1df88d2d8dee3f43713ad73
+size 5366682424

model-00008-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:93bf78a9104a7a58f840c9866f0955e06da563404e143595b24c2764456c798f
+size 5366689600

model-00009-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9dee730dad50640292c621b0e60e6adee6bcf5b2b8d45350297d37e7bec99463
+size 5366689728

model-00010-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ad1d46f2d90dca5f8e03a513a786a608523ae2766b88aa207cbefb39df8d6858
+size 5315234592

model-00011-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3712f6b7ca32955a73324ffd75360a2eb5cdbee0875b799efc89eaf85a675b4a
+size 5366645480

model-00012-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a1a4915a03a75c0129745bed1f6e97e9f2656e76c4496825480f201bbaba7495
+size 5366689592

model-00013-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b254e2ec10ff12de1fe7fe0d8d3bf02e3ac2ed73a65d402548f01a8420e7bf7
+size 5366682448

model-00014-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:762c51d514e8c601d23b522c0884446d5980736979ff06399fdf45cdcecfd370
+size 5366689744

model-00015-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e2ba03e2b58d7269860b7a9200051d10c6c4ed12319fcfa4408f9313bb15eaf8
+size 5362417904

model-00016-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:275b34b73886a8467058f832f3e95675a3e55a870a8fcd2a89d2ee61ec944881
+size 5366682376

model-00017-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a05238e1ce89f21381439a9316e5cf5f4efcf4e1dabbba99e553fc79467efe65
+size 5366689592

model-00018-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0dce4d864c37e970999e7eff6170c8cc454ad90e8cd4e4921d8a0aab3f7335c5
+size 5366689648

model-00019-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1dbfefe6450271a2f751cb2910ca3e4173a3744cc472f7f25022785564b000af
+size 5366682608

model-00020-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e393ec4338303282410ab3eb6d5ba83cd3adcabb8db620da7b0075295e677e29
+size 5362417904

model-00021-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b31114859ae368cdd1456f3846f712675bc1db143a370938238451b6fb6f179d
+size 5366689576

model-00022-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02dfdcfad0b864928c8fdbb2edc334860927d7fa2670201fbbae7e2a32fae1fa
+size 5366682424

model-00023-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d5143c6e2cebb9d3f563e3f0a85dfecfe3467a6611042e859837f1174c9d0826
+size 5366689680

model-00024-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fa85afd6adf5f24b420eee5eb9eb0778e83d8a621f036311c03872e3138c965f
+size 5366689808

model-00025-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c1bcc5f94841bb74626b88c5cb40f6834209984f190d76650f133d4ef0d7a672
+size 5362410784

model-00026-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b9160e8c68884ea25d3f38367ab2c17c3613c81579a73f81f9fa9c8500636f62
+size 5366689592

model-00027-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd7bc6562ab759ae5bed1ac09b9ddf0a1b56b6c8181e857c6e03efe78fc4fd92
+size 5366689592

model-00028-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9d6ea452ec60ab509e535b6b9e8a550d56c827cee4865ca571a19ad12eadb9ed
+size 5366682544

model-00029-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f9bf2f5e5f148c83fc03b8883e2152dc576874396389798e2dc02927188e668f
+size 5365076224

model-00030-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:372c694fb2a075121632346b595cfe5f2e1168620564fc5575e5f5154cae6f3f
+size 5364031192

model-00031-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9dd2544fa4ccb8747875e07daa32ab6dfec5436038736da5e79aa44e8f7eaab9
+size 5366682424

model-00032-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e392cce4c58d91603aa25f9555375912b9fab1473933f31feb35c4d08f9140d
+size 5366689616

model-00033-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1e8e886932de0c8aaf1dc382def28f434207f53fdcbfb64deda198d3475b5b50
+size 5366689744

model-00034-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5747c7e07c4a57fcdcb7358a0a956dd64745ba85aa3a9416adef3cec96d7621d
+size 5362410736

model-00035-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:87408cfa34ced774edc1c11a8e3945d00f9d36347efd1f27b4b53985d28658db
+size 5366689544

model-00036-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:649e734f09914d067203afd07b5cecce6ff7936165b3bacd5dcbf30fc79f0457
+size 5366689592

model-00037-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fb83c432807f5e601b03128fd9acec400ffa6088af2f008c670f3a28efd7c427
+size 5366682480

model-00038-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b5f3eed51f55b95c1738253091c1577e5d97120c410a1a45c3de2f27d23d360
+size 5366689776

model-00039-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:29d0671177bcdaa4a58e164c86d1568d21c530585632f1e2cc294693eeb860a7
+size 5362417904

model-00040-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:78430c317f9145f46d735877b61c62eaf76fdaea17cef8d4fbebe4c0664e52f2
+size 5366682408

model-00041-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:196452eb9891072262ed994936e9df573e26c68346ba5ffabb07e4168d30cb98
+size 4026197432

model-00042-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c5344ed0a545f40db7029547e1e473c2a306c1cc2084c5f4c97808f7705a74aa
+size 1551892608

special_tokens_map.json

@@ -0,0 +1,40 @@
+{
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "[MASK]",
+    "[gMASK]",
+    "[sMASK]",
+    "<sop>",
+    "<eop>",
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>",
+    "<|observation|>",
+    "<|begin_of_image|>",
+    "<|end_of_image|>",
+    "<|begin_of_video|>",
+    "<|end_of_video|>",
+    "<|begin_of_audio|>",
+    "<|end_of_audio|>",
+    "<|begin_of_transcription|>",
+    "<|end_of_transcription|>",
+    "<|code_prefix|>",
+    "<|code_middle|>",
+    "<|code_suffix|>",
+    "/nothink"
+  ],
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bda8e2146c3bb7b7e0fc96dcc4f0aeff041c6c27952e3ace0665663ebff346ba
+size 19970700

tokenizer_config.json

@@ -0,0 +1,325 @@
+{
+  "added_tokens_decoder": {
+    "151329": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151330": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151331": {
+      "content": "[gMASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151332": {
+      "content": "[sMASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151333": {
+      "content": "<sop>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151334": {
+      "content": "<eop>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151335": {
+      "content": "<|system|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151336": {
+      "content": "<|user|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151337": {
+      "content": "<|assistant|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151338": {
+      "content": "<|observation|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151339": {
+      "content": "<|begin_of_image|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151340": {
+      "content": "<|end_of_image|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151341": {
+      "content": "<|begin_of_video|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151342": {
+      "content": "<|end_of_video|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151343": {
+      "content": "<|begin_of_audio|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151344": {
+      "content": "<|end_of_audio|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151345": {
+      "content": "<|begin_of_transcription|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151346": {
+      "content": "<|end_of_transcription|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151347": {
+      "content": "<|code_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151348": {
+      "content": "<|code_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151349": {
+      "content": "<|code_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151350": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151351": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151352": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151353": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151354": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151355": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151356": {
+      "content": "<arg_key>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151357": {
+      "content": "</arg_key>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151358": {
+      "content": "<arg_value>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151359": {
+      "content": "</arg_value>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151360": {
+      "content": "/nothink",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151361": {
+      "content": "<|begin_of_box|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151362": {
+      "content": "<|end_of_box|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151363": {
+      "content": "<|image|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151364": {
+      "content": "<|video|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "[MASK]",
+    "[gMASK]",
+    "[sMASK]",
+    "<sop>",
+    "<eop>",
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>",
+    "<|observation|>",
+    "<|begin_of_image|>",
+    "<|end_of_image|>",
+    "<|begin_of_video|>",
+    "<|end_of_video|>",
+    "<|begin_of_audio|>",
+    "<|end_of_audio|>",
+    "<|begin_of_transcription|>",
+    "<|end_of_transcription|>",
+    "<|code_prefix|>",
+    "<|code_middle|>",
+    "<|code_suffix|>",
+    "/nothink"
+  ],
+  "clean_up_tokenization_spaces": false,
+  "do_lower_case": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 128000,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "left",
+  "remove_space": false,
+  "tokenizer_class": "PreTrainedTokenizerFast"
+}