diff --git a/.gitattributes b/.gitattributes
index a6344aac8c09253b3b630fb776ae94478aa0275b..52373fe24473b1aa44333d318f578ae6bf04b49b 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
diff --git a/LICENSE b/LICENSE
new file mode 100644
index 0000000000000000000000000000000000000000..f860919cd5a5377c0bc9d590ea9d1b8743475ca5
--- /dev/null
+++ b/LICENSE
@@ -0,0 +1,29 @@
+GLM-5.3 License
+
+Copyright (c) 2026 Z.AI
+
+Permission is hereby granted, free of charge, to any person or entity (the "Licensee") obtaining a copy of this software — including the model weights, parameters, configuration files, inference and training code, and associated documentation (collectively, the "Software") — to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software; to run, deploy, fine-tune, or otherwise modify the Software and create derivative works from it; and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+1. The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. The Licensee's use of the Software must comply with applicable laws and regulations.
+
+2. "Model as a Service" means giving a third party access to language model inference or fine-tuning (e.g., via API) in a manner that allows such third party to exercise meaningful control over the inputs, parameters, or training data. This does not include (a) end-user products with model capabilities solely embedded within specific features or harnesses, or (b) mere relaying of requests to models hosted by others.
+If the Licensee or any of its affiliates operates a Model as a Service business, and the aggregate revenue of the Licensee and its affiliates exceeds 10 billion US dollars (or the equivalent in other currencies) in total over any consecutive 12 months, the Licensee must pass Z.AI's security review before using the Software or its derivative works for any commercial purpose. The scope and method of the security review shall be reasonably determined by Z.AI.
+
+3. THE SOFTWARE AND ANY OUTPUT AND RESULTS THEREFROM ARE PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL Z.AI OR ITS AFFILIATES OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+For any questions regarding this license, please contact glmlicense@z.ai.
+
+-----
+
+版权所有 (c) 2026 Z.AI
+
+特此免费授予任何获得本软件副本的个人或实体（"被许可方"）——包括模型权重、参数、配置文件、推理和训练代码及相关文档（统称"软件"）——不受限制地处理本软件的权利，包括但不限于：使用、复制、修改、合并、发布、分发、再许可和/或销售软件副本；运行、部署、微调或以其他方式修改软件并创建衍生作品；以及允许获得软件的其他人行使上述权利，但须遵守以下条件：
+
+1. 上述版权声明和本许可声明应包含在软件的所有副本或实质性部分中。被许可方对软件的使用必须符合适用法律法规。
+
+2. "模型即服务"指以允许第三方对输入、参数或训练数据行使实质性控制的方式，向第三方提供语言模型推理或微调服务（如通过API）。不包括：(a) 模型能力仅嵌入特定功能或框架中的终端用户产品，或(b) 单纯转发请求至他人托管的模型。
+若被许可方或其关联方运营"模型即服务"业务，且被许可方及关联方在任意连续12个月内累计总收入超过100亿美元（或等值其他货币），则被许可方在使用软件或其衍生作品进行任何商业用途之前，须通过Z.AI的安全审查。安全审查的范围和方式由Z.AI合理确定。
+
+3. 软件及其任何输出和结果均按"现状"提供，不附带任何形式的保证，无论是明示还是暗示，包括但不限于适销性、特定用途适用性和不侵权的保证。在任何情况下，Z.AI或其关联方或版权持有人均不对任何索赔、损害或其他责任承担责任，无论该责任是基于合同、侵权或其他方式，因软件或使用软件而产生或与之相关。
+
+如对本许可有任何疑问，请联系 glmlicense@z.ai。
diff --git a/README.md b/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..b471876831f62a8e08b03e23831331baec14992a
--- /dev/null
+++ b/README.md
@@ -0,0 +1,94 @@
+---
+license: other
+license_name: glm-5.3
+license_link: LICENSE
+base_model: zai-org/GLM-5.3
+base_model_relation: quantized
+tags:
+- mlx
+- apple-silicon
+- glm_moe_dsa
+- mixture-of-experts
+- 4-bit
+pipeline_tag: text-generation
+library_name: mlx
+---
+
+# GLM-5.3-MLX-mixed-4_8bit
+
+MLX (Apple Silicon) build of [**GLM-5.3**](https://huggingface.co/zai-org/GLM-5.3) — 744B-parameter
+`glm_moe_dsa` MoE (256 experts, top-8; MLA with DeepSeek-V3.2-style sparse attention) — quantized
+to **4-bit experts / 8-bit everything else**.
+
+**These files are modified**: converted from the upstream **bfloat16** release
+([GLM-5.3-BF16](https://huggingface.co/zai-org/GLM-5.3-BF16)) to MLX and quantized; the architecture
+is unchanged. The multi-token-prediction layer (78) is not included.
+
+## Runtime — read this
+
+This checkpoint bundles `glm_moe_dsa.py` (declared via `model_file`) and needs it:
+
+```bash
+pip install -U mlx-lm
+mlx_lm.generate --model pipenetwork/GLM-5.3-MLX-mixed-4_8bit --trust-remote-code --prompt "..." --max-tokens 300
+```
+
+mlx-lm's own `glm_moe_dsa` builds a lightning indexer on all 78 layers, but GLM-5.2/5.3 ship
+indexer weights on 21 (`indexer_types`: the other 57 "shared" layers reuse the previous full layer's
+top-k selection). A strict load of the release fails with 285 missing parameters; `mlx_lm.load`
+loads leniently and leaves those 57 indexers at random initialisation. Prompts up to 2048 tokens are
+unaffected (the indexer is bypassed below `index_topk`); beyond that, 57 layers attend to keys
+chosen by random projections. The bundled runtime implements the schedule as the reference does,
+plus the reference's fp32 indexer scores and router logits and the indexer LayerNorm epsilon.
+Tiny-config parity against `transformers` 5.16 is **4e-7** with the sparse path live, cached decode
+exact; strict loading of this checkpoint reports zero missing and zero unexpected tensors. Details and
+tests: [https://github.com/PipeNetwork/glm53-mlx](https://github.com/PipeNetwork/glm53-mlx).
+
+## Size and what is quantized
+
+**427.8 GB** on disk. RAM: 512 GB, tight.
+
+| group | share of parameters | this build |
+|---|---:|---|
+| routed experts (`switch_mlp`, 75 layers × 256) | 724.8B (97.5%) | 4-bit, group 64 |
+| attention (MLA), shared experts, dense layers 0–2, embeddings, `lm_head` | 18.4B (2.5%) | 8-bit, group 64 |
+| lightning indexer (21 layers), MoE router + correction bias, norms | 0.3B | as stored (bf16 / fp32) |
+
+Source precision: the FP8 release is a lossy derivative of the bf16 one (dequantized FP8 weights
+differ from bf16 by up to 1.6e-2 on values of 0.46 — half an e4m3 step). The ladder row `fp8` is the FP8 release itself measured against bf16: its error is the floor any FP8-sourced build inherits.
+
+## Quality
+
+Two measurements, because at 744B most of the ladder cannot be loaded on a 512 GB machine:
+
+**Per-layer divergence vs bf16** (`scripts/eval_ladder.py`): every decoder layer run in bf16 and in
+each recipe on identical inputs (16,384 tokens of wikitext-2), *teacher-forced* (each layer
+sees bf16 inputs — isolates its own damage) and *free-running* (each recipe feeds itself — what
+inference does). Relative L2 error of the layer output; lower is better.
+
+| recipe | teacher-forced (mean over layers) | free-running (final layer) | cosine (final) |
+|---|---:|---:|---:|
+| 8bit | 0.00685 | 0.13119 | 0.98945 |
+| 6bit | 0.01465 | 0.16736 | 0.98389 |
+| 5bit | 0.02651 | 0.22521 | 0.97272 |
+| 4bit | 0.05161 | 0.35740 | 0.93390 |
+| mixed-4_8bit | 0.02524 | 0.24951 | 0.96710 |
+| mixed-3_6bit | 0.05242 | 0.42380 | 0.90624 |
+| fp8 | 0.01741 | 0.17321 | 0.98320 |
+
+**Perplexity** on wikitext-2 (test), 288,627 tokens in 141 windows of 2048, for the builds
+that fit this machine, scored on identical windows:
+
+| build | size | perplexity [95% CI] |
+|---|---:|---|
+| [4bit](https://huggingface.co/pipenetwork/GLM-5.3-MLX-4bit) | 418.6 GB | 2.8636 [2.6681, 3.0714] |
+| [mixed-4_8bit](https://huggingface.co/pipenetwork/GLM-5.3-MLX-mixed-4_8bit) | 427.8 GB | 2.7420 [2.5533, 2.9477] |
+| [mixed-3_6bit](https://huggingface.co/pipenetwork/GLM-5.3-MLX-mixed-3_6bit) | 332.6 GB | 3.0338 [2.8366, 3.2386] |
+
+**Recommendation.** For a 512 GB Mac, **mixed 4/8-bit** (427.7 GB): perplexity 2.7420, a paired 4.3% better than uniform 4-bit (ratio 0.9575 [0.9537, 0.9612], better on 98.6% of windows) for 9 GB more — the 2.5% of non-expert weights are worth their 8 bits, as on every model we have measured. Uniform 4-bit (418.6 GB) is the fallback when those 9 GB matter. **Mixed 3/6-bit** (332.6 GB) is the 384 GB-class option, at a real cost: 3.0338, +5.9% over 4-bit and +10.6% over mixed 4/8 — it leads the ladder for the first ten layers and then 3-bit expert damage compounds. Among the builds that cannot be run here, the ladder puts 8-bit closest to bfloat16 (free-running error 0.131), then 6-bit (0.167); the **upstream FP8 release scores 0.173, between 6-bit and 5-bit**, which is why these are converted from the bf16 release. 5-bit (0.225) sits just above mixed 4/8 (0.250) at 100 GB more.
+
+Greedy generation (a collapse detector, not a ranking) is coherent on every published build.
+
+## License
+
+[GLM-5.3 license](LICENSE), as the upstream model. Port code: [https://github.com/PipeNetwork/glm53-mlx](https://github.com/PipeNetwork/glm53-mlx).
diff --git a/__pycache__/glm_moe_dsa.cpython-314.pyc b/__pycache__/glm_moe_dsa.cpython-314.pyc
new file mode 100644
index 0000000000000000000000000000000000000000..b6a8712fd7199f63038149bdcc5495676c7efcf8
Binary files /dev/null and b/__pycache__/glm_moe_dsa.cpython-314.pyc differ
diff --git a/chat_template.jinja b/chat_template.jinja
new file mode 100644
index 0000000000000000000000000000000000000000..d226a6928e6ba040e2eadccaf832c037bc4cd98f
--- /dev/null
+++ b/chat_template.jinja
@@ -0,0 +1,251 @@
+[gMASK]<sop>
+{%- set effective_reasoning_effort = reasoning_effort if reasoning_effort is defined and reasoning_effort in ['low', 'high'] else 'max' -%}
+{%- if effective_reasoning_effort is not none -%}<|system|>Reasoning Effort: {{ effective_reasoning_effort | capitalize }}{%- endif -%}
+{%- set clear_thinking = clear_thinking if clear_thinking is defined else false -%}
+{%- if tools -%}
+{%- macro tool_to_json(tool) -%}
+    {%- set ns_tool = namespace(first=true) -%}
+    {{ '{' -}}
+    {%- for k, v in tool.items() -%}
+        {%- if k != 'defer_loading' and k != 'strict' -%}
+            {%- if not ns_tool.first -%}{{- ', ' -}}{%- endif -%}
+            {%- set ns_tool.first = false -%}
+            "{{ k }}": {{ v | tojson(ensure_ascii=False) }}
+        {%- endif -%}
+    {%- endfor -%}
+    {{- '}' -}}
+{%- endmacro -%}
+{%- macro tool_references_to_response(refs) -%}
+    {{- '<tool_response><tools>\n' -}}
+    {%- for tr in refs -%}
+        {%- for tool in tools -%}
+            {%- if 'function' in tool -%}
+                {%- set tool = tool['function'] -%}
+            {%- endif -%}
+            {%- if tool.name == tr.name -%}
+                {{- tool_to_json(tool) + '\n' -}}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- endfor -%}
+    {{- '</tools></tool_response>' -}}
+{%- endmacro -%}
+<|system|>
+# Tools
+
+You may call one or more functions to assist with the user query.
+
+You are provided with function signatures within <tools></tools> XML tags:
+<tools>
+{% for tool in tools %}
+{%- if 'function' in tool -%}
+    {%- set tool = tool['function'] -%}
+{%- endif -%}
+{% if tool.defer_loading is not defined or not tool.defer_loading %}
+{{ tool_to_json(tool) }}
+{% endif %}
+{% endfor %}
+</tools>
+
+For each function call, output the function name and arguments within the following XML format:
+<tool_call>{function-name}<arg_key>{arg-key-1}</arg_key><arg_value>{arg-value-1}</arg_value><arg_key>{arg-key-2}</arg_key><arg_value>{arg-value-2}</arg_value>...</tool_call>{%- endif -%}
+{%- macro visible_text(content) -%}
+    {%- if content is string -%}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping -%}
+        {%- for item in content -%}
+            {%- if item is mapping and item.type == 'text' -%}
+                {{- item.text }}
+            {%- elif item is string -%}
+                {{- item }}
+            {%- elif item is mapping and item.type in ['image', 'image_url', 'video', 'video_url', 'audio', 'audio_url', 'input_audio'] -%}
+                {%- set media_type = item.type | replace('_url', '') | replace('input_', '') -%}
+                {{- "<reminder>You are unable to process this " ~ media_type ~ " because you don't have multi-modal input ability. Try different methods.</reminder>" }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{- content }}
+    {%- endif -%}
+{%- endmacro -%}
+{%- macro tool_response(text) -%}
+{{- '<tool_response>' + text + '</tool_response>' -}}
+{%- endmacro -%}
+{%- macro render_tool_response(m) -%}
+{%- if m.content is string -%}
+    {{- tool_response(m.content) -}}
+{%- elif m.content and m.content is not mapping and m.content.0.type == "tool_reference" -%}
+    {{- tool_references_to_response(m.content) -}}
+{%- elif is_list_of_outputs(m) -%}
+    {%- for tr in m.content -%}
+        {%- if tr.output is iterable and tr.output is not string and tr.output is not mapping and tr.output and tr.output.0.type == "tool_reference" -%}
+            {{- tool_references_to_response(tr.output) -}}
+        {%- else -%}
+            {{- tool_response(visible_text(tr.output)) -}}
+        {%- endif -%}
+    {%- endfor -%}
+{%- else -%}
+    {{- tool_response(visible_text(m.content)) -}}
+{%- endif -%}
+{%- endmacro -%}
+{%- macro id_of(obj) -%}
+    {%- if obj.tool_call_id -%}
+        {{- obj.tool_call_id -}}
+    {%- elif obj.id -%}
+        {{- obj.id -}}
+    {%- endif -%}
+{%- endmacro -%}
+{%- macro is_list_of_outputs(m) -%}
+    {%- if m.content and m.content.0.output is defined -%}1{%- endif -%}
+{%- endmacro -%}
+{%- macro has_dup_tool_result_id(lo, hi, target) -%}
+    {%- set ns_cnt = namespace(n=0) -%}
+    {%- for k in range(lo, hi + 1) -%}
+        {%- set m = messages[k] -%}
+        {%- if is_list_of_outputs(m) -%}
+            {%- for entry in m.content -%}
+                {%- if id_of(entry) == target -%}
+                    {%- set ns_cnt.n = ns_cnt.n + 1 -%}
+                {%- endif -%}
+            {%- endfor -%}
+        {%- elif id_of(m) == target -%}
+            {%- set ns_cnt.n = ns_cnt.n + 1 -%}
+        {%- endif -%}
+        {%- if ns_cnt.n > 1 -%}{%- break -%}{%- endif -%}
+    {%- endfor -%}
+    {%- if ns_cnt.n > 1 -%}1{%- endif -%}
+{%- endmacro -%}
+{%- macro tc_id_exists(tcs, target) -%}
+    {%- set ns_f = namespace(found=false) -%}
+    {%- for tc in tcs -%}
+        {%- if id_of(tc) == target -%}
+            {%- set ns_f.found = true -%}
+            {%- break -%}
+        {%- endif -%}
+    {%- endfor -%}
+    {%- if ns_f.found -%}1{%- endif -%}
+{%- endmacro -%}
+{%- set ns = namespace(last_user_index=-1) -%}
+{%- for m in messages %}
+    {%- if m.role == 'user' %}
+        {%- set ns.last_user_index = loop.index0 -%}
+    {%- endif %}
+{%- endfor %}
+{%- for m in messages -%}
+{%- if m.role == 'user' -%}<|user|>{{ visible_text(m.content) }}
+{%- elif m.role == 'assistant' -%}
+<|assistant|>
+{%- set content = visible_text(m.content) %}
+{%- if m.reasoning_content is string %}
+    {%- set reasoning_content = m.reasoning_content %}
+{%- elif '</think>' in content %}
+    {%- set reasoning_content = content.split('</think>')[0].split('<think>')[-1] %}
+    {%- set content = content.split('</think>')[-1] %}
+{%- endif %}
+{%- if (not clear_thinking or loop.index0 > ns.last_user_index) and reasoning_content is defined -%}
+{{ '<think>' + reasoning_content +  '</think>'}}
+{%- else -%}
+{{ '<think></think>' }}
+{%- endif -%}
+{%- if content.strip() -%}
+{{ content.strip() }}
+{%- endif -%}
+{% if m.tool_calls %}
+{% for tc in m.tool_calls %}
+{%- if tc.function %}
+    {%- set tc = tc.function %}
+{%- endif %}
+{{- '<tool_call>' + tc.name -}}
+{% set _args = tc.arguments %}{% for k, v in _args.items() %}<arg_key>{{ k }}</arg_key><arg_value>{{ v | tojson(ensure_ascii=False) if v is not string else v }}</arg_value>{% endfor %}</tool_call>{% endfor %}
+{% endif %}
+{%- elif m.role == 'tool' -%}
+{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+    {{- '<|observation|>' -}}
+    {%- set block_start = loop.index0 -%}
+    {%- set ns_blk = namespace(end=block_start) -%}
+    {%- for j in range(block_start, messages|length) -%}
+        {%- if messages[j].role == 'tool' -%}
+            {%- set ns_blk.end = j -%}
+        {%- else -%}
+            {%- break -%}
+        {%- endif -%}
+    {%- endfor -%}
+    {%- set ns_a = namespace(tool_calls=none) -%}
+    {%- if block_start > 0 and messages[block_start - 1].role == 'assistant' and messages[block_start - 1].tool_calls -%}
+        {%- set ns_a.tool_calls = messages[block_start - 1].tool_calls -%}
+    {%- endif -%}
+    {%- set ns_chk = namespace(can_sort=true) -%}
+    {%- if not ns_a.tool_calls -%}
+        {%- set ns_chk.can_sort = false -%}
+    {%- else -%}
+        {%- for k in range(block_start, ns_blk.end + 1) -%}
+            {%- set m = messages[k] -%}
+            {%- if is_list_of_outputs(m) -%}
+                {%- for entry in m.content -%}
+                    {%- set eid = id_of(entry) -%}
+                    {%- if not eid -%}
+                        {%- set ns_chk.can_sort = false -%}
+                    {%- elif has_dup_tool_result_id(block_start, ns_blk.end, eid) -%}
+                        {%- set ns_chk.can_sort = false -%}
+                    {%- elif not tc_id_exists(ns_a.tool_calls, eid) -%}
+                        {%- set ns_chk.can_sort = false -%}
+                    {%- endif -%}
+                {%- endfor -%}
+            {%- else -%}
+                {%- set tk_id = id_of(m) -%}
+                {%- if not tk_id -%}
+                    {%- set ns_chk.can_sort = false -%}
+                {%- elif has_dup_tool_result_id(block_start, ns_blk.end, tk_id) -%}
+                    {%- set ns_chk.can_sort = false -%}
+                {%- elif not tc_id_exists(ns_a.tool_calls, tk_id) -%}
+                    {%- set ns_chk.can_sort = false -%}
+                {%- endif -%}
+            {%- endif -%}
+        {%- endfor -%}
+        {%- for i in range(ns_a.tool_calls | length) -%}
+            {%- set tc_id = id_of(ns_a.tool_calls[i]) -%}
+            {%- if not tc_id -%}
+                {%- set ns_chk.can_sort = false -%}
+            {%- endif -%}
+            {%- for j in range(i + 1, ns_a.tool_calls | length) -%}
+                {%- if id_of(ns_a.tool_calls[j]) == tc_id -%}
+                    {%- set ns_chk.can_sort = false -%}
+                {%- endif -%}
+            {%- endfor -%}
+        {%- endfor -%}
+    {%- endif -%}
+    {%- if ns_chk.can_sort -%}
+        {%- for tc in ns_a.tool_calls -%}
+            {%- set tc_id = id_of(tc) -%}
+            {%- for k in range(block_start, ns_blk.end + 1) -%}
+                {%- set m = messages[k] -%}
+                {%- if is_list_of_outputs(m) -%}
+                    {%- for entry in m.content -%}
+                        {%- set eid = id_of(entry) -%}
+                        {%- if eid == tc_id -%}
+                            {%- if entry.output is iterable and entry.output is not string and entry.output is not mapping and entry.output and entry.output.0.type == "tool_reference" -%}
+                                {{- tool_references_to_response(entry.output) -}}
+                            {%- else -%}
+                                {{- tool_response(visible_text(entry.output)) -}}
+                            {%- endif -%}
+                        {%- endif -%}
+                    {%- endfor -%}
+                {%- else -%}
+                    {%- set tk_id = id_of(m) -%}
+                    {%- if tk_id == tc_id -%}
+                        {{- render_tool_response(m) -}}
+                    {%- endif -%}
+                {%- endif -%}
+            {%- endfor -%}
+        {%- endfor -%}
+    {%- else -%}
+        {%- for k in range(block_start, ns_blk.end + 1) -%}
+            {{- render_tool_response(messages[k]) -}}
+        {%- endfor -%}
+    {%- endif -%}
+{% endif -%}
+{%- elif m.role == 'system' -%}
+<|system|>{{ visible_text(m.content) }}
+{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    <|assistant|>{{- '<think>' -}}
+{%- endif -%}
diff --git a/config.json b/config.json
new file mode 100644
index 0000000000000000000000000000000000000000..99eb1a6f6a36d9cd4c2d272d3f64027c9e26264b
--- /dev/null
+++ b/config.json
@@ -0,0 +1,5865 @@
+{
+  "architectures": [
+    "GlmMoeDsaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "dtype": "bfloat16",
+  "eos_token_id": [
+    154820,
+    154827,
+    154829
+  ],
+  "ep_size": 1,
+  "first_k_dense_replace": 3,
+  "head_dim": 192,
+  "hidden_act": "silu",
+  "hidden_size": 6144,
+  "index_head_dim": 128,
+  "index_n_heads": 32,
+  "index_share_for_mtp_iteration": true,
+  "index_skip_topk_offset": 3,
+  "index_topk": 2048,
+  "index_topk_freq": 4,
+  "index_topk_pattern": null,
+  "indexer_rope_interleave": true,
+  "indexer_types": [
+    "full",
+    "full",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared",
+    "full",
+    "shared",
+    "shared",
+    "shared"
+  ],
+  "initializer_range": 0.02,
+  "intermediate_size": 12288,
+  "kv_lora_rank": 512,
+  "max_position_embeddings": 1048576,
+  "mlp_layer_types": [
+    "dense",
+    "dense",
+    "dense",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse"
+  ],
+  "model_type": "glm_moe_dsa",
+  "moe_intermediate_size": 2048,
+  "moe_layer_freq": 1,
+  "moe_router_dtype": "float32",
+  "n_group": 1,
+  "n_routed_experts": 256,
+  "n_shared_experts": 1,
+  "norm_topk_prob": true,
+  "num_attention_heads": 64,
+  "num_experts_per_tok": 8,
+  "num_hidden_layers": 78,
+  "num_key_value_heads": 64,
+  "num_nextn_predict_layers": 1,
+  "pad_token_id": 154820,
+  "pretraining_tp": 1,
+  "q_lora_rank": 2048,
+  "qk_head_dim": 256,
+  "qk_nope_head_dim": 192,
+  "qk_rope_head_dim": 64,
+  "rms_norm_eps": 1e-05,
+  "rope_interleave": true,
+  "rope_parameters": {
+    "rope_theta": 8000000,
+    "rope_type": "default"
+  },
+  "routed_scaling_factor": 2.5,
+  "scoring_func": "sigmoid",
+  "tie_word_embeddings": false,
+  "topk_group": 1,
+  "topk_method": "noaux_tc",
+  "transformers_version": "5.15.0",
+  "use_cache": true,
+  "v_head_dim": 256,
+  "vocab_size": 154880,
+  "quantization": {
+    "group_size": 64,
+    "bits": 4,
+    "model.embed_tokens": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.0.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.0.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.0.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.0.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.0.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.0.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.0.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.0.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.0.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.1.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.1.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.1.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.1.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.1.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.1.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.1.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.1.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.1.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.2.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.2.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.2.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.2.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.2.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.2.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.2.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.2.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.2.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.3.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.3.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.3.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.3.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.3.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.3.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.3.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.3.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.3.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.4.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.4.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.4.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.4.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.4.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.4.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.4.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.4.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.4.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.5.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.5.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.5.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.5.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.5.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.5.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.5.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.5.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.5.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.6.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.6.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.6.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.6.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.6.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.6.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.6.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.6.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.6.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.7.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.7.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.7.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.7.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.7.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.7.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.7.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.7.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.7.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.8.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.8.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.8.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.8.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.8.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.8.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.8.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.8.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.8.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.9.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.9.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.9.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.9.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.9.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.9.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.9.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.9.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.9.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.10.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.10.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.10.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.10.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.10.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.10.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.10.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.10.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.10.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.11.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.11.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.11.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.11.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.11.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.11.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.11.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.11.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.11.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.12.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.12.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.12.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.12.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.12.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.12.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.12.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.12.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.12.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.13.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.13.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.13.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.13.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.13.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.13.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.13.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.13.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.13.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.14.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.14.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.14.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.14.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.14.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.14.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.14.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.14.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.14.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.15.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.15.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.15.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.15.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.15.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.15.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.15.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.15.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.15.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.16.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.16.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.16.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.16.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.16.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.16.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.16.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.16.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.16.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.17.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.17.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.17.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.17.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.17.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.17.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.17.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.17.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.17.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.18.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.18.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.18.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.18.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.18.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.18.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.18.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.18.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.18.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.19.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.19.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.19.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.19.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.19.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.19.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.19.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.19.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.19.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.20.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.20.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.20.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.20.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.20.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.20.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.20.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.20.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.20.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.21.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.21.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.21.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.21.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.21.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.21.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.21.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.21.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.21.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.22.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.22.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.22.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.22.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.22.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.22.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.22.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.22.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.22.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.23.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.23.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.23.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.23.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.23.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.23.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.23.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.23.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.23.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.24.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.24.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.24.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.24.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.24.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.24.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.24.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.24.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.24.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.25.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.25.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.25.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.25.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.25.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.25.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.25.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.25.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.25.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.26.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.26.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.26.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.26.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.26.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.26.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.26.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.26.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.26.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.27.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.27.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.27.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.27.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.27.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.27.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.27.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.27.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.27.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.28.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.28.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.28.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.28.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.28.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.28.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.28.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.28.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.28.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.29.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.29.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.29.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.29.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.29.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.29.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.29.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.29.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.29.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.30.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.30.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.30.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.30.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.30.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.30.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.30.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.30.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.30.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.31.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.31.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.31.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.31.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.31.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.31.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.31.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.31.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.31.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.32.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.32.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.32.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.32.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.32.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.32.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.32.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.32.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.32.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.33.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.33.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.33.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.33.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.33.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.33.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.33.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.33.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.33.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.34.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.34.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.34.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.34.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.34.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.34.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.34.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.34.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.34.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.35.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.35.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.35.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.35.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.35.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.35.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.35.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.35.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.35.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.36.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.36.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.36.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.36.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.36.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.36.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.36.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.36.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.36.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.37.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.37.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.37.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.37.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.37.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.37.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.37.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.37.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.37.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.38.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.38.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.38.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.38.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.38.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.38.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.38.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.38.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.38.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.39.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.39.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.39.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.39.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.39.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.39.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.39.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.39.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.39.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.40.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.40.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.40.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.40.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.40.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.40.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.40.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.40.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.40.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.41.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.41.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.41.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.41.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.41.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.41.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.41.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.41.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.41.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.42.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.42.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.42.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.42.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.42.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.42.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.42.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.42.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.42.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.43.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.43.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.43.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.43.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.43.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.43.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.43.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.43.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.43.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.44.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.44.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.44.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.44.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.44.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.44.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.44.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.44.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.44.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.45.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.45.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.45.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.45.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.45.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.45.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.45.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.45.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.45.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.46.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.46.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.46.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.46.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.46.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.46.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.46.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.46.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.46.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.47.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.47.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.47.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.47.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.47.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.47.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.47.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.47.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.47.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.48.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.48.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.48.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.48.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.48.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.48.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.48.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.48.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.48.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.49.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.49.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.49.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.49.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.49.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.49.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.49.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.49.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.49.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.50.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.50.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.50.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.50.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.50.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.50.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.50.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.50.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.50.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.51.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.51.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.51.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.51.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.51.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.51.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.51.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.51.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.51.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.52.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.52.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.52.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.52.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.52.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.52.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.52.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.52.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.52.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.53.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.53.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.53.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.53.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.53.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.53.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.53.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.53.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.53.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.54.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.54.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.54.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.54.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.54.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.54.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.54.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.54.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.54.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.55.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.55.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.55.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.55.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.55.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.55.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.55.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.55.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.55.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.56.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.56.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.56.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.56.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.56.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.56.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.56.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.56.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.56.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.57.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.57.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.57.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.57.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.57.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.57.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.57.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.57.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.57.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.58.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.58.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.58.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.58.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.58.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.58.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.58.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.58.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.58.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.59.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.59.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.59.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.59.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.59.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.59.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.59.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.59.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.59.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.60.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.60.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.60.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.60.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.60.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.60.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.60.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.60.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.60.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.61.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.61.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.61.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.61.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.61.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.61.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.61.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.61.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.61.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.62.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.62.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.62.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.62.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.62.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.62.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.62.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.62.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.62.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.63.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.63.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.63.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.63.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.63.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.63.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.63.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.63.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.63.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.64.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.64.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.64.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.64.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.64.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.64.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.64.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.64.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.64.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.65.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.65.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.65.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.65.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.65.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.65.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.65.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.65.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.65.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.66.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.66.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.66.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.66.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.66.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.66.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.66.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.66.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.66.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.67.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.67.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.67.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.67.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.67.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.67.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.67.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.67.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.67.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.68.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.68.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.68.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.68.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.68.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.68.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.68.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.68.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.68.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.69.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.69.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.69.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.69.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.69.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.69.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.69.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.69.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.69.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.70.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.70.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.70.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.70.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.70.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.70.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.70.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.70.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.70.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.71.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.71.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.71.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.71.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.71.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.71.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.71.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.71.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.71.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.72.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.72.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.72.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.72.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.72.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.72.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.72.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.72.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.72.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.73.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.73.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.73.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.73.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.73.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.73.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.73.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.73.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.73.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.74.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.74.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.74.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.74.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.74.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.74.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.74.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.74.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.74.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.75.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.75.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.75.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.75.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.75.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.75.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.75.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.75.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.75.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.76.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.76.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.76.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.76.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.76.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.76.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.76.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.76.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.76.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.77.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.77.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.77.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.77.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.77.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.77.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.77.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.77.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.77.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "lm_head": {
+      "group_size": 64,
+      "bits": 8
+    }
+  },
+  "quantization_config": {
+    "group_size": 64,
+    "bits": 4,
+    "model.embed_tokens": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.0.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.0.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.0.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.0.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.0.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.0.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.0.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.0.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.0.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.1.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.1.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.1.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.1.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.1.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.1.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.1.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.1.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.1.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.2.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.2.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.2.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.2.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.2.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.2.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.2.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.2.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.2.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.3.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.3.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.3.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.3.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.3.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.3.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.3.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.3.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.3.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.4.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.4.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.4.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.4.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.4.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.4.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.4.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.4.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.4.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.5.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.5.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.5.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.5.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.5.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.5.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.5.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.5.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.5.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.6.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.6.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.6.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.6.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.6.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.6.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.6.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.6.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.6.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.7.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.7.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.7.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.7.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.7.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.7.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.7.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.7.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.7.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.8.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.8.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.8.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.8.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.8.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.8.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.8.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.8.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.8.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.9.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.9.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.9.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.9.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.9.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.9.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.9.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.9.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.9.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.10.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.10.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.10.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.10.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.10.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.10.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.10.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.10.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.10.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.11.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.11.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.11.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.11.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.11.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.11.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.11.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.11.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.11.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.12.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.12.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.12.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.12.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.12.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.12.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.12.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.12.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.12.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.13.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.13.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.13.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.13.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.13.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.13.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.13.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.13.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.13.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.14.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.14.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.14.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.14.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.14.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.14.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.14.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.14.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.14.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.15.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.15.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.15.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.15.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.15.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.15.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.15.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.15.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.15.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.16.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.16.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.16.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.16.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.16.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.16.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.16.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.16.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.16.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.17.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.17.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.17.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.17.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.17.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.17.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.17.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.17.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.17.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.18.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.18.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.18.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.18.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.18.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.18.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.18.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.18.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.18.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.19.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.19.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.19.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.19.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.19.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.19.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.19.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.19.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.19.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.20.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.20.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.20.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.20.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.20.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.20.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.20.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.20.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.20.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.21.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.21.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.21.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.21.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.21.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.21.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.21.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.21.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.21.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.22.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.22.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.22.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.22.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.22.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.22.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.22.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.22.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.22.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.23.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.23.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.23.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.23.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.23.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.23.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.23.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.23.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.23.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.24.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.24.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.24.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.24.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.24.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.24.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.24.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.24.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.24.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.25.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.25.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.25.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.25.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.25.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.25.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.25.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.25.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.25.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.26.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.26.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.26.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.26.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.26.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.26.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.26.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.26.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.26.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.27.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.27.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.27.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.27.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.27.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.27.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.27.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.27.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.27.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.28.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.28.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.28.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.28.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.28.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.28.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.28.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.28.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.28.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.29.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.29.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.29.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.29.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.29.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.29.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.29.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.29.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.29.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.30.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.30.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.30.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.30.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.30.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.30.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.30.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.30.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.30.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.31.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.31.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.31.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.31.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.31.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.31.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.31.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.31.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.31.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.32.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.32.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.32.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.32.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.32.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.32.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.32.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.32.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.32.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.33.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.33.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.33.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.33.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.33.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.33.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.33.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.33.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.33.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.34.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.34.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.34.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.34.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.34.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.34.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.34.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.34.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.34.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.35.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.35.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.35.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.35.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.35.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.35.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.35.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.35.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.35.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.36.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.36.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.36.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.36.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.36.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.36.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.36.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.36.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.36.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.37.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.37.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.37.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.37.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.37.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.37.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.37.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.37.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.37.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.38.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.38.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.38.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.38.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.38.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.38.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.38.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.38.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.38.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.39.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.39.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.39.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.39.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.39.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.39.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.39.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.39.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.39.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.40.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.40.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.40.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.40.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.40.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.40.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.40.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.40.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.40.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.41.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.41.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.41.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.41.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.41.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.41.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.41.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.41.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.41.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.42.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.42.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.42.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.42.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.42.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.42.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.42.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.42.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.42.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.43.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.43.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.43.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.43.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.43.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.43.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.43.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.43.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.43.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.44.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.44.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.44.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.44.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.44.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.44.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.44.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.44.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.44.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.45.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.45.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.45.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.45.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.45.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.45.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.45.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.45.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.45.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.46.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.46.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.46.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.46.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.46.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.46.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.46.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.46.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.46.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.47.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.47.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.47.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.47.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.47.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.47.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.47.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.47.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.47.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.48.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.48.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.48.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.48.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.48.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.48.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.48.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.48.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.48.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.49.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.49.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.49.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.49.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.49.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.49.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.49.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.49.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.49.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.50.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.50.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.50.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.50.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.50.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.50.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.50.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.50.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.50.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.51.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.51.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.51.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.51.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.51.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.51.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.51.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.51.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.51.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.52.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.52.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.52.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.52.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.52.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.52.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.52.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.52.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.52.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.53.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.53.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.53.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.53.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.53.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.53.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.53.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.53.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.53.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.54.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.54.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.54.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.54.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.54.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.54.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.54.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.54.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.54.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.55.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.55.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.55.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.55.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.55.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.55.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.55.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.55.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.55.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.56.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.56.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.56.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.56.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.56.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.56.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.56.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.56.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.56.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.57.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.57.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.57.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.57.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.57.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.57.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.57.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.57.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.57.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.58.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.58.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.58.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.58.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.58.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.58.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.58.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.58.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.58.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.59.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.59.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.59.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.59.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.59.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.59.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.59.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.59.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.59.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.60.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.60.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.60.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.60.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.60.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.60.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.60.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.60.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.60.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.61.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.61.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.61.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.61.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.61.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.61.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.61.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.61.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.61.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.62.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.62.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.62.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.62.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.62.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.62.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.62.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.62.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.62.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.63.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.63.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.63.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.63.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.63.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.63.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.63.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.63.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.63.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.64.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.64.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.64.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.64.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.64.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.64.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.64.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.64.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.64.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.65.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.65.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.65.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.65.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.65.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.65.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.65.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.65.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.65.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.66.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.66.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.66.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.66.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.66.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.66.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.66.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.66.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.66.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.67.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.67.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.67.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.67.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.67.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.67.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.67.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.67.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.67.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.68.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.68.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.68.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.68.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.68.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.68.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.68.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.68.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.68.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.69.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.69.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.69.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.69.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.69.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.69.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.69.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.69.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.69.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.70.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.70.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.70.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.70.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.70.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.70.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.70.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.70.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.70.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.71.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.71.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.71.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.71.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.71.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.71.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.71.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.71.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.71.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.72.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.72.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.72.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.72.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.72.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.72.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.72.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.72.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.72.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.73.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.73.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.73.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.73.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.73.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.73.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.73.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.73.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.73.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.74.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.74.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.74.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.74.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.74.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.74.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.74.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.74.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.74.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.75.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.75.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.75.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.75.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.75.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.75.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.75.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.75.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.75.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.76.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.76.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.76.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.76.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.76.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.76.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.76.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.76.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.76.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.77.self_attn.q_a_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.77.self_attn.q_b_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.77.self_attn.kv_a_proj_with_mqa": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.77.self_attn.embed_q": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.77.self_attn.unembed_out": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.77.self_attn.o_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.77.mlp.shared_experts.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.77.mlp.shared_experts.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "model.layers.77.mlp.shared_experts.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "lm_head": {
+      "group_size": 64,
+      "bits": 8
+    }
+  },
+  "model_file": "glm_moe_dsa.py"
+}
\ No newline at end of file
diff --git a/generation_config.json b/generation_config.json
new file mode 100644
index 0000000000000000000000000000000000000000..216bbb092d0df0dda1dc4cedd789b92286c4c001
--- /dev/null
+++ b/generation_config.json
@@ -0,0 +1,12 @@
+{
+  "_from_model_config": true,
+  "eos_token_id": [
+    154820,
+    154827,
+    154829
+  ],
+  "pad_token_id": 154820,
+  "temperature": 1.0,
+  "top_p": 0.95,
+  "transformers_version": "5.12.0"
+}
diff --git a/glm_moe_dsa.py b/glm_moe_dsa.py
new file mode 100644
index 0000000000000000000000000000000000000000..233864a54415a460ab55cf50db2a8e65aaf786ea
--- /dev/null
+++ b/glm_moe_dsa.py
@@ -0,0 +1,703 @@
+# Copyright © 2025 Apple Inc.
+
+import math
+from dataclasses import dataclass
+from typing import Any, Dict, Optional
+
+import mlx.core as mx
+import mlx.nn as nn
+from mlx.nn.layers.distributed import shard_inplace, shard_linear, sum_gradients
+
+from mlx_lm.models.activations import swiglu
+from mlx_lm.models.base import BaseModelArgs, create_attention_mask, scaled_dot_product_attention
+from mlx_lm.models.cache import CacheList, KVCache
+from mlx_lm.models.mla import MultiLinear
+from mlx_lm.models.rope_utils import initialize_rope
+from mlx_lm.models.switch_layers import SwitchGLU
+
+
+@dataclass
+class ModelArgs(BaseModelArgs):
+    model_type: str
+    vocab_size: int
+    hidden_size: int
+    index_head_dim: int
+    index_n_heads: int
+    index_topk: int
+    intermediate_size: int
+    moe_intermediate_size: int
+    num_hidden_layers: int
+    num_attention_heads: int
+    num_key_value_heads: int
+    n_shared_experts: Optional[int]
+    n_routed_experts: Optional[int]
+    routed_scaling_factor: float
+    kv_lora_rank: int
+    q_lora_rank: int
+    qk_rope_head_dim: int
+    v_head_dim: int
+    qk_nope_head_dim: int
+    topk_method: str
+    scoring_func: str
+    norm_topk_prob: bool
+    n_group: int
+    topk_group: int
+    num_experts_per_tok: int
+    moe_layer_freq: int
+    first_k_dense_replace: int
+    max_position_embeddings: int
+    rms_norm_eps: float
+    rope_parameters: Dict
+    attention_bias: bool
+    rope_scaling: Dict = None
+    rope_theta: Optional[float] = None
+    indexer_rope_interleave: bool = True
+    rope_interleave: bool = True
+    indexer_types: Optional[list] = None
+    index_topk_freq: int = 1
+    index_skip_topk_offset: int = 0
+    index_topk_pattern: Optional[Any] = None
+    num_nextn_predict_layers: int = 0
+    mlp_layer_types: Optional[list] = None
+
+    def __post_init__(self):
+        self.rope_scaling = self.rope_parameters
+        self.rope_theta = self.rope_parameters["rope_theta"]
+        if self.indexer_types is None:
+            # Reference schedule: a layer runs its own indexer ("full") iff
+            # max(i - offset + 1, 0) % freq == 0; the others ("shared") reuse the most recent
+            # full layer's top-k selection and carry no indexer weights.
+            f, o = self.index_topk_freq, self.index_skip_topk_offset
+            self.indexer_types = [
+                "full" if max(i - o + 1, 0) % f == 0 else "shared"
+                for i in range(self.num_hidden_layers)
+            ]
+
+
+
+class Indexer(nn.Module):
+    def __init__(self, args: ModelArgs):
+        super().__init__()
+        self.dim = args.hidden_size
+        self.n_heads = args.index_n_heads
+        self.head_dim = args.index_head_dim
+        self.rope_head_dim = args.qk_rope_head_dim
+        self.index_topk = args.index_topk
+        self.q_lora_rank = args.q_lora_rank
+        self.wq_b = nn.Linear(
+            self.q_lora_rank, self.n_heads * self.head_dim, bias=False
+        )
+        self.wk = nn.Linear(self.dim, self.head_dim, bias=False)
+        self.k_norm = nn.LayerNorm(self.head_dim, eps=1e-6)
+        self.weights_proj = nn.Linear(self.dim, self.n_heads, bias=False)
+        self.softmax_scale = self.head_dim**-0.5
+        self.rope = initialize_rope(
+            dims=args.qk_rope_head_dim,
+            base=args.rope_theta,
+            traditional=args.indexer_rope_interleave,
+            max_position_embeddings=args.max_position_embeddings,
+            scaling_config=args.rope_scaling,
+        )
+
+    def __call__(
+        self,
+        x: mx.array,
+        qr: mx.array,
+        mask: Optional[mx.array],
+        cache: Optional[Any] = None,
+    ):
+        # Computes top_k indices for attention
+        b, s, _ = x.shape
+        q = self.wq_b(qr)
+        q = q.reshape(b, s, self.n_heads, self.head_dim).swapaxes(1, 2)
+        k = self.wk(x)
+        k = self.k_norm(k)
+        k = mx.reshape(k, (b, 1, s, self.head_dim))
+
+        offset = cache.offset if cache is not None else 0
+
+        q = self.rope(q, offset=offset)
+        k = self.rope(k, offset=offset)
+
+        if cache is not None:
+            k, _ = cache.update_and_fetch(k, mx.zeros([b, 1, s, 0]))
+        if k.shape[2] <= self.index_topk:
+            return None
+        # Scores in float32, as the reference (which also keeps weights_proj in fp32): in bf16
+        # keys at the top-k boundary flip.
+        scores = q.astype(mx.float32) @ k.astype(mx.float32).swapaxes(-1, -2)
+        scores = mx.maximum(scores, 0)
+        weights = self.weights_proj(x).astype(mx.float32) * (self.n_heads**-0.5 * self.softmax_scale)
+        weights = weights.swapaxes(-1, -2)[..., None]
+        scores = scores * weights
+        scores = scores.sum(axis=1, keepdims=True)
+        if mask is not None:
+            scores = mx.where(mask, scores, -float("inf"))
+        return mx.argpartition(scores, kth=-self.index_topk, axis=-1)[
+            ..., -self.index_topk :
+        ]
+
+
+class DeepseekV32Attention(nn.Module):
+    def __init__(self, config: ModelArgs, layer_idx: int = 0):
+        super().__init__()
+        self.config = config
+        self.hidden_size = config.hidden_size
+        self.num_heads = config.num_attention_heads
+        self.max_position_embeddings = config.max_position_embeddings
+        self.rope_theta = config.rope_theta
+        self.q_lora_rank = config.q_lora_rank
+        self.qk_rope_head_dim = config.qk_rope_head_dim
+        self.kv_lora_rank = config.kv_lora_rank
+        self.v_head_dim = config.v_head_dim
+        self.qk_nope_head_dim = config.qk_nope_head_dim
+        self.q_head_dim = config.qk_nope_head_dim + config.qk_rope_head_dim
+
+        self.scale = self.q_head_dim**-0.5
+
+        self.q_a_proj = nn.Linear(
+            self.hidden_size, self.q_lora_rank, bias=config.attention_bias
+        )
+        self.q_a_layernorm = nn.RMSNorm(self.q_lora_rank, eps=1e-6)
+        self.q_b_proj = nn.Linear(
+            self.q_lora_rank, self.num_heads * self.q_head_dim, bias=False
+        )
+
+        self.kv_a_proj_with_mqa = nn.Linear(
+            self.hidden_size,
+            self.kv_lora_rank + self.qk_rope_head_dim,
+            bias=config.attention_bias,
+        )
+        self.kv_a_layernorm = nn.RMSNorm(self.kv_lora_rank, eps=1e-6)
+        self.embed_q = MultiLinear(
+            self.qk_nope_head_dim, self.kv_lora_rank, self.num_heads
+        )
+        self.unembed_out = MultiLinear(
+            self.kv_lora_rank, self.v_head_dim, self.num_heads
+        )
+
+        self.o_proj = nn.Linear(
+            self.num_heads * self.v_head_dim,
+            self.hidden_size,
+            bias=config.attention_bias,
+        )
+
+        if self.config.rope_scaling is not None:
+            mscale_all_dim = self.config.rope_scaling.get("mscale_all_dim", 0)
+            if mscale_all_dim:
+                scaling_factor = self.config.rope_scaling["factor"]
+                if scaling_factor > 1:
+                    s = 0.1 * mscale_all_dim * math.log(scaling_factor) + 1.0
+                    self.scale = self.scale * s * s
+
+        # "shared" layers carry no indexer weights: they reuse the previous full layer's top-k.
+        self.indexer = Indexer(config) if config.indexer_types[layer_idx] == "full" else None
+        self.rope = initialize_rope(
+            dims=self.qk_rope_head_dim,
+            base=self.rope_theta,
+            traditional=True,
+            max_position_embeddings=self.max_position_embeddings,
+            scaling_config=self.config.rope_scaling,
+        )
+
+    def __call__(
+        self,
+        x: mx.array,
+        mask: Optional[mx.array] = None,
+        cache: Optional[Any] = None,
+        prev_topk_indices: Optional[mx.array] = None,
+    ):
+        B, L, D = x.shape
+
+        qr = self.q_a_layernorm(self.q_a_proj(x))
+        q = self.q_b_proj(qr)
+
+        q = q.reshape(B, L, self.num_heads, self.q_head_dim).transpose(0, 2, 1, 3)
+        q_nope, q_pe = mx.split(q, [self.qk_nope_head_dim], axis=-1)
+        compressed_kv = self.kv_a_proj_with_mqa(x)
+        compressed_kv, k_pe = mx.split(compressed_kv, [self.kv_lora_rank], axis=-1)
+        k_pe = k_pe.reshape(B, L, 1, self.qk_rope_head_dim).transpose(0, 2, 1, 3)
+        kv_latent = self.kv_a_layernorm(compressed_kv)
+
+        offset = cache[0].offset if cache is not None else 0
+        q_pe = self.rope(q_pe, offset)
+        k_pe = self.rope(k_pe, offset)
+
+        kv_latent = mx.expand_dims(kv_latent, axis=1)
+
+        if cache is not None:
+            kv_latent, k_pe = cache[0].update_and_fetch(kv_latent, k_pe)
+        else:
+            cache = [None] * 2
+
+        if self.indexer is not None:
+            topk_indices = self.indexer(x, qr, mask, cache=cache[1])
+        else:
+            topk_indices = prev_topk_indices
+        if topk_indices is not None:
+            if L == 1:
+                idx = topk_indices[:, :, 0, :, None]
+                kv_latent = mx.take_along_axis(
+                    kv_latent,
+                    mx.broadcast_to(idx, idx.shape[:-1] + (kv_latent.shape[-1],)),
+                    axis=2,
+                )
+                k_pe = mx.take_along_axis(
+                    k_pe,
+                    mx.broadcast_to(idx, idx.shape[:-1] + (k_pe.shape[-1],)),
+                    axis=2,
+                )
+                if mask is not None:
+                    mask = mx.take_along_axis(mask, topk_indices, axis=-1)
+            else:
+                shape = list(topk_indices.shape)
+                shape[-1] = kv_latent.shape[2]
+                sparse_mask = mx.zeros(shape, dtype=mx.bool_)
+                sparse_mask = mx.put_along_axis(
+                    sparse_mask, topk_indices, mx.array(True), axis=-1
+                )
+                if mask is not None:
+                    sparse_mask = sparse_mask & mask
+                mask = sparse_mask
+        # Ensure the indexer cache is evaluated even if the topk_indices are unused
+        # to keep the graph from getting too large
+        if self.indexer is not None and cache is not None and cache[0] is not None:
+            cache[0].keys = mx.depends(cache[0].keys, (cache[1].keys, cache[1].values))
+
+        pe_scores = (q_pe * self.scale) @ k_pe.swapaxes(-1, -2)
+        if mask is not None:
+            pe_scores = mx.where(
+                mask,
+                pe_scores,
+                mx.array(mx.finfo(pe_scores.dtype).min, pe_scores.dtype),
+            )
+
+        if L == 1:
+            q_nope = self.embed_q(q_nope)
+            k = v = kv_latent
+        else:
+            k = self.embed_q(kv_latent, transpose=False)
+            v = self.unembed_out(kv_latent)
+
+        output = scaled_dot_product_attention(
+            q_nope, k, v, cache=cache, scale=self.scale, mask=pe_scores
+        )
+        if L == 1:
+            output = self.unembed_out(output)
+
+        output = output.transpose(0, 2, 1, 3).reshape(B, L, -1)
+        return self.o_proj(output), topk_indices
+
+
+class DeepseekV32MLP(nn.Module):
+    def __init__(
+        self, config: ModelArgs, hidden_size: int = None, intermediate_size: int = None
+    ):
+        super().__init__()
+        self.config = config
+        self.hidden_size = config.hidden_size if hidden_size is None else hidden_size
+        self.intermediate_size = (
+            config.intermediate_size if intermediate_size is None else intermediate_size
+        )
+
+        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
+        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
+        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
+
+    def __call__(self, x):
+        down_proj = self.down_proj(swiglu(self.gate_proj(x), self.up_proj(x)))
+        return down_proj
+
+
+@mx.compile
+def group_expert_select(
+    gates,
+    e_score_correction_bias,
+    top_k,
+    n_group,
+    topk_group,
+    routed_scaling_factor,
+    norm_topk_prob,
+):
+
+    scores = mx.sigmoid(gates.astype(mx.float32))
+    orig_scores = scores
+    scores = scores + e_score_correction_bias
+    if n_group > 1:
+        scores = mx.unflatten(scores, axis=-1, shape=(n_group, -1))
+        group_scores = mx.topk(scores, 2, axis=-1).sum(axis=-1, keepdims=True)
+        k = n_group - topk_group
+        group_idx = mx.argpartition(group_scores, kth=k - 1, axis=-2)[..., :k, :]
+        scores = mx.put_along_axis(
+            scores, mx.stop_gradient(group_idx), mx.array(0.0), axis=-2
+        )
+        scores = mx.flatten(scores, -2, -1)
+
+    k = top_k
+    inds = mx.argpartition(-scores, kth=k - 1, axis=-1)[..., :k]
+    scores = mx.take_along_axis(orig_scores, inds, axis=-1)
+    if top_k > 1 and norm_topk_prob:
+        denominator = scores.sum(axis=-1, keepdims=True)
+        scores = scores / denominator
+    scores = scores * routed_scaling_factor
+
+    return inds, scores
+
+
+class MoEGate(nn.Module):
+    def __init__(self, config: ModelArgs):
+        super().__init__()
+        self.config = config
+        self.top_k = config.num_experts_per_tok
+        self.norm_topk_prob = config.norm_topk_prob
+        self.n_routed_experts = config.n_routed_experts
+        self.routed_scaling_factor = config.routed_scaling_factor
+        self.n_group = config.n_group
+        self.topk_group = config.topk_group
+        self.weight = mx.zeros((self.n_routed_experts, config.hidden_size))
+        self.e_score_correction_bias = mx.zeros((self.n_routed_experts,))
+        assert config.topk_method == "noaux_tc", "Unsupported topk method."
+
+    def __call__(self, x):
+        return group_expert_select(
+            x.astype(mx.float32) @ self.weight.astype(mx.float32).T,
+            self.e_score_correction_bias,
+            self.top_k,
+            self.n_group,
+            self.topk_group,
+            self.routed_scaling_factor,
+            self.norm_topk_prob,
+        )
+
+
+class DeepseekV32MoE(nn.Module):
+    def __init__(self, config: ModelArgs):
+        super().__init__()
+        self.config = config
+        self.num_experts_per_tok = config.num_experts_per_tok
+        self.switch_mlp = SwitchGLU(
+            config.hidden_size,
+            config.moe_intermediate_size,
+            config.n_routed_experts,
+        )
+
+        self.gate = MoEGate(config)
+        if config.n_shared_experts is not None:
+            intermediate_size = config.moe_intermediate_size * config.n_shared_experts
+            self.shared_experts = DeepseekV32MLP(
+                config=config, intermediate_size=intermediate_size
+            )
+
+        self.sharding_group = None
+
+    def __call__(self, x):
+        if self.sharding_group is not None:
+            x = sum_gradients(self.sharding_group)(x)
+
+        inds, scores = self.gate(x)
+        y = self.switch_mlp(x, inds)
+        y = (y * scores[..., None]).sum(axis=-2).astype(y.dtype)
+        if self.config.n_shared_experts is not None:
+            y = y + self.shared_experts(x)
+
+        if self.sharding_group is not None:
+            y = mx.distributed.all_sum(y, group=self.sharding_group)
+
+        return y
+
+
+class DeepseekV32DecoderLayer(nn.Module):
+    def __init__(self, config: ModelArgs, layer_idx: int):
+        super().__init__()
+        self.self_attn = DeepseekV32Attention(config, layer_idx)
+        self.mlp = (
+            DeepseekV32MoE(config)
+            if (
+                config.n_routed_experts is not None
+                and layer_idx >= config.first_k_dense_replace
+                and layer_idx % config.moe_layer_freq == 0
+            )
+            else DeepseekV32MLP(config)
+        )
+        self.input_layernorm = nn.RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.post_attention_layernorm = nn.RMSNorm(
+            config.hidden_size, eps=config.rms_norm_eps
+        )
+
+    def __call__(
+        self,
+        x: mx.array,
+        mask: Optional[mx.array] = None,
+        cache: Optional[Any] = None,
+        prev_topk_indices: Optional[mx.array] = None,
+    ):
+        r, topk_indices = self.self_attn(self.input_layernorm(x), mask, cache, prev_topk_indices)
+        h = x + r
+        r = self.mlp(self.post_attention_layernorm(h))
+        return h + r, topk_indices
+
+
+class DeepseekV32Model(nn.Module):
+    def __init__(self, config: ModelArgs):
+        super().__init__()
+        self.vocab_size = config.vocab_size
+        self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size)
+        self.layers = [
+            DeepseekV32DecoderLayer(config, idx)
+            for idx in range(config.num_hidden_layers)
+        ]
+        self.start_idx = 0
+        self.end_idx = len(self.layers)
+        self.num_layers = self.end_idx
+
+        self.norm = nn.RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.pipeline_rank = 0
+        self.pipeline_size = 1
+
+    def pipeline(self, group):
+        # Split layers in reverse so rank=0 gets the last layers and
+        # rank=pipeline_size-1 gets the first
+        self.pipeline_rank = group.rank()
+        self.pipeline_size = group.size()
+        layers_per_rank = len(self.layers) // self.pipeline_size
+        extra = len(self.layers) - layers_per_rank * self.pipeline_size
+        if self.pipeline_rank < extra:
+            layers_per_rank += 1
+        self.start_idx = (self.pipeline_size - self.pipeline_rank - 1) * layers_per_rank
+        self.end_idx = self.start_idx + layers_per_rank
+        self.layers = self.layers[: self.end_idx]
+        self.layers[: self.start_idx] = [None] * self.start_idx
+        self.num_layers = len(self.layers) - self.start_idx
+
+    def __call__(
+        self,
+        x: mx.array,
+        cache: Optional[Any] = None,
+    ) -> mx.array:
+        h = self.embed_tokens(x)
+
+        pipeline_rank = self.pipeline_rank
+        pipeline_size = self.pipeline_size
+
+        if cache is None:
+            cache = [None] * self.num_layers
+        mask = create_attention_mask(
+            h, cache[0][0] if cache[0] else None, return_array=True
+        )
+
+        # Receive from the previous process in the pipeline
+
+        if pipeline_rank < pipeline_size - 1:
+            h = mx.distributed.recv_like(h, (pipeline_rank + 1))
+
+        # A "shared" layer reuses the top-k selection of the most recent "full" layer.
+        topk = None
+        for i in range(self.num_layers):
+            h, topk = self.layers[self.start_idx + i](h, mask, cache[i], topk)
+
+        # Send to the next process in the pipeline
+        if pipeline_rank != 0:
+            h = mx.distributed.send(h, (pipeline_rank - 1) % pipeline_size)
+            if cache[-1] is not None:
+                cache[-1][0].keys = mx.depends(cache[-1][0].keys, h)
+
+        # Broadcast h while keeping it in the graph
+        if pipeline_size > 1:
+            h = mx.distributed.all_gather(h)[: h.shape[0]]
+
+        return self.norm(h)
+
+
+class Model(nn.Module):
+    def __init__(self, config: ModelArgs):
+        super().__init__()
+        self.args = config
+        self.model_type = config.model_type
+        self.model = DeepseekV32Model(config)
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+
+    def __call__(
+        self,
+        inputs: mx.array,
+        cache: Optional[Any] = None,
+    ):
+        out = self.model(inputs, cache)
+        return self.lm_head(out)
+
+    def sanitize(self, weights):
+        # Remove multi-token prediction layers
+        mpt_layer = self.args.num_hidden_layers
+        new_weights = {}
+        for k, v in weights.items():
+            parts = k.split(".")
+            if len(parts) >= 3 and parts[1] == "layers" and int(parts[2]) >= mpt_layer:
+                continue
+            new_weights[k] = v
+        weights = new_weights
+
+        def dequant(weight, scale_inv):
+            dtype = mx.bfloat16
+            weight = mx.from_fp8(weight, dtype=mx.bfloat16)
+            bs = 128  # block size
+            m, n = weight.shape
+            pad_bottom = (-m) % bs
+            pad_side = (-n) % bs
+            weight = mx.pad(weight, ((0, pad_bottom), (0, pad_side)))
+            weight = weight.reshape(
+                ((m + pad_bottom) // bs, bs, (n + pad_side) // bs, bs)
+            )
+            weight = (weight * scale_inv[:, None, :, None]).reshape(
+                m + pad_bottom, n + pad_side
+            )
+            return weight[:m, :n].astype(dtype)
+
+        # Dequantize
+        new_weights = {}
+        for k, v in weights.items():
+            if "weight_scale_inv" in k:
+                scale_inv = v
+                wk = k.replace("_scale_inv", "")
+                weight = weights[wk]
+                weight = dequant(weight, scale_inv)
+                new_weights[wk] = weight
+            elif k not in new_weights:
+                new_weights[k] = v
+        weights = new_weights
+
+        # Stack experts
+        for l in range(self.args.num_hidden_layers):
+            prefix = f"model.layers.{l}"
+            for n, m in [("w1", "gate_proj"), ("w2", "down_proj"), ("w3", "up_proj")]:
+                for k in ["weight", "scales", "biases"]:
+                    if f"{prefix}.mlp.experts.0.{m}.{k}" in weights:
+                        to_join = [
+                            weights.pop(f"{prefix}.mlp.experts.{e}.{m}.{k}")
+                            for e in range(self.args.n_routed_experts)
+                        ]
+                        weights[f"{prefix}.mlp.switch_mlp.{m}.{k}"] = mx.stack(to_join)
+            prefix = f"model.layers.{l}.self_attn"
+            if f"{prefix}.kv_b_proj.weight" in weights:
+                layer = self.model.layers[l].self_attn.embed_q
+                quantized = f"{prefix}.kv_b_proj.scales" in weights
+                v = weights.pop(f"{prefix}.kv_b_proj.weight")
+                head_dim = self.args.qk_nope_head_dim + self.args.v_head_dim
+
+                if quantized:
+                    dims = self.args.kv_lora_rank
+                    scales = weights.pop(f"{prefix}.kv_b_proj.scales")
+                    biases = weights.pop(f"{prefix}.kv_b_proj.biases")
+                    # Try to infer bits and group size
+                    bits = (v.shape[-1] * 32) // dims
+                    group_size = dims // scales.shape[-1]
+                    v = mx.dequantize(
+                        v, scales, biases, bits=bits, group_size=group_size
+                    )
+                num_heads = self.args.num_attention_heads
+                v = v.reshape(num_heads, head_dim, -1)
+                wk = mx.contiguous(
+                    v[:, : self.args.qk_nope_head_dim, :].swapaxes(-1, -2)
+                )
+                wv = mx.contiguous(v[:, self.args.qk_nope_head_dim :, :])
+                if quantized:
+                    wk, wk_scales, wk_biases = mx.quantize(
+                        wk, bits=bits, group_size=group_size
+                    )
+                    wv, wv_scales, wv_biases = mx.quantize(
+                        wv, bits=bits, group_size=group_size
+                    )
+                    weights[f"{prefix}.embed_q.scales"] = wk_scales
+                    weights[f"{prefix}.unembed_out.scales"] = wv_scales
+                    weights[f"{prefix}.embed_q.biases"] = wk_biases
+                    weights[f"{prefix}.unembed_out.biases"] = wv_biases
+                weights[f"{prefix}.embed_q.weight"] = wk
+                weights[f"{prefix}.unembed_out.weight"] = wv
+
+        return weights
+
+    def shard(self, group: Optional[mx.distributed.Group] = None):
+        group = group or mx.distributed.init()
+        N = group.size()
+        rank = group.rank()
+        for layer in self.model.layers:
+            layer.self_attn.q_b_proj = shard_linear(
+                layer.self_attn.q_b_proj, "all-to-sharded", group=group
+            )
+
+            layer.self_attn.o_proj = shard_linear(
+                layer.self_attn.o_proj, "sharded-to-all", group=group
+            )
+            layer.self_attn.num_heads //= N
+            num_heads = layer.self_attn.num_heads
+            sh = rank * num_heads
+            eh = sh + num_heads
+
+            def shard_heads(w):
+                return w[sh:eh]
+
+            layer.self_attn.embed_q.apply(shard_heads)
+            layer.self_attn.unembed_out.apply(shard_heads)
+
+            # Shard the MLP
+            if isinstance(layer.mlp, DeepseekV32MLP):
+                layer.mlp.gate_proj = shard_linear(
+                    layer.mlp.gate_proj, "all-to-sharded", group=group
+                )
+                layer.mlp.down_proj = shard_linear(
+                    layer.mlp.down_proj, "sharded-to-all", group=group
+                )
+                layer.mlp.up_proj = shard_linear(
+                    layer.mlp.up_proj, "all-to-sharded", group=group
+                )
+
+            # Shard the MoE. Shard in place since the MoE should be responsible
+            # for aggregating the results.
+            else:
+                layer.mlp.sharding_group = group = group
+                shard_inplace(
+                    layer.mlp.shared_experts.gate_proj, "all-to-sharded", group=group
+                )
+                shard_inplace(
+                    layer.mlp.shared_experts.down_proj, "sharded-to-all", group=group
+                )
+                shard_inplace(
+                    layer.mlp.shared_experts.up_proj, "all-to-sharded", group=group
+                )
+                shard_inplace(
+                    layer.mlp.switch_mlp.gate_proj, "all-to-sharded", group=group
+                )
+                shard_inplace(
+                    layer.mlp.switch_mlp.down_proj, "sharded-to-all", group=group
+                )
+                shard_inplace(
+                    layer.mlp.switch_mlp.up_proj, "all-to-sharded", group=group
+                )
+
+    @property
+    def layers(self):
+        return self.model.layers[self.model.start_idx : self.model.end_idx]
+
+    @property
+    def cast_predicate(self):
+        def predicate(k):
+            return "e_score_correction_bias" not in k
+
+        return predicate
+
+    @property
+    def quant_predicate(self):
+        def predicate(path, module):
+            # The lightning indexer (0.03% of parameters, reference keeps weights_proj in fp32) and the
+            # MoE router decide *what* is computed; keep them as stored.
+            if ".indexer." in path or path.endswith("mlp.gate"):
+                return False
+            return True
+
+        return predicate
+
+    def make_cache(self):
+        # Shared layers have no indexer, so no indexer key cache: an empty second KVCache would
+        # trip mlx-lm's `cache.state` evaluation (keys is None) during generation.
+        return [
+            CacheList(KVCache(), KVCache()) if layer.self_attn.indexer is not None else CacheList(KVCache())
+            for layer in self.layers
+        ]
diff --git a/model-00001.safetensors b/model-00001.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..5b7adf18476439a41934ccc54b1bbf562521cfe7
--- /dev/null
+++ b/model-00001.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cb52ed958f4afddcc845718cc20f60e6b96e03d3d305e0764b6faf4c3a252438
+size 2022126266
diff --git a/model-00002.safetensors b/model-00002.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..053d9a47abb4d421725da9aacbb0dba33243faba
--- /dev/null
+++ b/model-00002.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cdb4e5479a2a24ef0d660b9a7bd6c74599bc700ff2d33230711fcdc459758451
+size 434759225
diff --git a/model-00003.safetensors b/model-00003.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..74c7426b145cdfe204c7acb296c773f328a8c96f
--- /dev/null
+++ b/model-00003.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:038ed2e8f41354037a5d295398f3364e63df1f9cd9d7b488b530bb0ed77d1fb7
+size 434759229
diff --git a/model-00004.safetensors b/model-00004.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..c9affe66029495321cdd87b3de474472ea355e50
--- /dev/null
+++ b/model-00004.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b724d7ebb3f6b4e3b7498f8e9baacd5d2530087f8f1ef0046a540f81c7fcdbf
+size 434759239
diff --git a/model-00005.safetensors b/model-00005.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..af40608447f803f83f0a5acc0ac08ad457752d81
--- /dev/null
+++ b/model-00005.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0a0e43a328b1a3d098a5ba4b9952896673aafbe2f1aaa589c0a9dd99336c735c
+size 5654440972
diff --git a/model-00006.safetensors b/model-00006.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..d91001cb69e9e96b770f7d46b45af02c24b9e3e6
--- /dev/null
+++ b/model-00006.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a08642a17e3c3cce2793e4ecae7767cf7591b8da1d23316ae21bbfde55ab9d0c
+size 5654440950
diff --git a/model-00007.safetensors b/model-00007.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..6f7769efe150d274eb0a5c5bdb25d5f1e8b15a28
--- /dev/null
+++ b/model-00007.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:01ebd833aec9ce5a335bc4b19a52403a75d981b7f6cc9ad25c8a1b0bb654db24
+size 5654440966
diff --git a/model-00008.safetensors b/model-00008.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..e9eabcb8b71a4952169ff352502ce1f80a8bd141
--- /dev/null
+++ b/model-00008.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b959fb6215865a7b5e9f718a1610086fa9e7344a250213dda1220d0e0890ce71
+size 5673185378
diff --git a/model-00009.safetensors b/model-00009.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..9824a80248a036d18aec0d059623e7ba32dddcd2
--- /dev/null
+++ b/model-00009.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:03c7b0257fbe6051f172ad7fa57e53dd8578b36242b23f310f9f274def1e173b
+size 5654440976
diff --git a/model-00010.safetensors b/model-00010.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..cf37ef43642ab9a0477baf2e5284c1d982f53922
--- /dev/null
+++ b/model-00010.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ca66123082a599d1c7d16f9105a7936d9d5df4ed7abf5fdfb62377fd6380ec30
+size 5654440928
diff --git a/model-00011.safetensors b/model-00011.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..e1841c223865ec71911d3ea85d4a69f2595459f6
--- /dev/null
+++ b/model-00011.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:62bff009c28b462e6551fbb5d5e7c2a4fcca973eb220aa3f56d8277819a1427b
+size 5654440962
diff --git a/model-00012.safetensors b/model-00012.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..437a182dc72d71ffdeca95911717ad78a9065c32
--- /dev/null
+++ b/model-00012.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9a933c88743bb6218888f30beda5d79a4a55221e040829df5855552992e12328
+size 5673185423
diff --git a/model-00013.safetensors b/model-00013.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..91c673b0da8240ed9358ea2c347312c2c628de1c
--- /dev/null
+++ b/model-00013.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:710cb3cfe8b3592f0cf1d719ef8b000081396d31faad1f6428cb1875e374fdc9
+size 5654441016
diff --git a/model-00014.safetensors b/model-00014.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..7912ddd7908be4346bd904cac3b3c3a1ab7f4071
--- /dev/null
+++ b/model-00014.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:375cfecd56d0ccc3ef3883b1c857589247558123e302f4f83c83a89ad492f877
+size 5654440962
diff --git a/model-00015.safetensors b/model-00015.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..0777ffd45427248c6588babedc4687316de959a5
--- /dev/null
+++ b/model-00015.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:47e8e725c2fad4aa2888561efe3bc56749ed5ad577145be4626546250fe3810b
+size 5654441020
diff --git a/model-00016.safetensors b/model-00016.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..86e127cd08d824be7e68136a3bb1979d3322b22a
--- /dev/null
+++ b/model-00016.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7c5b185352ef2e5d24b1b7f7f2276ec05e761a5881c4ed150c6a9d71dfe468e
+size 5673185429
diff --git a/model-00017.safetensors b/model-00017.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..eb821798e124747ce261b7cfc31dbe014445adbf
--- /dev/null
+++ b/model-00017.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:74f50dbf7ec9d50442464607ff6b746641783930c8e667d43e1a66568c706d46
+size 5654441006
diff --git a/model-00018.safetensors b/model-00018.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..981a532b86286dcbdfa76592920b2776c17c9284
--- /dev/null
+++ b/model-00018.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d027e90c9095860cb2118294d9a0817165cb1d59754a8c23ff23949a08ba934a
+size 5654441016
diff --git a/model-00019.safetensors b/model-00019.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..7281edfd76147f2fcf86925a35201a999465febf
--- /dev/null
+++ b/model-00019.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:83571e3805670fd6a82e5f7f0dc4322b2e741e20e81ec15dfe33e008b4bb405d
+size 5654440992
diff --git a/model-00020.safetensors b/model-00020.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..1b307df32ee43a4225b932644312a85bc3370f38
--- /dev/null
+++ b/model-00020.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2296278bb1aac80e5ec09407666ad462058bb5dec01f74139975beab824ea9a9
+size 5673185441
diff --git a/model-00021.safetensors b/model-00021.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..6bf5e5c4724920f4feb290a2c5970eab6bda7c85
--- /dev/null
+++ b/model-00021.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:47076d3b43359046543509cef7de8e874cd0dc2948123ae5e350474900859d1a
+size 5654441014
diff --git a/model-00022.safetensors b/model-00022.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..121b27b07c030f1dea43fe0e23cbc90d751a8518
--- /dev/null
+++ b/model-00022.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:519f3f1e08e40f7a5886ce19c194be58b7e99bfdafb6b7085d840841748f0c48
+size 5654441020
diff --git a/model-00023.safetensors b/model-00023.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..d5b6c2250ae8cffface9ae277073fd834a81f0c6
--- /dev/null
+++ b/model-00023.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:63fdf17e4e154d90a14d02f7f3bee27abf778b37c52b37ce7f94b1ae5eb893c1
+size 5654441002
diff --git a/model-00024.safetensors b/model-00024.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..943e2672d947b3299d14f7dd80dc6fb81c4a819b
--- /dev/null
+++ b/model-00024.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8c814700b592587cdbd4fd68217231d6064c3803b3dd35dde5f06032118a6487
+size 5673185439
diff --git a/model-00025.safetensors b/model-00025.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..cbee05641c7617b53a0e3efb3cd8c95bec2aec7c
--- /dev/null
+++ b/model-00025.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:579ebe7d8a8058a980fc89d1f1efd7dca80a079e0adf5fd1e88a8d4a8b32a5c7
+size 5654441014
diff --git a/model-00026.safetensors b/model-00026.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..ddb928241fc8db26e2e0609de8dcaac357e05761
--- /dev/null
+++ b/model-00026.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:74fd5cd44af12367b733631d5d0615cbb4f7292000f48372514941bf9cb809aa
+size 5654441024
diff --git a/model-00027.safetensors b/model-00027.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..6c4d9078536ede6faa66c920b71dea1ca1981fed
--- /dev/null
+++ b/model-00027.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a35ef3f8ee4ab9d388441674058a7791688aec44e556c22c26b7882ffcdc8e4a
+size 5654441016
diff --git a/model-00028.safetensors b/model-00028.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..863dfeaafeb5b58e59a82554964ac14e91fa2447
--- /dev/null
+++ b/model-00028.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b6af5c807c0fb6e8749d541cbfdbf699d96b4dbe956679ac1dc8208ff434474d
+size 5673185435
diff --git a/model-00029.safetensors b/model-00029.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..3ad32cca873653163f7df0ef83b246c8bac94ace
--- /dev/null
+++ b/model-00029.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ac1f0d604df14ebbbd7a1ba26e670b8900a5305e4cf6e34d7609c7e8e6223ac
+size 5654441002
diff --git a/model-00030.safetensors b/model-00030.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..12f2ae2ac3f194762e8815edb7bca77662b70067
--- /dev/null
+++ b/model-00030.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d73dc52a50b7edf041538776ace79fd2f26ec6732112a60cf87f01ab1b377e4e
+size 5654441012
diff --git a/model-00031.safetensors b/model-00031.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..6f2e508ab454647466037dd9ebe97a9b18c60eef
--- /dev/null
+++ b/model-00031.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6fc71222ec47a3b8f259c83bb1896576ea73047d6db61815b825a0531799310a
+size 5654441002
diff --git a/model-00032.safetensors b/model-00032.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..0398ae845b8307443a99395135ece3e399a0d1d8
--- /dev/null
+++ b/model-00032.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:348448460d5be4ccbefd13c85e2b27a75b0034a2c58e50d06491d40b913bba68
+size 5673185427
diff --git a/model-00033.safetensors b/model-00033.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..88fb06e3a8bc318e55f55016a46e02a25abc439d
--- /dev/null
+++ b/model-00033.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5a9ec5931d985198807b1a17e4b09c54b562522da5f7310548a1e16aff167434
+size 5654441014
diff --git a/model-00034.safetensors b/model-00034.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..e430f13e82b9b0a6012ae4a249c5347db082b247
--- /dev/null
+++ b/model-00034.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e417db5b84fb9b74eca74f8915b646cfd046507d6d470fd086e9513a41b275d
+size 5654441020
diff --git a/model-00035.safetensors b/model-00035.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..9618d5c1e38633f2e41ce90854a540525f4a77cf
--- /dev/null
+++ b/model-00035.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c82bfe7ef1ff7cd5d779ea7cc0515bf0c28f4f8929fbdc48f8adbf939cbe7691
+size 5654441018
diff --git a/model-00036.safetensors b/model-00036.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..d09a3e9d7bd909ca8cf3bf9925130823e149fac3
--- /dev/null
+++ b/model-00036.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7ed84c9bbd13aef0fe6e848aa1fb3908688325b5765f6ba4f06aa220a7b16c4
+size 5673185427
diff --git a/model-00037.safetensors b/model-00037.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..9d04c88ac6e8553f4e68423b372c90eaeae5efe6
--- /dev/null
+++ b/model-00037.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4dc8bc6a2674015de66fe3c71651eeac5eb2f4b14613e368388a60721ab490f0
+size 5654441008
diff --git a/model-00038.safetensors b/model-00038.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..42f3bf0aed3eb18f54224579c1d6492f35c5237e
--- /dev/null
+++ b/model-00038.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0569d743258eae09e1613686d14addcced00d8a1786951bcf0b776fb6311b387
+size 5654441002
diff --git a/model-00039.safetensors b/model-00039.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..71e14e79971b0fda6e9ccd683522a99d2762dc04
--- /dev/null
+++ b/model-00039.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0cc9cb0fb26587287f962851278f5e52b386d95706d0793d3a5d78f3027c35d6
+size 5654441006
diff --git a/model-00040.safetensors b/model-00040.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..e106c95a01d2e552e8545ff964a329583790984a
--- /dev/null
+++ b/model-00040.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b2602f6d150c1514683bf60dd272220ce4e7718aab9821288d4c0e7cf9fdace8
+size 5673185431
diff --git a/model-00041.safetensors b/model-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..d1fbbe91a6506e628d9c33e8807ba9efde25230a
--- /dev/null
+++ b/model-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b73be78f56af8f4201f679ae1b825ebb571668dce9afd536b0665b2b57b230b1
+size 5654441026
diff --git a/model-00042.safetensors b/model-00042.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..575f26a333234f87da6f63edee90772b65231511
--- /dev/null
+++ b/model-00042.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eec2d3de8f633ef6f709d2a52e16e99ffe9f8a12d77242c0e058b190091333e3
+size 5654441012
diff --git a/model-00043.safetensors b/model-00043.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..e394a033940c09577d687642929452cab85283c1
--- /dev/null
+++ b/model-00043.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9a6301afe679cac1a20278c2415d2a9508cb0f8ee647282420d934eb7c1c4660
+size 5654441014
diff --git a/model-00044.safetensors b/model-00044.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..0d85e79d550f8ed7df5fe6dcc7f5264cfbedfa03
--- /dev/null
+++ b/model-00044.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6009ff8da7befacc808aae0a53320a2ae27e4ba3667ce67a15d0d5401bc4dda6
+size 5673185415
diff --git a/model-00045.safetensors b/model-00045.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..e7cf2447bd95bf8e47c406997ad1f54401fe6f8e
--- /dev/null
+++ b/model-00045.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:29012a11ef1d3a710eb77b1ff9fc417e4674268c2e1dc1e9eca28b070bbfb884
+size 5654441012
diff --git a/model-00046.safetensors b/model-00046.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..fa66f837c7f3de55b5438a6451926a26a8928eab
--- /dev/null
+++ b/model-00046.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f670d408199acfa37c982eca956e369f70c6bbe1ff3e6fb4b290d6a543c9d031
+size 5654441004
diff --git a/model-00047.safetensors b/model-00047.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..638fa0091a2892ff03054d27c1bb88681e5ddd10
--- /dev/null
+++ b/model-00047.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0f6a8e28e8ee644a6572faa9ae2b79a3c8acdca6d0f618f2a8ce753016805ecc
+size 5654441010
diff --git a/model-00048.safetensors b/model-00048.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..eac0fca515fd51aa7fda4f32a02a5ffce177f1ee
--- /dev/null
+++ b/model-00048.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:89bdc71e969a817e2a17ee37a5a605ba677100f54ffd408e37d8cbed49cff9c8
+size 5673185435
diff --git a/model-00049.safetensors b/model-00049.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..ae2da98d5c51bfc2c34f3538ad8596fcaa6df733
--- /dev/null
+++ b/model-00049.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:32deff9daa44832aea14bf00cad1fc3d7f0933e34a7cfc7a983656bdcb6fa077
+size 5654441014
diff --git a/model-00050.safetensors b/model-00050.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..ee10ecbafc47dac60359dd269dcc10ae4fb05d97
--- /dev/null
+++ b/model-00050.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2fe0637a2e32910b14950dd794729e347cc1d9ea23f8186a1a576c3946ee9c43
+size 5654441018
diff --git a/model-00051.safetensors b/model-00051.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..3efe008cbb1b995656dd71f325603c9b29c805e3
--- /dev/null
+++ b/model-00051.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d9e84afb9343bed2819359718a06e40a3d426494cb785ecfce293dab3ec4c8be
+size 5654440994
diff --git a/model-00052.safetensors b/model-00052.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..b226569d639d2bc9ecc2fcebdcb09e62b2ace2a4
--- /dev/null
+++ b/model-00052.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:af4ea9cb8b942394b83dfb0bb62a927cdc27369a1328572c16bdf93d3f5bbe16
+size 5673185437
diff --git a/model-00053.safetensors b/model-00053.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..2b763111284f8ac54bb1443f8edcfb9e244f3c68
--- /dev/null
+++ b/model-00053.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:33ad5630273210ebff75378e2b1bf989bcf9ec04e515d48970085664fc276878
+size 5654441006
diff --git a/model-00054.safetensors b/model-00054.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..80846718abbc400d85f85fc2488e6e5c38ee9447
--- /dev/null
+++ b/model-00054.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bc487b9e749c0f973288b054244050da99324874dab7c78bb98cf59dc3ade0e2
+size 5654441010
diff --git a/model-00055.safetensors b/model-00055.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..0dfa178cb8a3d97d373d40133fa18fa01d957370
--- /dev/null
+++ b/model-00055.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:85e26a564ff7b2df9747d4be1fd49d3620f4dee869fb2a5d67560ead05980833
+size 5654441020
diff --git a/model-00056.safetensors b/model-00056.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..7f05f4eb7235588be04dbc86ba55c3b8bc58ec89
--- /dev/null
+++ b/model-00056.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:31a7783e2c3bdda12c72082be065d3bfd130e1fa143b8db003de1774f964dae9
+size 5673185437
diff --git a/model-00057.safetensors b/model-00057.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..49973a141a3eb3bfc9fed8d8849ca73f5de48f52
--- /dev/null
+++ b/model-00057.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:57e2cc9f75a1a614d636abf6eb9393d60ade0b3ab1c0b0d1f1465d3425d6baa2
+size 5654441000
diff --git a/model-00058.safetensors b/model-00058.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..51d713da236b9baa94703fa871c5e89fcc5949e6
--- /dev/null
+++ b/model-00058.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:127919c2dad3d6d89ee69c66b170c97489be437d726ef265f71c54e2f9f27b30
+size 5654441014
diff --git a/model-00059.safetensors b/model-00059.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..c956202c827a74bbcbc0995b7808b795bc5471e4
--- /dev/null
+++ b/model-00059.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e48143ea92fa9b7bca9d4f526dcdb37026219eb8e9f2361982bcafffe7ef8be9
+size 5654441008
diff --git a/model-00060.safetensors b/model-00060.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..9293e96e760eab46adbe76203f10489575c8e0be
--- /dev/null
+++ b/model-00060.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fe810ffa8a687e921dd4f5143f68a1a5f7587733c4b85425764289d3edb7c564
+size 5673185431
diff --git a/model-00061.safetensors b/model-00061.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..41bce27a14c8d2687b5c2a7d5f9e7a22824a11c0
--- /dev/null
+++ b/model-00061.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6dec32ada0879a7ba83513d31389e81258fc056bf1162de1c476320b677cb817
+size 5654441012
diff --git a/model-00062.safetensors b/model-00062.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..9fb8bbad6cb047b1eef0320b33832eaf216d1e17
--- /dev/null
+++ b/model-00062.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d3c3bc6fa28beb1538252e55f382258f0d21f1783b8666ec778a484abc221b94
+size 5654441004
diff --git a/model-00063.safetensors b/model-00063.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..d81e659c99a7f90c690a081ee4fa9a7371f829b8
--- /dev/null
+++ b/model-00063.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e04cefbd09837b7aea80a61c7bb1c01817c19ffa80b348b5490563a525c1cbd9
+size 5654441004
diff --git a/model-00064.safetensors b/model-00064.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..08e121532f6eb162f791acd8f4654882313aa8b8
--- /dev/null
+++ b/model-00064.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ebc795b5834907bd6da53584253bcf512c11bb683b1ccafcdba572ec84881cd
+size 5673185411
diff --git a/model-00065.safetensors b/model-00065.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..92b073dca58a5af2eb730a9094cafe0381d8e7fd
--- /dev/null
+++ b/model-00065.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0211631b372485bfe187dffd1e0c206b64780c7eafbf84eaaf9de3a635650e03
+size 5654441022
diff --git a/model-00066.safetensors b/model-00066.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..5d7df5553b8dfc3e4bb51a3295e9dd476c257f68
--- /dev/null
+++ b/model-00066.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:de67164941b8ecb4d08cb8a2edd2839325f74778ff433e7ec4cdb0d0cb017793
+size 5654441018
diff --git a/model-00067.safetensors b/model-00067.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..ff079b5c1e4ead86f7763293d9858fe1c10444f9
--- /dev/null
+++ b/model-00067.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ab35cbe32a93ec583533e12786026109b494146d5f7e8bb7be9bab525f6398f
+size 5654441010
diff --git a/model-00068.safetensors b/model-00068.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..67224eec396e23a68c28c89f9a84c52ed16ef204
--- /dev/null
+++ b/model-00068.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:298f53e732210ea24f4a0a30ec80d22d164d68d9c84b9fd8c4a5752105f8abf5
+size 5673185443
diff --git a/model-00069.safetensors b/model-00069.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..ece1cbb8314e5ee03750c8a5f0e625475ae453c8
--- /dev/null
+++ b/model-00069.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b6f0de748e2c79f423046a5ad42e6850ce9c5b30dc16ea18f006f0d25d96c26b
+size 5654441000
diff --git a/model-00070.safetensors b/model-00070.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..6f6b795d26a208060d0fa76337485be220b02ded
--- /dev/null
+++ b/model-00070.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:534e17ebede9f84878ee126df58f85f2841c2fe992779afc895175dcb43ae33a
+size 5654441006
diff --git a/model-00071.safetensors b/model-00071.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..14908a53a7677fe196299b6415adda27249fc1fb
--- /dev/null
+++ b/model-00071.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:97c190490872c8754653aa9f11dd26f8354b794e469cb6f5d2065d10c152c13c
+size 5654441012
diff --git a/model-00072.safetensors b/model-00072.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..554fd2fb077fd12bdda884a81a42f2c1eaa04a6d
--- /dev/null
+++ b/model-00072.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:64dab187368be338ccbbdb7f571f180cdcd61aa9eed88ebc1fd8e2ab25adf4ca
+size 5673185429
diff --git a/model-00073.safetensors b/model-00073.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..0eb8199c5dcbeef6994df42596eaeba8728d35c8
--- /dev/null
+++ b/model-00073.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d35d1cedcbe048e32a021a942a1706321300fea5bed2e879198bb3009d385e1
+size 5654441018
diff --git a/model-00074.safetensors b/model-00074.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..ec4f9ab4621b31b86cde7bf81b93bc41458746cf
--- /dev/null
+++ b/model-00074.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e0f358f7a90a272f9eea5ae415ee312cb5c7363a735cfa96efe5ecd5005169ff
+size 5654441008
diff --git a/model-00075.safetensors b/model-00075.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..ae12fc7f936c522c267ed90d6e8b07f9db2baa6d
--- /dev/null
+++ b/model-00075.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:98098d807e5ef917524c40362307290c746b640c9c5271889ffff98286a5fb4f
+size 5654441016
diff --git a/model-00076.safetensors b/model-00076.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..258fa31356b256e73b155743766f89f48233995d
--- /dev/null
+++ b/model-00076.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82d0e3394d880618ec81895c70120fcf8433bf20a152b023ed0123b52770dd2c
+size 5673185429
diff --git a/model-00077.safetensors b/model-00077.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..c7fe2042d289d8de720857ab9741ec4f620f9506
--- /dev/null
+++ b/model-00077.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d2ba4b7560c664da916be0008072afb35e1634b8ffac1470ae50fe15774ad57a
+size 5654441010
diff --git a/model-00078.safetensors b/model-00078.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..0a36bed2601d81d6f359b59a17efbbb1681da3a2
--- /dev/null
+++ b/model-00078.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:09c114d55e964f87928f479d5ea9e6630a6b7114e05ea5cc546de3dc49caa118
+size 5654441010
diff --git a/model-00079.safetensors b/model-00079.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..089752dbc6d24921b32e553827e6febc41b97ad8
--- /dev/null
+++ b/model-00079.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b1db4a054190044b502ecce54f2dc3fcba9094779151d5a44c3c09d2b38df30
+size 5654440996
diff --git a/model.safetensors.index.json b/model.safetensors.index.json
new file mode 100644
index 0000000000000000000000000000000000000000..0bdd3ffe507cb113d5e6b245eaa4673f672ab246
--- /dev/null
+++ b/model.safetensors.index.json
@@ -0,0 +1,3362 @@
+{
+  "metadata": {
+    "total_size": 427746878904
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00001.safetensors",
+    "lm_head.scales": "model-00001.safetensors",
+    "lm_head.biases": "model-00001.safetensors",
+    "model.embed_tokens.weight": "model-00001.safetensors",
+    "model.embed_tokens.scales": "model-00001.safetensors",
+    "model.embed_tokens.biases": "model-00001.safetensors",
+    "model.norm.weight": "model-00001.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00002.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00002.safetensors",
+    "model.layers.0.mlp.down_proj.scales": "model-00002.safetensors",
+    "model.layers.0.mlp.down_proj.biases": "model-00002.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00002.safetensors",
+    "model.layers.0.mlp.gate_proj.scales": "model-00002.safetensors",
+    "model.layers.0.mlp.gate_proj.biases": "model-00002.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00002.safetensors",
+    "model.layers.0.mlp.up_proj.scales": "model-00002.safetensors",
+    "model.layers.0.mlp.up_proj.biases": "model-00002.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00002.safetensors",
+    "model.layers.0.self_attn.indexer.k_norm.bias": "model-00002.safetensors",
+    "model.layers.0.self_attn.indexer.k_norm.weight": "model-00002.safetensors",
+    "model.layers.0.self_attn.indexer.weights_proj.weight": "model-00002.safetensors",
+    "model.layers.0.self_attn.indexer.wk.weight": "model-00002.safetensors",
+    "model.layers.0.self_attn.indexer.wq_b.weight": "model-00002.safetensors",
+    "model.layers.0.self_attn.kv_a_layernorm.weight": "model-00002.safetensors",
+    "model.layers.0.self_attn.kv_a_proj_with_mqa.weight": "model-00002.safetensors",
+    "model.layers.0.self_attn.kv_a_proj_with_mqa.scales": "model-00002.safetensors",
+    "model.layers.0.self_attn.kv_a_proj_with_mqa.biases": "model-00002.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00002.safetensors",
+    "model.layers.0.self_attn.o_proj.scales": "model-00002.safetensors",
+    "model.layers.0.self_attn.o_proj.biases": "model-00002.safetensors",
+    "model.layers.0.self_attn.q_a_layernorm.weight": "model-00002.safetensors",
+    "model.layers.0.self_attn.q_a_proj.weight": "model-00002.safetensors",
+    "model.layers.0.self_attn.q_a_proj.scales": "model-00002.safetensors",
+    "model.layers.0.self_attn.q_a_proj.biases": "model-00002.safetensors",
+    "model.layers.0.self_attn.q_b_proj.weight": "model-00002.safetensors",
+    "model.layers.0.self_attn.q_b_proj.scales": "model-00002.safetensors",
+    "model.layers.0.self_attn.q_b_proj.biases": "model-00002.safetensors",
+    "model.layers.0.self_attn.embed_q.weight": "model-00002.safetensors",
+    "model.layers.0.self_attn.embed_q.scales": "model-00002.safetensors",
+    "model.layers.0.self_attn.embed_q.biases": "model-00002.safetensors",
+    "model.layers.0.self_attn.unembed_out.weight": "model-00002.safetensors",
+    "model.layers.0.self_attn.unembed_out.scales": "model-00002.safetensors",
+    "model.layers.0.self_attn.unembed_out.biases": "model-00002.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00003.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00003.safetensors",
+    "model.layers.1.mlp.down_proj.scales": "model-00003.safetensors",
+    "model.layers.1.mlp.down_proj.biases": "model-00003.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00003.safetensors",
+    "model.layers.1.mlp.gate_proj.scales": "model-00003.safetensors",
+    "model.layers.1.mlp.gate_proj.biases": "model-00003.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00003.safetensors",
+    "model.layers.1.mlp.up_proj.scales": "model-00003.safetensors",
+    "model.layers.1.mlp.up_proj.biases": "model-00003.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00003.safetensors",
+    "model.layers.1.self_attn.indexer.k_norm.bias": "model-00003.safetensors",
+    "model.layers.1.self_attn.indexer.k_norm.weight": "model-00003.safetensors",
+    "model.layers.1.self_attn.indexer.weights_proj.weight": "model-00003.safetensors",
+    "model.layers.1.self_attn.indexer.wk.weight": "model-00003.safetensors",
+    "model.layers.1.self_attn.indexer.wq_b.weight": "model-00003.safetensors",
+    "model.layers.1.self_attn.kv_a_layernorm.weight": "model-00003.safetensors",
+    "model.layers.1.self_attn.kv_a_proj_with_mqa.weight": "model-00003.safetensors",
+    "model.layers.1.self_attn.kv_a_proj_with_mqa.scales": "model-00003.safetensors",
+    "model.layers.1.self_attn.kv_a_proj_with_mqa.biases": "model-00003.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00003.safetensors",
+    "model.layers.1.self_attn.o_proj.scales": "model-00003.safetensors",
+    "model.layers.1.self_attn.o_proj.biases": "model-00003.safetensors",
+    "model.layers.1.self_attn.q_a_layernorm.weight": "model-00003.safetensors",
+    "model.layers.1.self_attn.q_a_proj.weight": "model-00003.safetensors",
+    "model.layers.1.self_attn.q_a_proj.scales": "model-00003.safetensors",
+    "model.layers.1.self_attn.q_a_proj.biases": "model-00003.safetensors",
+    "model.layers.1.self_attn.q_b_proj.weight": "model-00003.safetensors",
+    "model.layers.1.self_attn.q_b_proj.scales": "model-00003.safetensors",
+    "model.layers.1.self_attn.q_b_proj.biases": "model-00003.safetensors",
+    "model.layers.1.self_attn.embed_q.weight": "model-00003.safetensors",
+    "model.layers.1.self_attn.embed_q.scales": "model-00003.safetensors",
+    "model.layers.1.self_attn.embed_q.biases": "model-00003.safetensors",
+    "model.layers.1.self_attn.unembed_out.weight": "model-00003.safetensors",
+    "model.layers.1.self_attn.unembed_out.scales": "model-00003.safetensors",
+    "model.layers.1.self_attn.unembed_out.biases": "model-00003.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00004.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00004.safetensors",
+    "model.layers.2.mlp.down_proj.scales": "model-00004.safetensors",
+    "model.layers.2.mlp.down_proj.biases": "model-00004.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00004.safetensors",
+    "model.layers.2.mlp.gate_proj.scales": "model-00004.safetensors",
+    "model.layers.2.mlp.gate_proj.biases": "model-00004.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00004.safetensors",
+    "model.layers.2.mlp.up_proj.scales": "model-00004.safetensors",
+    "model.layers.2.mlp.up_proj.biases": "model-00004.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00004.safetensors",
+    "model.layers.2.self_attn.indexer.k_norm.bias": "model-00004.safetensors",
+    "model.layers.2.self_attn.indexer.k_norm.weight": "model-00004.safetensors",
+    "model.layers.2.self_attn.indexer.weights_proj.weight": "model-00004.safetensors",
+    "model.layers.2.self_attn.indexer.wk.weight": "model-00004.safetensors",
+    "model.layers.2.self_attn.indexer.wq_b.weight": "model-00004.safetensors",
+    "model.layers.2.self_attn.kv_a_layernorm.weight": "model-00004.safetensors",
+    "model.layers.2.self_attn.kv_a_proj_with_mqa.weight": "model-00004.safetensors",
+    "model.layers.2.self_attn.kv_a_proj_with_mqa.scales": "model-00004.safetensors",
+    "model.layers.2.self_attn.kv_a_proj_with_mqa.biases": "model-00004.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00004.safetensors",
+    "model.layers.2.self_attn.o_proj.scales": "model-00004.safetensors",
+    "model.layers.2.self_attn.o_proj.biases": "model-00004.safetensors",
+    "model.layers.2.self_attn.q_a_layernorm.weight": "model-00004.safetensors",
+    "model.layers.2.self_attn.q_a_proj.weight": "model-00004.safetensors",
+    "model.layers.2.self_attn.q_a_proj.scales": "model-00004.safetensors",
+    "model.layers.2.self_attn.q_a_proj.biases": "model-00004.safetensors",
+    "model.layers.2.self_attn.q_b_proj.weight": "model-00004.safetensors",
+    "model.layers.2.self_attn.q_b_proj.scales": "model-00004.safetensors",
+    "model.layers.2.self_attn.q_b_proj.biases": "model-00004.safetensors",
+    "model.layers.2.self_attn.embed_q.weight": "model-00004.safetensors",
+    "model.layers.2.self_attn.embed_q.scales": "model-00004.safetensors",
+    "model.layers.2.self_attn.embed_q.biases": "model-00004.safetensors",
+    "model.layers.2.self_attn.unembed_out.weight": "model-00004.safetensors",
+    "model.layers.2.self_attn.unembed_out.scales": "model-00004.safetensors",
+    "model.layers.2.self_attn.unembed_out.biases": "model-00004.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00005.safetensors",
+    "model.layers.3.mlp.gate.e_score_correction_bias": "model-00005.safetensors",
+    "model.layers.3.mlp.gate.weight": "model-00005.safetensors",
+    "model.layers.3.mlp.shared_experts.down_proj.weight": "model-00005.safetensors",
+    "model.layers.3.mlp.shared_experts.down_proj.scales": "model-00005.safetensors",
+    "model.layers.3.mlp.shared_experts.down_proj.biases": "model-00005.safetensors",
+    "model.layers.3.mlp.shared_experts.gate_proj.weight": "model-00005.safetensors",
+    "model.layers.3.mlp.shared_experts.gate_proj.scales": "model-00005.safetensors",
+    "model.layers.3.mlp.shared_experts.gate_proj.biases": "model-00005.safetensors",
+    "model.layers.3.mlp.shared_experts.up_proj.weight": "model-00005.safetensors",
+    "model.layers.3.mlp.shared_experts.up_proj.scales": "model-00005.safetensors",
+    "model.layers.3.mlp.shared_experts.up_proj.biases": "model-00005.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00005.safetensors",
+    "model.layers.3.self_attn.kv_a_layernorm.weight": "model-00005.safetensors",
+    "model.layers.3.self_attn.kv_a_proj_with_mqa.weight": "model-00005.safetensors",
+    "model.layers.3.self_attn.kv_a_proj_with_mqa.scales": "model-00005.safetensors",
+    "model.layers.3.self_attn.kv_a_proj_with_mqa.biases": "model-00005.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00005.safetensors",
+    "model.layers.3.self_attn.o_proj.scales": "model-00005.safetensors",
+    "model.layers.3.self_attn.o_proj.biases": "model-00005.safetensors",
+    "model.layers.3.self_attn.q_a_layernorm.weight": "model-00005.safetensors",
+    "model.layers.3.self_attn.q_a_proj.weight": "model-00005.safetensors",
+    "model.layers.3.self_attn.q_a_proj.scales": "model-00005.safetensors",
+    "model.layers.3.self_attn.q_a_proj.biases": "model-00005.safetensors",
+    "model.layers.3.self_attn.q_b_proj.weight": "model-00005.safetensors",
+    "model.layers.3.self_attn.q_b_proj.scales": "model-00005.safetensors",
+    "model.layers.3.self_attn.q_b_proj.biases": "model-00005.safetensors",
+    "model.layers.3.mlp.switch_mlp.gate_proj.weight": "model-00005.safetensors",
+    "model.layers.3.mlp.switch_mlp.gate_proj.scales": "model-00005.safetensors",
+    "model.layers.3.mlp.switch_mlp.gate_proj.biases": "model-00005.safetensors",
+    "model.layers.3.mlp.switch_mlp.down_proj.weight": "model-00005.safetensors",
+    "model.layers.3.mlp.switch_mlp.down_proj.scales": "model-00005.safetensors",
+    "model.layers.3.mlp.switch_mlp.down_proj.biases": "model-00005.safetensors",
+    "model.layers.3.mlp.switch_mlp.up_proj.weight": "model-00005.safetensors",
+    "model.layers.3.mlp.switch_mlp.up_proj.scales": "model-00005.safetensors",
+    "model.layers.3.mlp.switch_mlp.up_proj.biases": "model-00005.safetensors",
+    "model.layers.3.self_attn.embed_q.weight": "model-00005.safetensors",
+    "model.layers.3.self_attn.embed_q.scales": "model-00005.safetensors",
+    "model.layers.3.self_attn.embed_q.biases": "model-00005.safetensors",
+    "model.layers.3.self_attn.unembed_out.weight": "model-00005.safetensors",
+    "model.layers.3.self_attn.unembed_out.scales": "model-00005.safetensors",
+    "model.layers.3.self_attn.unembed_out.biases": "model-00005.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00006.safetensors",
+    "model.layers.4.mlp.gate.e_score_correction_bias": "model-00006.safetensors",
+    "model.layers.4.mlp.gate.weight": "model-00006.safetensors",
+    "model.layers.4.mlp.shared_experts.down_proj.weight": "model-00006.safetensors",
+    "model.layers.4.mlp.shared_experts.down_proj.scales": "model-00006.safetensors",
+    "model.layers.4.mlp.shared_experts.down_proj.biases": "model-00006.safetensors",
+    "model.layers.4.mlp.shared_experts.gate_proj.weight": "model-00006.safetensors",
+    "model.layers.4.mlp.shared_experts.gate_proj.scales": "model-00006.safetensors",
+    "model.layers.4.mlp.shared_experts.gate_proj.biases": "model-00006.safetensors",
+    "model.layers.4.mlp.shared_experts.up_proj.weight": "model-00006.safetensors",
+    "model.layers.4.mlp.shared_experts.up_proj.scales": "model-00006.safetensors",
+    "model.layers.4.mlp.shared_experts.up_proj.biases": "model-00006.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00006.safetensors",
+    "model.layers.4.self_attn.kv_a_layernorm.weight": "model-00006.safetensors",
+    "model.layers.4.self_attn.kv_a_proj_with_mqa.weight": "model-00006.safetensors",
+    "model.layers.4.self_attn.kv_a_proj_with_mqa.scales": "model-00006.safetensors",
+    "model.layers.4.self_attn.kv_a_proj_with_mqa.biases": "model-00006.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00006.safetensors",
+    "model.layers.4.self_attn.o_proj.scales": "model-00006.safetensors",
+    "model.layers.4.self_attn.o_proj.biases": "model-00006.safetensors",
+    "model.layers.4.self_attn.q_a_layernorm.weight": "model-00006.safetensors",
+    "model.layers.4.self_attn.q_a_proj.weight": "model-00006.safetensors",
+    "model.layers.4.self_attn.q_a_proj.scales": "model-00006.safetensors",
+    "model.layers.4.self_attn.q_a_proj.biases": "model-00006.safetensors",
+    "model.layers.4.self_attn.q_b_proj.weight": "model-00006.safetensors",
+    "model.layers.4.self_attn.q_b_proj.scales": "model-00006.safetensors",
+    "model.layers.4.self_attn.q_b_proj.biases": "model-00006.safetensors",
+    "model.layers.4.mlp.switch_mlp.gate_proj.weight": "model-00006.safetensors",
+    "model.layers.4.mlp.switch_mlp.gate_proj.scales": "model-00006.safetensors",
+    "model.layers.4.mlp.switch_mlp.gate_proj.biases": "model-00006.safetensors",
+    "model.layers.4.mlp.switch_mlp.down_proj.weight": "model-00006.safetensors",
+    "model.layers.4.mlp.switch_mlp.down_proj.scales": "model-00006.safetensors",
+    "model.layers.4.mlp.switch_mlp.down_proj.biases": "model-00006.safetensors",
+    "model.layers.4.mlp.switch_mlp.up_proj.weight": "model-00006.safetensors",
+    "model.layers.4.mlp.switch_mlp.up_proj.scales": "model-00006.safetensors",
+    "model.layers.4.mlp.switch_mlp.up_proj.biases": "model-00006.safetensors",
+    "model.layers.4.self_attn.embed_q.weight": "model-00006.safetensors",
+    "model.layers.4.self_attn.embed_q.scales": "model-00006.safetensors",
+    "model.layers.4.self_attn.embed_q.biases": "model-00006.safetensors",
+    "model.layers.4.self_attn.unembed_out.weight": "model-00006.safetensors",
+    "model.layers.4.self_attn.unembed_out.scales": "model-00006.safetensors",
+    "model.layers.4.self_attn.unembed_out.biases": "model-00006.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00007.safetensors",
+    "model.layers.5.mlp.gate.e_score_correction_bias": "model-00007.safetensors",
+    "model.layers.5.mlp.gate.weight": "model-00007.safetensors",
+    "model.layers.5.mlp.shared_experts.down_proj.weight": "model-00007.safetensors",
+    "model.layers.5.mlp.shared_experts.down_proj.scales": "model-00007.safetensors",
+    "model.layers.5.mlp.shared_experts.down_proj.biases": "model-00007.safetensors",
+    "model.layers.5.mlp.shared_experts.gate_proj.weight": "model-00007.safetensors",
+    "model.layers.5.mlp.shared_experts.gate_proj.scales": "model-00007.safetensors",
+    "model.layers.5.mlp.shared_experts.gate_proj.biases": "model-00007.safetensors",
+    "model.layers.5.mlp.shared_experts.up_proj.weight": "model-00007.safetensors",
+    "model.layers.5.mlp.shared_experts.up_proj.scales": "model-00007.safetensors",
+    "model.layers.5.mlp.shared_experts.up_proj.biases": "model-00007.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00007.safetensors",
+    "model.layers.5.self_attn.kv_a_layernorm.weight": "model-00007.safetensors",
+    "model.layers.5.self_attn.kv_a_proj_with_mqa.weight": "model-00007.safetensors",
+    "model.layers.5.self_attn.kv_a_proj_with_mqa.scales": "model-00007.safetensors",
+    "model.layers.5.self_attn.kv_a_proj_with_mqa.biases": "model-00007.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00007.safetensors",
+    "model.layers.5.self_attn.o_proj.scales": "model-00007.safetensors",
+    "model.layers.5.self_attn.o_proj.biases": "model-00007.safetensors",
+    "model.layers.5.self_attn.q_a_layernorm.weight": "model-00007.safetensors",
+    "model.layers.5.self_attn.q_a_proj.weight": "model-00007.safetensors",
+    "model.layers.5.self_attn.q_a_proj.scales": "model-00007.safetensors",
+    "model.layers.5.self_attn.q_a_proj.biases": "model-00007.safetensors",
+    "model.layers.5.self_attn.q_b_proj.weight": "model-00007.safetensors",
+    "model.layers.5.self_attn.q_b_proj.scales": "model-00007.safetensors",
+    "model.layers.5.self_attn.q_b_proj.biases": "model-00007.safetensors",
+    "model.layers.5.mlp.switch_mlp.gate_proj.weight": "model-00007.safetensors",
+    "model.layers.5.mlp.switch_mlp.gate_proj.scales": "model-00007.safetensors",
+    "model.layers.5.mlp.switch_mlp.gate_proj.biases": "model-00007.safetensors",
+    "model.layers.5.mlp.switch_mlp.down_proj.weight": "model-00007.safetensors",
+    "model.layers.5.mlp.switch_mlp.down_proj.scales": "model-00007.safetensors",
+    "model.layers.5.mlp.switch_mlp.down_proj.biases": "model-00007.safetensors",
+    "model.layers.5.mlp.switch_mlp.up_proj.weight": "model-00007.safetensors",
+    "model.layers.5.mlp.switch_mlp.up_proj.scales": "model-00007.safetensors",
+    "model.layers.5.mlp.switch_mlp.up_proj.biases": "model-00007.safetensors",
+    "model.layers.5.self_attn.embed_q.weight": "model-00007.safetensors",
+    "model.layers.5.self_attn.embed_q.scales": "model-00007.safetensors",
+    "model.layers.5.self_attn.embed_q.biases": "model-00007.safetensors",
+    "model.layers.5.self_attn.unembed_out.weight": "model-00007.safetensors",
+    "model.layers.5.self_attn.unembed_out.scales": "model-00007.safetensors",
+    "model.layers.5.self_attn.unembed_out.biases": "model-00007.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00008.safetensors",
+    "model.layers.6.mlp.gate.e_score_correction_bias": "model-00008.safetensors",
+    "model.layers.6.mlp.gate.weight": "model-00008.safetensors",
+    "model.layers.6.mlp.shared_experts.down_proj.weight": "model-00008.safetensors",
+    "model.layers.6.mlp.shared_experts.down_proj.scales": "model-00008.safetensors",
+    "model.layers.6.mlp.shared_experts.down_proj.biases": "model-00008.safetensors",
+    "model.layers.6.mlp.shared_experts.gate_proj.weight": "model-00008.safetensors",
+    "model.layers.6.mlp.shared_experts.gate_proj.scales": "model-00008.safetensors",
+    "model.layers.6.mlp.shared_experts.gate_proj.biases": "model-00008.safetensors",
+    "model.layers.6.mlp.shared_experts.up_proj.weight": "model-00008.safetensors",
+    "model.layers.6.mlp.shared_experts.up_proj.scales": "model-00008.safetensors",
+    "model.layers.6.mlp.shared_experts.up_proj.biases": "model-00008.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00008.safetensors",
+    "model.layers.6.self_attn.indexer.k_norm.bias": "model-00008.safetensors",
+    "model.layers.6.self_attn.indexer.k_norm.weight": "model-00008.safetensors",
+    "model.layers.6.self_attn.indexer.weights_proj.weight": "model-00008.safetensors",
+    "model.layers.6.self_attn.indexer.wk.weight": "model-00008.safetensors",
+    "model.layers.6.self_attn.indexer.wq_b.weight": "model-00008.safetensors",
+    "model.layers.6.self_attn.kv_a_layernorm.weight": "model-00008.safetensors",
+    "model.layers.6.self_attn.kv_a_proj_with_mqa.weight": "model-00008.safetensors",
+    "model.layers.6.self_attn.kv_a_proj_with_mqa.scales": "model-00008.safetensors",
+    "model.layers.6.self_attn.kv_a_proj_with_mqa.biases": "model-00008.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00008.safetensors",
+    "model.layers.6.self_attn.o_proj.scales": "model-00008.safetensors",
+    "model.layers.6.self_attn.o_proj.biases": "model-00008.safetensors",
+    "model.layers.6.self_attn.q_a_layernorm.weight": "model-00008.safetensors",
+    "model.layers.6.self_attn.q_a_proj.weight": "model-00008.safetensors",
+    "model.layers.6.self_attn.q_a_proj.scales": "model-00008.safetensors",
+    "model.layers.6.self_attn.q_a_proj.biases": "model-00008.safetensors",
+    "model.layers.6.self_attn.q_b_proj.weight": "model-00008.safetensors",
+    "model.layers.6.self_attn.q_b_proj.scales": "model-00008.safetensors",
+    "model.layers.6.self_attn.q_b_proj.biases": "model-00008.safetensors",
+    "model.layers.6.mlp.switch_mlp.gate_proj.weight": "model-00008.safetensors",
+    "model.layers.6.mlp.switch_mlp.gate_proj.scales": "model-00008.safetensors",
+    "model.layers.6.mlp.switch_mlp.gate_proj.biases": "model-00008.safetensors",
+    "model.layers.6.mlp.switch_mlp.down_proj.weight": "model-00008.safetensors",
+    "model.layers.6.mlp.switch_mlp.down_proj.scales": "model-00008.safetensors",
+    "model.layers.6.mlp.switch_mlp.down_proj.biases": "model-00008.safetensors",
+    "model.layers.6.mlp.switch_mlp.up_proj.weight": "model-00008.safetensors",
+    "model.layers.6.mlp.switch_mlp.up_proj.scales": "model-00008.safetensors",
+    "model.layers.6.mlp.switch_mlp.up_proj.biases": "model-00008.safetensors",
+    "model.layers.6.self_attn.embed_q.weight": "model-00008.safetensors",
+    "model.layers.6.self_attn.embed_q.scales": "model-00008.safetensors",
+    "model.layers.6.self_attn.embed_q.biases": "model-00008.safetensors",
+    "model.layers.6.self_attn.unembed_out.weight": "model-00008.safetensors",
+    "model.layers.6.self_attn.unembed_out.scales": "model-00008.safetensors",
+    "model.layers.6.self_attn.unembed_out.biases": "model-00008.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00009.safetensors",
+    "model.layers.7.mlp.gate.e_score_correction_bias": "model-00009.safetensors",
+    "model.layers.7.mlp.gate.weight": "model-00009.safetensors",
+    "model.layers.7.mlp.shared_experts.down_proj.weight": "model-00009.safetensors",
+    "model.layers.7.mlp.shared_experts.down_proj.scales": "model-00009.safetensors",
+    "model.layers.7.mlp.shared_experts.down_proj.biases": "model-00009.safetensors",
+    "model.layers.7.mlp.shared_experts.gate_proj.weight": "model-00009.safetensors",
+    "model.layers.7.mlp.shared_experts.gate_proj.scales": "model-00009.safetensors",
+    "model.layers.7.mlp.shared_experts.gate_proj.biases": "model-00009.safetensors",
+    "model.layers.7.mlp.shared_experts.up_proj.weight": "model-00009.safetensors",
+    "model.layers.7.mlp.shared_experts.up_proj.scales": "model-00009.safetensors",
+    "model.layers.7.mlp.shared_experts.up_proj.biases": "model-00009.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00009.safetensors",
+    "model.layers.7.self_attn.kv_a_layernorm.weight": "model-00009.safetensors",
+    "model.layers.7.self_attn.kv_a_proj_with_mqa.weight": "model-00009.safetensors",
+    "model.layers.7.self_attn.kv_a_proj_with_mqa.scales": "model-00009.safetensors",
+    "model.layers.7.self_attn.kv_a_proj_with_mqa.biases": "model-00009.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00009.safetensors",
+    "model.layers.7.self_attn.o_proj.scales": "model-00009.safetensors",
+    "model.layers.7.self_attn.o_proj.biases": "model-00009.safetensors",
+    "model.layers.7.self_attn.q_a_layernorm.weight": "model-00009.safetensors",
+    "model.layers.7.self_attn.q_a_proj.weight": "model-00009.safetensors",
+    "model.layers.7.self_attn.q_a_proj.scales": "model-00009.safetensors",
+    "model.layers.7.self_attn.q_a_proj.biases": "model-00009.safetensors",
+    "model.layers.7.self_attn.q_b_proj.weight": "model-00009.safetensors",
+    "model.layers.7.self_attn.q_b_proj.scales": "model-00009.safetensors",
+    "model.layers.7.self_attn.q_b_proj.biases": "model-00009.safetensors",
+    "model.layers.7.mlp.switch_mlp.gate_proj.weight": "model-00009.safetensors",
+    "model.layers.7.mlp.switch_mlp.gate_proj.scales": "model-00009.safetensors",
+    "model.layers.7.mlp.switch_mlp.gate_proj.biases": "model-00009.safetensors",
+    "model.layers.7.mlp.switch_mlp.down_proj.weight": "model-00009.safetensors",
+    "model.layers.7.mlp.switch_mlp.down_proj.scales": "model-00009.safetensors",
+    "model.layers.7.mlp.switch_mlp.down_proj.biases": "model-00009.safetensors",
+    "model.layers.7.mlp.switch_mlp.up_proj.weight": "model-00009.safetensors",
+    "model.layers.7.mlp.switch_mlp.up_proj.scales": "model-00009.safetensors",
+    "model.layers.7.mlp.switch_mlp.up_proj.biases": "model-00009.safetensors",
+    "model.layers.7.self_attn.embed_q.weight": "model-00009.safetensors",
+    "model.layers.7.self_attn.embed_q.scales": "model-00009.safetensors",
+    "model.layers.7.self_attn.embed_q.biases": "model-00009.safetensors",
+    "model.layers.7.self_attn.unembed_out.weight": "model-00009.safetensors",
+    "model.layers.7.self_attn.unembed_out.scales": "model-00009.safetensors",
+    "model.layers.7.self_attn.unembed_out.biases": "model-00009.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00010.safetensors",
+    "model.layers.8.mlp.gate.e_score_correction_bias": "model-00010.safetensors",
+    "model.layers.8.mlp.gate.weight": "model-00010.safetensors",
+    "model.layers.8.mlp.shared_experts.down_proj.weight": "model-00010.safetensors",
+    "model.layers.8.mlp.shared_experts.down_proj.scales": "model-00010.safetensors",
+    "model.layers.8.mlp.shared_experts.down_proj.biases": "model-00010.safetensors",
+    "model.layers.8.mlp.shared_experts.gate_proj.weight": "model-00010.safetensors",
+    "model.layers.8.mlp.shared_experts.gate_proj.scales": "model-00010.safetensors",
+    "model.layers.8.mlp.shared_experts.gate_proj.biases": "model-00010.safetensors",
+    "model.layers.8.mlp.shared_experts.up_proj.weight": "model-00010.safetensors",
+    "model.layers.8.mlp.shared_experts.up_proj.scales": "model-00010.safetensors",
+    "model.layers.8.mlp.shared_experts.up_proj.biases": "model-00010.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00010.safetensors",
+    "model.layers.8.self_attn.kv_a_layernorm.weight": "model-00010.safetensors",
+    "model.layers.8.self_attn.kv_a_proj_with_mqa.weight": "model-00010.safetensors",
+    "model.layers.8.self_attn.kv_a_proj_with_mqa.scales": "model-00010.safetensors",
+    "model.layers.8.self_attn.kv_a_proj_with_mqa.biases": "model-00010.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00010.safetensors",
+    "model.layers.8.self_attn.o_proj.scales": "model-00010.safetensors",
+    "model.layers.8.self_attn.o_proj.biases": "model-00010.safetensors",
+    "model.layers.8.self_attn.q_a_layernorm.weight": "model-00010.safetensors",
+    "model.layers.8.self_attn.q_a_proj.weight": "model-00010.safetensors",
+    "model.layers.8.self_attn.q_a_proj.scales": "model-00010.safetensors",
+    "model.layers.8.self_attn.q_a_proj.biases": "model-00010.safetensors",
+    "model.layers.8.self_attn.q_b_proj.weight": "model-00010.safetensors",
+    "model.layers.8.self_attn.q_b_proj.scales": "model-00010.safetensors",
+    "model.layers.8.self_attn.q_b_proj.biases": "model-00010.safetensors",
+    "model.layers.8.mlp.switch_mlp.gate_proj.weight": "model-00010.safetensors",
+    "model.layers.8.mlp.switch_mlp.gate_proj.scales": "model-00010.safetensors",
+    "model.layers.8.mlp.switch_mlp.gate_proj.biases": "model-00010.safetensors",
+    "model.layers.8.mlp.switch_mlp.down_proj.weight": "model-00010.safetensors",
+    "model.layers.8.mlp.switch_mlp.down_proj.scales": "model-00010.safetensors",
+    "model.layers.8.mlp.switch_mlp.down_proj.biases": "model-00010.safetensors",
+    "model.layers.8.mlp.switch_mlp.up_proj.weight": "model-00010.safetensors",
+    "model.layers.8.mlp.switch_mlp.up_proj.scales": "model-00010.safetensors",
+    "model.layers.8.mlp.switch_mlp.up_proj.biases": "model-00010.safetensors",
+    "model.layers.8.self_attn.embed_q.weight": "model-00010.safetensors",
+    "model.layers.8.self_attn.embed_q.scales": "model-00010.safetensors",
+    "model.layers.8.self_attn.embed_q.biases": "model-00010.safetensors",
+    "model.layers.8.self_attn.unembed_out.weight": "model-00010.safetensors",
+    "model.layers.8.self_attn.unembed_out.scales": "model-00010.safetensors",
+    "model.layers.8.self_attn.unembed_out.biases": "model-00010.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00011.safetensors",
+    "model.layers.9.mlp.gate.e_score_correction_bias": "model-00011.safetensors",
+    "model.layers.9.mlp.gate.weight": "model-00011.safetensors",
+    "model.layers.9.mlp.shared_experts.down_proj.weight": "model-00011.safetensors",
+    "model.layers.9.mlp.shared_experts.down_proj.scales": "model-00011.safetensors",
+    "model.layers.9.mlp.shared_experts.down_proj.biases": "model-00011.safetensors",
+    "model.layers.9.mlp.shared_experts.gate_proj.weight": "model-00011.safetensors",
+    "model.layers.9.mlp.shared_experts.gate_proj.scales": "model-00011.safetensors",
+    "model.layers.9.mlp.shared_experts.gate_proj.biases": "model-00011.safetensors",
+    "model.layers.9.mlp.shared_experts.up_proj.weight": "model-00011.safetensors",
+    "model.layers.9.mlp.shared_experts.up_proj.scales": "model-00011.safetensors",
+    "model.layers.9.mlp.shared_experts.up_proj.biases": "model-00011.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00011.safetensors",
+    "model.layers.9.self_attn.kv_a_layernorm.weight": "model-00011.safetensors",
+    "model.layers.9.self_attn.kv_a_proj_with_mqa.weight": "model-00011.safetensors",
+    "model.layers.9.self_attn.kv_a_proj_with_mqa.scales": "model-00011.safetensors",
+    "model.layers.9.self_attn.kv_a_proj_with_mqa.biases": "model-00011.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00011.safetensors",
+    "model.layers.9.self_attn.o_proj.scales": "model-00011.safetensors",
+    "model.layers.9.self_attn.o_proj.biases": "model-00011.safetensors",
+    "model.layers.9.self_attn.q_a_layernorm.weight": "model-00011.safetensors",
+    "model.layers.9.self_attn.q_a_proj.weight": "model-00011.safetensors",
+    "model.layers.9.self_attn.q_a_proj.scales": "model-00011.safetensors",
+    "model.layers.9.self_attn.q_a_proj.biases": "model-00011.safetensors",
+    "model.layers.9.self_attn.q_b_proj.weight": "model-00011.safetensors",
+    "model.layers.9.self_attn.q_b_proj.scales": "model-00011.safetensors",
+    "model.layers.9.self_attn.q_b_proj.biases": "model-00011.safetensors",
+    "model.layers.9.mlp.switch_mlp.gate_proj.weight": "model-00011.safetensors",
+    "model.layers.9.mlp.switch_mlp.gate_proj.scales": "model-00011.safetensors",
+    "model.layers.9.mlp.switch_mlp.gate_proj.biases": "model-00011.safetensors",
+    "model.layers.9.mlp.switch_mlp.down_proj.weight": "model-00011.safetensors",
+    "model.layers.9.mlp.switch_mlp.down_proj.scales": "model-00011.safetensors",
+    "model.layers.9.mlp.switch_mlp.down_proj.biases": "model-00011.safetensors",
+    "model.layers.9.mlp.switch_mlp.up_proj.weight": "model-00011.safetensors",
+    "model.layers.9.mlp.switch_mlp.up_proj.scales": "model-00011.safetensors",
+    "model.layers.9.mlp.switch_mlp.up_proj.biases": "model-00011.safetensors",
+    "model.layers.9.self_attn.embed_q.weight": "model-00011.safetensors",
+    "model.layers.9.self_attn.embed_q.scales": "model-00011.safetensors",
+    "model.layers.9.self_attn.embed_q.biases": "model-00011.safetensors",
+    "model.layers.9.self_attn.unembed_out.weight": "model-00011.safetensors",
+    "model.layers.9.self_attn.unembed_out.scales": "model-00011.safetensors",
+    "model.layers.9.self_attn.unembed_out.biases": "model-00011.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00012.safetensors",
+    "model.layers.10.mlp.gate.e_score_correction_bias": "model-00012.safetensors",
+    "model.layers.10.mlp.gate.weight": "model-00012.safetensors",
+    "model.layers.10.mlp.shared_experts.down_proj.weight": "model-00012.safetensors",
+    "model.layers.10.mlp.shared_experts.down_proj.scales": "model-00012.safetensors",
+    "model.layers.10.mlp.shared_experts.down_proj.biases": "model-00012.safetensors",
+    "model.layers.10.mlp.shared_experts.gate_proj.weight": "model-00012.safetensors",
+    "model.layers.10.mlp.shared_experts.gate_proj.scales": "model-00012.safetensors",
+    "model.layers.10.mlp.shared_experts.gate_proj.biases": "model-00012.safetensors",
+    "model.layers.10.mlp.shared_experts.up_proj.weight": "model-00012.safetensors",
+    "model.layers.10.mlp.shared_experts.up_proj.scales": "model-00012.safetensors",
+    "model.layers.10.mlp.shared_experts.up_proj.biases": "model-00012.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00012.safetensors",
+    "model.layers.10.self_attn.indexer.k_norm.bias": "model-00012.safetensors",
+    "model.layers.10.self_attn.indexer.k_norm.weight": "model-00012.safetensors",
+    "model.layers.10.self_attn.indexer.weights_proj.weight": "model-00012.safetensors",
+    "model.layers.10.self_attn.indexer.wk.weight": "model-00012.safetensors",
+    "model.layers.10.self_attn.indexer.wq_b.weight": "model-00012.safetensors",
+    "model.layers.10.self_attn.kv_a_layernorm.weight": "model-00012.safetensors",
+    "model.layers.10.self_attn.kv_a_proj_with_mqa.weight": "model-00012.safetensors",
+    "model.layers.10.self_attn.kv_a_proj_with_mqa.scales": "model-00012.safetensors",
+    "model.layers.10.self_attn.kv_a_proj_with_mqa.biases": "model-00012.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00012.safetensors",
+    "model.layers.10.self_attn.o_proj.scales": "model-00012.safetensors",
+    "model.layers.10.self_attn.o_proj.biases": "model-00012.safetensors",
+    "model.layers.10.self_attn.q_a_layernorm.weight": "model-00012.safetensors",
+    "model.layers.10.self_attn.q_a_proj.weight": "model-00012.safetensors",
+    "model.layers.10.self_attn.q_a_proj.scales": "model-00012.safetensors",
+    "model.layers.10.self_attn.q_a_proj.biases": "model-00012.safetensors",
+    "model.layers.10.self_attn.q_b_proj.weight": "model-00012.safetensors",
+    "model.layers.10.self_attn.q_b_proj.scales": "model-00012.safetensors",
+    "model.layers.10.self_attn.q_b_proj.biases": "model-00012.safetensors",
+    "model.layers.10.mlp.switch_mlp.gate_proj.weight": "model-00012.safetensors",
+    "model.layers.10.mlp.switch_mlp.gate_proj.scales": "model-00012.safetensors",
+    "model.layers.10.mlp.switch_mlp.gate_proj.biases": "model-00012.safetensors",
+    "model.layers.10.mlp.switch_mlp.down_proj.weight": "model-00012.safetensors",
+    "model.layers.10.mlp.switch_mlp.down_proj.scales": "model-00012.safetensors",
+    "model.layers.10.mlp.switch_mlp.down_proj.biases": "model-00012.safetensors",
+    "model.layers.10.mlp.switch_mlp.up_proj.weight": "model-00012.safetensors",
+    "model.layers.10.mlp.switch_mlp.up_proj.scales": "model-00012.safetensors",
+    "model.layers.10.mlp.switch_mlp.up_proj.biases": "model-00012.safetensors",
+    "model.layers.10.self_attn.embed_q.weight": "model-00012.safetensors",
+    "model.layers.10.self_attn.embed_q.scales": "model-00012.safetensors",
+    "model.layers.10.self_attn.embed_q.biases": "model-00012.safetensors",
+    "model.layers.10.self_attn.unembed_out.weight": "model-00012.safetensors",
+    "model.layers.10.self_attn.unembed_out.scales": "model-00012.safetensors",
+    "model.layers.10.self_attn.unembed_out.biases": "model-00012.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00013.safetensors",
+    "model.layers.11.mlp.gate.e_score_correction_bias": "model-00013.safetensors",
+    "model.layers.11.mlp.gate.weight": "model-00013.safetensors",
+    "model.layers.11.mlp.shared_experts.down_proj.weight": "model-00013.safetensors",
+    "model.layers.11.mlp.shared_experts.down_proj.scales": "model-00013.safetensors",
+    "model.layers.11.mlp.shared_experts.down_proj.biases": "model-00013.safetensors",
+    "model.layers.11.mlp.shared_experts.gate_proj.weight": "model-00013.safetensors",
+    "model.layers.11.mlp.shared_experts.gate_proj.scales": "model-00013.safetensors",
+    "model.layers.11.mlp.shared_experts.gate_proj.biases": "model-00013.safetensors",
+    "model.layers.11.mlp.shared_experts.up_proj.weight": "model-00013.safetensors",
+    "model.layers.11.mlp.shared_experts.up_proj.scales": "model-00013.safetensors",
+    "model.layers.11.mlp.shared_experts.up_proj.biases": "model-00013.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00013.safetensors",
+    "model.layers.11.self_attn.kv_a_layernorm.weight": "model-00013.safetensors",
+    "model.layers.11.self_attn.kv_a_proj_with_mqa.weight": "model-00013.safetensors",
+    "model.layers.11.self_attn.kv_a_proj_with_mqa.scales": "model-00013.safetensors",
+    "model.layers.11.self_attn.kv_a_proj_with_mqa.biases": "model-00013.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00013.safetensors",
+    "model.layers.11.self_attn.o_proj.scales": "model-00013.safetensors",
+    "model.layers.11.self_attn.o_proj.biases": "model-00013.safetensors",
+    "model.layers.11.self_attn.q_a_layernorm.weight": "model-00013.safetensors",
+    "model.layers.11.self_attn.q_a_proj.weight": "model-00013.safetensors",
+    "model.layers.11.self_attn.q_a_proj.scales": "model-00013.safetensors",
+    "model.layers.11.self_attn.q_a_proj.biases": "model-00013.safetensors",
+    "model.layers.11.self_attn.q_b_proj.weight": "model-00013.safetensors",
+    "model.layers.11.self_attn.q_b_proj.scales": "model-00013.safetensors",
+    "model.layers.11.self_attn.q_b_proj.biases": "model-00013.safetensors",
+    "model.layers.11.mlp.switch_mlp.gate_proj.weight": "model-00013.safetensors",
+    "model.layers.11.mlp.switch_mlp.gate_proj.scales": "model-00013.safetensors",
+    "model.layers.11.mlp.switch_mlp.gate_proj.biases": "model-00013.safetensors",
+    "model.layers.11.mlp.switch_mlp.down_proj.weight": "model-00013.safetensors",
+    "model.layers.11.mlp.switch_mlp.down_proj.scales": "model-00013.safetensors",
+    "model.layers.11.mlp.switch_mlp.down_proj.biases": "model-00013.safetensors",
+    "model.layers.11.mlp.switch_mlp.up_proj.weight": "model-00013.safetensors",
+    "model.layers.11.mlp.switch_mlp.up_proj.scales": "model-00013.safetensors",
+    "model.layers.11.mlp.switch_mlp.up_proj.biases": "model-00013.safetensors",
+    "model.layers.11.self_attn.embed_q.weight": "model-00013.safetensors",
+    "model.layers.11.self_attn.embed_q.scales": "model-00013.safetensors",
+    "model.layers.11.self_attn.embed_q.biases": "model-00013.safetensors",
+    "model.layers.11.self_attn.unembed_out.weight": "model-00013.safetensors",
+    "model.layers.11.self_attn.unembed_out.scales": "model-00013.safetensors",
+    "model.layers.11.self_attn.unembed_out.biases": "model-00013.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00014.safetensors",
+    "model.layers.12.mlp.gate.e_score_correction_bias": "model-00014.safetensors",
+    "model.layers.12.mlp.gate.weight": "model-00014.safetensors",
+    "model.layers.12.mlp.shared_experts.down_proj.weight": "model-00014.safetensors",
+    "model.layers.12.mlp.shared_experts.down_proj.scales": "model-00014.safetensors",
+    "model.layers.12.mlp.shared_experts.down_proj.biases": "model-00014.safetensors",
+    "model.layers.12.mlp.shared_experts.gate_proj.weight": "model-00014.safetensors",
+    "model.layers.12.mlp.shared_experts.gate_proj.scales": "model-00014.safetensors",
+    "model.layers.12.mlp.shared_experts.gate_proj.biases": "model-00014.safetensors",
+    "model.layers.12.mlp.shared_experts.up_proj.weight": "model-00014.safetensors",
+    "model.layers.12.mlp.shared_experts.up_proj.scales": "model-00014.safetensors",
+    "model.layers.12.mlp.shared_experts.up_proj.biases": "model-00014.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00014.safetensors",
+    "model.layers.12.self_attn.kv_a_layernorm.weight": "model-00014.safetensors",
+    "model.layers.12.self_attn.kv_a_proj_with_mqa.weight": "model-00014.safetensors",
+    "model.layers.12.self_attn.kv_a_proj_with_mqa.scales": "model-00014.safetensors",
+    "model.layers.12.self_attn.kv_a_proj_with_mqa.biases": "model-00014.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00014.safetensors",
+    "model.layers.12.self_attn.o_proj.scales": "model-00014.safetensors",
+    "model.layers.12.self_attn.o_proj.biases": "model-00014.safetensors",
+    "model.layers.12.self_attn.q_a_layernorm.weight": "model-00014.safetensors",
+    "model.layers.12.self_attn.q_a_proj.weight": "model-00014.safetensors",
+    "model.layers.12.self_attn.q_a_proj.scales": "model-00014.safetensors",
+    "model.layers.12.self_attn.q_a_proj.biases": "model-00014.safetensors",
+    "model.layers.12.self_attn.q_b_proj.weight": "model-00014.safetensors",
+    "model.layers.12.self_attn.q_b_proj.scales": "model-00014.safetensors",
+    "model.layers.12.self_attn.q_b_proj.biases": "model-00014.safetensors",
+    "model.layers.12.mlp.switch_mlp.gate_proj.weight": "model-00014.safetensors",
+    "model.layers.12.mlp.switch_mlp.gate_proj.scales": "model-00014.safetensors",
+    "model.layers.12.mlp.switch_mlp.gate_proj.biases": "model-00014.safetensors",
+    "model.layers.12.mlp.switch_mlp.down_proj.weight": "model-00014.safetensors",
+    "model.layers.12.mlp.switch_mlp.down_proj.scales": "model-00014.safetensors",
+    "model.layers.12.mlp.switch_mlp.down_proj.biases": "model-00014.safetensors",
+    "model.layers.12.mlp.switch_mlp.up_proj.weight": "model-00014.safetensors",
+    "model.layers.12.mlp.switch_mlp.up_proj.scales": "model-00014.safetensors",
+    "model.layers.12.mlp.switch_mlp.up_proj.biases": "model-00014.safetensors",
+    "model.layers.12.self_attn.embed_q.weight": "model-00014.safetensors",
+    "model.layers.12.self_attn.embed_q.scales": "model-00014.safetensors",
+    "model.layers.12.self_attn.embed_q.biases": "model-00014.safetensors",
+    "model.layers.12.self_attn.unembed_out.weight": "model-00014.safetensors",
+    "model.layers.12.self_attn.unembed_out.scales": "model-00014.safetensors",
+    "model.layers.12.self_attn.unembed_out.biases": "model-00014.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00015.safetensors",
+    "model.layers.13.mlp.gate.e_score_correction_bias": "model-00015.safetensors",
+    "model.layers.13.mlp.gate.weight": "model-00015.safetensors",
+    "model.layers.13.mlp.shared_experts.down_proj.weight": "model-00015.safetensors",
+    "model.layers.13.mlp.shared_experts.down_proj.scales": "model-00015.safetensors",
+    "model.layers.13.mlp.shared_experts.down_proj.biases": "model-00015.safetensors",
+    "model.layers.13.mlp.shared_experts.gate_proj.weight": "model-00015.safetensors",
+    "model.layers.13.mlp.shared_experts.gate_proj.scales": "model-00015.safetensors",
+    "model.layers.13.mlp.shared_experts.gate_proj.biases": "model-00015.safetensors",
+    "model.layers.13.mlp.shared_experts.up_proj.weight": "model-00015.safetensors",
+    "model.layers.13.mlp.shared_experts.up_proj.scales": "model-00015.safetensors",
+    "model.layers.13.mlp.shared_experts.up_proj.biases": "model-00015.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00015.safetensors",
+    "model.layers.13.self_attn.kv_a_layernorm.weight": "model-00015.safetensors",
+    "model.layers.13.self_attn.kv_a_proj_with_mqa.weight": "model-00015.safetensors",
+    "model.layers.13.self_attn.kv_a_proj_with_mqa.scales": "model-00015.safetensors",
+    "model.layers.13.self_attn.kv_a_proj_with_mqa.biases": "model-00015.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00015.safetensors",
+    "model.layers.13.self_attn.o_proj.scales": "model-00015.safetensors",
+    "model.layers.13.self_attn.o_proj.biases": "model-00015.safetensors",
+    "model.layers.13.self_attn.q_a_layernorm.weight": "model-00015.safetensors",
+    "model.layers.13.self_attn.q_a_proj.weight": "model-00015.safetensors",
+    "model.layers.13.self_attn.q_a_proj.scales": "model-00015.safetensors",
+    "model.layers.13.self_attn.q_a_proj.biases": "model-00015.safetensors",
+    "model.layers.13.self_attn.q_b_proj.weight": "model-00015.safetensors",
+    "model.layers.13.self_attn.q_b_proj.scales": "model-00015.safetensors",
+    "model.layers.13.self_attn.q_b_proj.biases": "model-00015.safetensors",
+    "model.layers.13.mlp.switch_mlp.gate_proj.weight": "model-00015.safetensors",
+    "model.layers.13.mlp.switch_mlp.gate_proj.scales": "model-00015.safetensors",
+    "model.layers.13.mlp.switch_mlp.gate_proj.biases": "model-00015.safetensors",
+    "model.layers.13.mlp.switch_mlp.down_proj.weight": "model-00015.safetensors",
+    "model.layers.13.mlp.switch_mlp.down_proj.scales": "model-00015.safetensors",
+    "model.layers.13.mlp.switch_mlp.down_proj.biases": "model-00015.safetensors",
+    "model.layers.13.mlp.switch_mlp.up_proj.weight": "model-00015.safetensors",
+    "model.layers.13.mlp.switch_mlp.up_proj.scales": "model-00015.safetensors",
+    "model.layers.13.mlp.switch_mlp.up_proj.biases": "model-00015.safetensors",
+    "model.layers.13.self_attn.embed_q.weight": "model-00015.safetensors",
+    "model.layers.13.self_attn.embed_q.scales": "model-00015.safetensors",
+    "model.layers.13.self_attn.embed_q.biases": "model-00015.safetensors",
+    "model.layers.13.self_attn.unembed_out.weight": "model-00015.safetensors",
+    "model.layers.13.self_attn.unembed_out.scales": "model-00015.safetensors",
+    "model.layers.13.self_attn.unembed_out.biases": "model-00015.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00016.safetensors",
+    "model.layers.14.mlp.gate.e_score_correction_bias": "model-00016.safetensors",
+    "model.layers.14.mlp.gate.weight": "model-00016.safetensors",
+    "model.layers.14.mlp.shared_experts.down_proj.weight": "model-00016.safetensors",
+    "model.layers.14.mlp.shared_experts.down_proj.scales": "model-00016.safetensors",
+    "model.layers.14.mlp.shared_experts.down_proj.biases": "model-00016.safetensors",
+    "model.layers.14.mlp.shared_experts.gate_proj.weight": "model-00016.safetensors",
+    "model.layers.14.mlp.shared_experts.gate_proj.scales": "model-00016.safetensors",
+    "model.layers.14.mlp.shared_experts.gate_proj.biases": "model-00016.safetensors",
+    "model.layers.14.mlp.shared_experts.up_proj.weight": "model-00016.safetensors",
+    "model.layers.14.mlp.shared_experts.up_proj.scales": "model-00016.safetensors",
+    "model.layers.14.mlp.shared_experts.up_proj.biases": "model-00016.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00016.safetensors",
+    "model.layers.14.self_attn.indexer.k_norm.bias": "model-00016.safetensors",
+    "model.layers.14.self_attn.indexer.k_norm.weight": "model-00016.safetensors",
+    "model.layers.14.self_attn.indexer.weights_proj.weight": "model-00016.safetensors",
+    "model.layers.14.self_attn.indexer.wk.weight": "model-00016.safetensors",
+    "model.layers.14.self_attn.indexer.wq_b.weight": "model-00016.safetensors",
+    "model.layers.14.self_attn.kv_a_layernorm.weight": "model-00016.safetensors",
+    "model.layers.14.self_attn.kv_a_proj_with_mqa.weight": "model-00016.safetensors",
+    "model.layers.14.self_attn.kv_a_proj_with_mqa.scales": "model-00016.safetensors",
+    "model.layers.14.self_attn.kv_a_proj_with_mqa.biases": "model-00016.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00016.safetensors",
+    "model.layers.14.self_attn.o_proj.scales": "model-00016.safetensors",
+    "model.layers.14.self_attn.o_proj.biases": "model-00016.safetensors",
+    "model.layers.14.self_attn.q_a_layernorm.weight": "model-00016.safetensors",
+    "model.layers.14.self_attn.q_a_proj.weight": "model-00016.safetensors",
+    "model.layers.14.self_attn.q_a_proj.scales": "model-00016.safetensors",
+    "model.layers.14.self_attn.q_a_proj.biases": "model-00016.safetensors",
+    "model.layers.14.self_attn.q_b_proj.weight": "model-00016.safetensors",
+    "model.layers.14.self_attn.q_b_proj.scales": "model-00016.safetensors",
+    "model.layers.14.self_attn.q_b_proj.biases": "model-00016.safetensors",
+    "model.layers.14.mlp.switch_mlp.gate_proj.weight": "model-00016.safetensors",
+    "model.layers.14.mlp.switch_mlp.gate_proj.scales": "model-00016.safetensors",
+    "model.layers.14.mlp.switch_mlp.gate_proj.biases": "model-00016.safetensors",
+    "model.layers.14.mlp.switch_mlp.down_proj.weight": "model-00016.safetensors",
+    "model.layers.14.mlp.switch_mlp.down_proj.scales": "model-00016.safetensors",
+    "model.layers.14.mlp.switch_mlp.down_proj.biases": "model-00016.safetensors",
+    "model.layers.14.mlp.switch_mlp.up_proj.weight": "model-00016.safetensors",
+    "model.layers.14.mlp.switch_mlp.up_proj.scales": "model-00016.safetensors",
+    "model.layers.14.mlp.switch_mlp.up_proj.biases": "model-00016.safetensors",
+    "model.layers.14.self_attn.embed_q.weight": "model-00016.safetensors",
+    "model.layers.14.self_attn.embed_q.scales": "model-00016.safetensors",
+    "model.layers.14.self_attn.embed_q.biases": "model-00016.safetensors",
+    "model.layers.14.self_attn.unembed_out.weight": "model-00016.safetensors",
+    "model.layers.14.self_attn.unembed_out.scales": "model-00016.safetensors",
+    "model.layers.14.self_attn.unembed_out.biases": "model-00016.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00017.safetensors",
+    "model.layers.15.mlp.gate.e_score_correction_bias": "model-00017.safetensors",
+    "model.layers.15.mlp.gate.weight": "model-00017.safetensors",
+    "model.layers.15.mlp.shared_experts.down_proj.weight": "model-00017.safetensors",
+    "model.layers.15.mlp.shared_experts.down_proj.scales": "model-00017.safetensors",
+    "model.layers.15.mlp.shared_experts.down_proj.biases": "model-00017.safetensors",
+    "model.layers.15.mlp.shared_experts.gate_proj.weight": "model-00017.safetensors",
+    "model.layers.15.mlp.shared_experts.gate_proj.scales": "model-00017.safetensors",
+    "model.layers.15.mlp.shared_experts.gate_proj.biases": "model-00017.safetensors",
+    "model.layers.15.mlp.shared_experts.up_proj.weight": "model-00017.safetensors",
+    "model.layers.15.mlp.shared_experts.up_proj.scales": "model-00017.safetensors",
+    "model.layers.15.mlp.shared_experts.up_proj.biases": "model-00017.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00017.safetensors",
+    "model.layers.15.self_attn.kv_a_layernorm.weight": "model-00017.safetensors",
+    "model.layers.15.self_attn.kv_a_proj_with_mqa.weight": "model-00017.safetensors",
+    "model.layers.15.self_attn.kv_a_proj_with_mqa.scales": "model-00017.safetensors",
+    "model.layers.15.self_attn.kv_a_proj_with_mqa.biases": "model-00017.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00017.safetensors",
+    "model.layers.15.self_attn.o_proj.scales": "model-00017.safetensors",
+    "model.layers.15.self_attn.o_proj.biases": "model-00017.safetensors",
+    "model.layers.15.self_attn.q_a_layernorm.weight": "model-00017.safetensors",
+    "model.layers.15.self_attn.q_a_proj.weight": "model-00017.safetensors",
+    "model.layers.15.self_attn.q_a_proj.scales": "model-00017.safetensors",
+    "model.layers.15.self_attn.q_a_proj.biases": "model-00017.safetensors",
+    "model.layers.15.self_attn.q_b_proj.weight": "model-00017.safetensors",
+    "model.layers.15.self_attn.q_b_proj.scales": "model-00017.safetensors",
+    "model.layers.15.self_attn.q_b_proj.biases": "model-00017.safetensors",
+    "model.layers.15.mlp.switch_mlp.gate_proj.weight": "model-00017.safetensors",
+    "model.layers.15.mlp.switch_mlp.gate_proj.scales": "model-00017.safetensors",
+    "model.layers.15.mlp.switch_mlp.gate_proj.biases": "model-00017.safetensors",
+    "model.layers.15.mlp.switch_mlp.down_proj.weight": "model-00017.safetensors",
+    "model.layers.15.mlp.switch_mlp.down_proj.scales": "model-00017.safetensors",
+    "model.layers.15.mlp.switch_mlp.down_proj.biases": "model-00017.safetensors",
+    "model.layers.15.mlp.switch_mlp.up_proj.weight": "model-00017.safetensors",
+    "model.layers.15.mlp.switch_mlp.up_proj.scales": "model-00017.safetensors",
+    "model.layers.15.mlp.switch_mlp.up_proj.biases": "model-00017.safetensors",
+    "model.layers.15.self_attn.embed_q.weight": "model-00017.safetensors",
+    "model.layers.15.self_attn.embed_q.scales": "model-00017.safetensors",
+    "model.layers.15.self_attn.embed_q.biases": "model-00017.safetensors",
+    "model.layers.15.self_attn.unembed_out.weight": "model-00017.safetensors",
+    "model.layers.15.self_attn.unembed_out.scales": "model-00017.safetensors",
+    "model.layers.15.self_attn.unembed_out.biases": "model-00017.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00018.safetensors",
+    "model.layers.16.mlp.gate.e_score_correction_bias": "model-00018.safetensors",
+    "model.layers.16.mlp.gate.weight": "model-00018.safetensors",
+    "model.layers.16.mlp.shared_experts.down_proj.weight": "model-00018.safetensors",
+    "model.layers.16.mlp.shared_experts.down_proj.scales": "model-00018.safetensors",
+    "model.layers.16.mlp.shared_experts.down_proj.biases": "model-00018.safetensors",
+    "model.layers.16.mlp.shared_experts.gate_proj.weight": "model-00018.safetensors",
+    "model.layers.16.mlp.shared_experts.gate_proj.scales": "model-00018.safetensors",
+    "model.layers.16.mlp.shared_experts.gate_proj.biases": "model-00018.safetensors",
+    "model.layers.16.mlp.shared_experts.up_proj.weight": "model-00018.safetensors",
+    "model.layers.16.mlp.shared_experts.up_proj.scales": "model-00018.safetensors",
+    "model.layers.16.mlp.shared_experts.up_proj.biases": "model-00018.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00018.safetensors",
+    "model.layers.16.self_attn.kv_a_layernorm.weight": "model-00018.safetensors",
+    "model.layers.16.self_attn.kv_a_proj_with_mqa.weight": "model-00018.safetensors",
+    "model.layers.16.self_attn.kv_a_proj_with_mqa.scales": "model-00018.safetensors",
+    "model.layers.16.self_attn.kv_a_proj_with_mqa.biases": "model-00018.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00018.safetensors",
+    "model.layers.16.self_attn.o_proj.scales": "model-00018.safetensors",
+    "model.layers.16.self_attn.o_proj.biases": "model-00018.safetensors",
+    "model.layers.16.self_attn.q_a_layernorm.weight": "model-00018.safetensors",
+    "model.layers.16.self_attn.q_a_proj.weight": "model-00018.safetensors",
+    "model.layers.16.self_attn.q_a_proj.scales": "model-00018.safetensors",
+    "model.layers.16.self_attn.q_a_proj.biases": "model-00018.safetensors",
+    "model.layers.16.self_attn.q_b_proj.weight": "model-00018.safetensors",
+    "model.layers.16.self_attn.q_b_proj.scales": "model-00018.safetensors",
+    "model.layers.16.self_attn.q_b_proj.biases": "model-00018.safetensors",
+    "model.layers.16.mlp.switch_mlp.gate_proj.weight": "model-00018.safetensors",
+    "model.layers.16.mlp.switch_mlp.gate_proj.scales": "model-00018.safetensors",
+    "model.layers.16.mlp.switch_mlp.gate_proj.biases": "model-00018.safetensors",
+    "model.layers.16.mlp.switch_mlp.down_proj.weight": "model-00018.safetensors",
+    "model.layers.16.mlp.switch_mlp.down_proj.scales": "model-00018.safetensors",
+    "model.layers.16.mlp.switch_mlp.down_proj.biases": "model-00018.safetensors",
+    "model.layers.16.mlp.switch_mlp.up_proj.weight": "model-00018.safetensors",
+    "model.layers.16.mlp.switch_mlp.up_proj.scales": "model-00018.safetensors",
+    "model.layers.16.mlp.switch_mlp.up_proj.biases": "model-00018.safetensors",
+    "model.layers.16.self_attn.embed_q.weight": "model-00018.safetensors",
+    "model.layers.16.self_attn.embed_q.scales": "model-00018.safetensors",
+    "model.layers.16.self_attn.embed_q.biases": "model-00018.safetensors",
+    "model.layers.16.self_attn.unembed_out.weight": "model-00018.safetensors",
+    "model.layers.16.self_attn.unembed_out.scales": "model-00018.safetensors",
+    "model.layers.16.self_attn.unembed_out.biases": "model-00018.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00019.safetensors",
+    "model.layers.17.mlp.gate.e_score_correction_bias": "model-00019.safetensors",
+    "model.layers.17.mlp.gate.weight": "model-00019.safetensors",
+    "model.layers.17.mlp.shared_experts.down_proj.weight": "model-00019.safetensors",
+    "model.layers.17.mlp.shared_experts.down_proj.scales": "model-00019.safetensors",
+    "model.layers.17.mlp.shared_experts.down_proj.biases": "model-00019.safetensors",
+    "model.layers.17.mlp.shared_experts.gate_proj.weight": "model-00019.safetensors",
+    "model.layers.17.mlp.shared_experts.gate_proj.scales": "model-00019.safetensors",
+    "model.layers.17.mlp.shared_experts.gate_proj.biases": "model-00019.safetensors",
+    "model.layers.17.mlp.shared_experts.up_proj.weight": "model-00019.safetensors",
+    "model.layers.17.mlp.shared_experts.up_proj.scales": "model-00019.safetensors",
+    "model.layers.17.mlp.shared_experts.up_proj.biases": "model-00019.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00019.safetensors",
+    "model.layers.17.self_attn.kv_a_layernorm.weight": "model-00019.safetensors",
+    "model.layers.17.self_attn.kv_a_proj_with_mqa.weight": "model-00019.safetensors",
+    "model.layers.17.self_attn.kv_a_proj_with_mqa.scales": "model-00019.safetensors",
+    "model.layers.17.self_attn.kv_a_proj_with_mqa.biases": "model-00019.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00019.safetensors",
+    "model.layers.17.self_attn.o_proj.scales": "model-00019.safetensors",
+    "model.layers.17.self_attn.o_proj.biases": "model-00019.safetensors",
+    "model.layers.17.self_attn.q_a_layernorm.weight": "model-00019.safetensors",
+    "model.layers.17.self_attn.q_a_proj.weight": "model-00019.safetensors",
+    "model.layers.17.self_attn.q_a_proj.scales": "model-00019.safetensors",
+    "model.layers.17.self_attn.q_a_proj.biases": "model-00019.safetensors",
+    "model.layers.17.self_attn.q_b_proj.weight": "model-00019.safetensors",
+    "model.layers.17.self_attn.q_b_proj.scales": "model-00019.safetensors",
+    "model.layers.17.self_attn.q_b_proj.biases": "model-00019.safetensors",
+    "model.layers.17.mlp.switch_mlp.gate_proj.weight": "model-00019.safetensors",
+    "model.layers.17.mlp.switch_mlp.gate_proj.scales": "model-00019.safetensors",
+    "model.layers.17.mlp.switch_mlp.gate_proj.biases": "model-00019.safetensors",
+    "model.layers.17.mlp.switch_mlp.down_proj.weight": "model-00019.safetensors",
+    "model.layers.17.mlp.switch_mlp.down_proj.scales": "model-00019.safetensors",
+    "model.layers.17.mlp.switch_mlp.down_proj.biases": "model-00019.safetensors",
+    "model.layers.17.mlp.switch_mlp.up_proj.weight": "model-00019.safetensors",
+    "model.layers.17.mlp.switch_mlp.up_proj.scales": "model-00019.safetensors",
+    "model.layers.17.mlp.switch_mlp.up_proj.biases": "model-00019.safetensors",
+    "model.layers.17.self_attn.embed_q.weight": "model-00019.safetensors",
+    "model.layers.17.self_attn.embed_q.scales": "model-00019.safetensors",
+    "model.layers.17.self_attn.embed_q.biases": "model-00019.safetensors",
+    "model.layers.17.self_attn.unembed_out.weight": "model-00019.safetensors",
+    "model.layers.17.self_attn.unembed_out.scales": "model-00019.safetensors",
+    "model.layers.17.self_attn.unembed_out.biases": "model-00019.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00020.safetensors",
+    "model.layers.18.mlp.gate.e_score_correction_bias": "model-00020.safetensors",
+    "model.layers.18.mlp.gate.weight": "model-00020.safetensors",
+    "model.layers.18.mlp.shared_experts.down_proj.weight": "model-00020.safetensors",
+    "model.layers.18.mlp.shared_experts.down_proj.scales": "model-00020.safetensors",
+    "model.layers.18.mlp.shared_experts.down_proj.biases": "model-00020.safetensors",
+    "model.layers.18.mlp.shared_experts.gate_proj.weight": "model-00020.safetensors",
+    "model.layers.18.mlp.shared_experts.gate_proj.scales": "model-00020.safetensors",
+    "model.layers.18.mlp.shared_experts.gate_proj.biases": "model-00020.safetensors",
+    "model.layers.18.mlp.shared_experts.up_proj.weight": "model-00020.safetensors",
+    "model.layers.18.mlp.shared_experts.up_proj.scales": "model-00020.safetensors",
+    "model.layers.18.mlp.shared_experts.up_proj.biases": "model-00020.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00020.safetensors",
+    "model.layers.18.self_attn.indexer.k_norm.bias": "model-00020.safetensors",
+    "model.layers.18.self_attn.indexer.k_norm.weight": "model-00020.safetensors",
+    "model.layers.18.self_attn.indexer.weights_proj.weight": "model-00020.safetensors",
+    "model.layers.18.self_attn.indexer.wk.weight": "model-00020.safetensors",
+    "model.layers.18.self_attn.indexer.wq_b.weight": "model-00020.safetensors",
+    "model.layers.18.self_attn.kv_a_layernorm.weight": "model-00020.safetensors",
+    "model.layers.18.self_attn.kv_a_proj_with_mqa.weight": "model-00020.safetensors",
+    "model.layers.18.self_attn.kv_a_proj_with_mqa.scales": "model-00020.safetensors",
+    "model.layers.18.self_attn.kv_a_proj_with_mqa.biases": "model-00020.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00020.safetensors",
+    "model.layers.18.self_attn.o_proj.scales": "model-00020.safetensors",
+    "model.layers.18.self_attn.o_proj.biases": "model-00020.safetensors",
+    "model.layers.18.self_attn.q_a_layernorm.weight": "model-00020.safetensors",
+    "model.layers.18.self_attn.q_a_proj.weight": "model-00020.safetensors",
+    "model.layers.18.self_attn.q_a_proj.scales": "model-00020.safetensors",
+    "model.layers.18.self_attn.q_a_proj.biases": "model-00020.safetensors",
+    "model.layers.18.self_attn.q_b_proj.weight": "model-00020.safetensors",
+    "model.layers.18.self_attn.q_b_proj.scales": "model-00020.safetensors",
+    "model.layers.18.self_attn.q_b_proj.biases": "model-00020.safetensors",
+    "model.layers.18.mlp.switch_mlp.gate_proj.weight": "model-00020.safetensors",
+    "model.layers.18.mlp.switch_mlp.gate_proj.scales": "model-00020.safetensors",
+    "model.layers.18.mlp.switch_mlp.gate_proj.biases": "model-00020.safetensors",
+    "model.layers.18.mlp.switch_mlp.down_proj.weight": "model-00020.safetensors",
+    "model.layers.18.mlp.switch_mlp.down_proj.scales": "model-00020.safetensors",
+    "model.layers.18.mlp.switch_mlp.down_proj.biases": "model-00020.safetensors",
+    "model.layers.18.mlp.switch_mlp.up_proj.weight": "model-00020.safetensors",
+    "model.layers.18.mlp.switch_mlp.up_proj.scales": "model-00020.safetensors",
+    "model.layers.18.mlp.switch_mlp.up_proj.biases": "model-00020.safetensors",
+    "model.layers.18.self_attn.embed_q.weight": "model-00020.safetensors",
+    "model.layers.18.self_attn.embed_q.scales": "model-00020.safetensors",
+    "model.layers.18.self_attn.embed_q.biases": "model-00020.safetensors",
+    "model.layers.18.self_attn.unembed_out.weight": "model-00020.safetensors",
+    "model.layers.18.self_attn.unembed_out.scales": "model-00020.safetensors",
+    "model.layers.18.self_attn.unembed_out.biases": "model-00020.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00021.safetensors",
+    "model.layers.19.mlp.gate.e_score_correction_bias": "model-00021.safetensors",
+    "model.layers.19.mlp.gate.weight": "model-00021.safetensors",
+    "model.layers.19.mlp.shared_experts.down_proj.weight": "model-00021.safetensors",
+    "model.layers.19.mlp.shared_experts.down_proj.scales": "model-00021.safetensors",
+    "model.layers.19.mlp.shared_experts.down_proj.biases": "model-00021.safetensors",
+    "model.layers.19.mlp.shared_experts.gate_proj.weight": "model-00021.safetensors",
+    "model.layers.19.mlp.shared_experts.gate_proj.scales": "model-00021.safetensors",
+    "model.layers.19.mlp.shared_experts.gate_proj.biases": "model-00021.safetensors",
+    "model.layers.19.mlp.shared_experts.up_proj.weight": "model-00021.safetensors",
+    "model.layers.19.mlp.shared_experts.up_proj.scales": "model-00021.safetensors",
+    "model.layers.19.mlp.shared_experts.up_proj.biases": "model-00021.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00021.safetensors",
+    "model.layers.19.self_attn.kv_a_layernorm.weight": "model-00021.safetensors",
+    "model.layers.19.self_attn.kv_a_proj_with_mqa.weight": "model-00021.safetensors",
+    "model.layers.19.self_attn.kv_a_proj_with_mqa.scales": "model-00021.safetensors",
+    "model.layers.19.self_attn.kv_a_proj_with_mqa.biases": "model-00021.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00021.safetensors",
+    "model.layers.19.self_attn.o_proj.scales": "model-00021.safetensors",
+    "model.layers.19.self_attn.o_proj.biases": "model-00021.safetensors",
+    "model.layers.19.self_attn.q_a_layernorm.weight": "model-00021.safetensors",
+    "model.layers.19.self_attn.q_a_proj.weight": "model-00021.safetensors",
+    "model.layers.19.self_attn.q_a_proj.scales": "model-00021.safetensors",
+    "model.layers.19.self_attn.q_a_proj.biases": "model-00021.safetensors",
+    "model.layers.19.self_attn.q_b_proj.weight": "model-00021.safetensors",
+    "model.layers.19.self_attn.q_b_proj.scales": "model-00021.safetensors",
+    "model.layers.19.self_attn.q_b_proj.biases": "model-00021.safetensors",
+    "model.layers.19.mlp.switch_mlp.gate_proj.weight": "model-00021.safetensors",
+    "model.layers.19.mlp.switch_mlp.gate_proj.scales": "model-00021.safetensors",
+    "model.layers.19.mlp.switch_mlp.gate_proj.biases": "model-00021.safetensors",
+    "model.layers.19.mlp.switch_mlp.down_proj.weight": "model-00021.safetensors",
+    "model.layers.19.mlp.switch_mlp.down_proj.scales": "model-00021.safetensors",
+    "model.layers.19.mlp.switch_mlp.down_proj.biases": "model-00021.safetensors",
+    "model.layers.19.mlp.switch_mlp.up_proj.weight": "model-00021.safetensors",
+    "model.layers.19.mlp.switch_mlp.up_proj.scales": "model-00021.safetensors",
+    "model.layers.19.mlp.switch_mlp.up_proj.biases": "model-00021.safetensors",
+    "model.layers.19.self_attn.embed_q.weight": "model-00021.safetensors",
+    "model.layers.19.self_attn.embed_q.scales": "model-00021.safetensors",
+    "model.layers.19.self_attn.embed_q.biases": "model-00021.safetensors",
+    "model.layers.19.self_attn.unembed_out.weight": "model-00021.safetensors",
+    "model.layers.19.self_attn.unembed_out.scales": "model-00021.safetensors",
+    "model.layers.19.self_attn.unembed_out.biases": "model-00021.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00022.safetensors",
+    "model.layers.20.mlp.gate.e_score_correction_bias": "model-00022.safetensors",
+    "model.layers.20.mlp.gate.weight": "model-00022.safetensors",
+    "model.layers.20.mlp.shared_experts.down_proj.weight": "model-00022.safetensors",
+    "model.layers.20.mlp.shared_experts.down_proj.scales": "model-00022.safetensors",
+    "model.layers.20.mlp.shared_experts.down_proj.biases": "model-00022.safetensors",
+    "model.layers.20.mlp.shared_experts.gate_proj.weight": "model-00022.safetensors",
+    "model.layers.20.mlp.shared_experts.gate_proj.scales": "model-00022.safetensors",
+    "model.layers.20.mlp.shared_experts.gate_proj.biases": "model-00022.safetensors",
+    "model.layers.20.mlp.shared_experts.up_proj.weight": "model-00022.safetensors",
+    "model.layers.20.mlp.shared_experts.up_proj.scales": "model-00022.safetensors",
+    "model.layers.20.mlp.shared_experts.up_proj.biases": "model-00022.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00022.safetensors",
+    "model.layers.20.self_attn.kv_a_layernorm.weight": "model-00022.safetensors",
+    "model.layers.20.self_attn.kv_a_proj_with_mqa.weight": "model-00022.safetensors",
+    "model.layers.20.self_attn.kv_a_proj_with_mqa.scales": "model-00022.safetensors",
+    "model.layers.20.self_attn.kv_a_proj_with_mqa.biases": "model-00022.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00022.safetensors",
+    "model.layers.20.self_attn.o_proj.scales": "model-00022.safetensors",
+    "model.layers.20.self_attn.o_proj.biases": "model-00022.safetensors",
+    "model.layers.20.self_attn.q_a_layernorm.weight": "model-00022.safetensors",
+    "model.layers.20.self_attn.q_a_proj.weight": "model-00022.safetensors",
+    "model.layers.20.self_attn.q_a_proj.scales": "model-00022.safetensors",
+    "model.layers.20.self_attn.q_a_proj.biases": "model-00022.safetensors",
+    "model.layers.20.self_attn.q_b_proj.weight": "model-00022.safetensors",
+    "model.layers.20.self_attn.q_b_proj.scales": "model-00022.safetensors",
+    "model.layers.20.self_attn.q_b_proj.biases": "model-00022.safetensors",
+    "model.layers.20.mlp.switch_mlp.gate_proj.weight": "model-00022.safetensors",
+    "model.layers.20.mlp.switch_mlp.gate_proj.scales": "model-00022.safetensors",
+    "model.layers.20.mlp.switch_mlp.gate_proj.biases": "model-00022.safetensors",
+    "model.layers.20.mlp.switch_mlp.down_proj.weight": "model-00022.safetensors",
+    "model.layers.20.mlp.switch_mlp.down_proj.scales": "model-00022.safetensors",
+    "model.layers.20.mlp.switch_mlp.down_proj.biases": "model-00022.safetensors",
+    "model.layers.20.mlp.switch_mlp.up_proj.weight": "model-00022.safetensors",
+    "model.layers.20.mlp.switch_mlp.up_proj.scales": "model-00022.safetensors",
+    "model.layers.20.mlp.switch_mlp.up_proj.biases": "model-00022.safetensors",
+    "model.layers.20.self_attn.embed_q.weight": "model-00022.safetensors",
+    "model.layers.20.self_attn.embed_q.scales": "model-00022.safetensors",
+    "model.layers.20.self_attn.embed_q.biases": "model-00022.safetensors",
+    "model.layers.20.self_attn.unembed_out.weight": "model-00022.safetensors",
+    "model.layers.20.self_attn.unembed_out.scales": "model-00022.safetensors",
+    "model.layers.20.self_attn.unembed_out.biases": "model-00022.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00023.safetensors",
+    "model.layers.21.mlp.gate.e_score_correction_bias": "model-00023.safetensors",
+    "model.layers.21.mlp.gate.weight": "model-00023.safetensors",
+    "model.layers.21.mlp.shared_experts.down_proj.weight": "model-00023.safetensors",
+    "model.layers.21.mlp.shared_experts.down_proj.scales": "model-00023.safetensors",
+    "model.layers.21.mlp.shared_experts.down_proj.biases": "model-00023.safetensors",
+    "model.layers.21.mlp.shared_experts.gate_proj.weight": "model-00023.safetensors",
+    "model.layers.21.mlp.shared_experts.gate_proj.scales": "model-00023.safetensors",
+    "model.layers.21.mlp.shared_experts.gate_proj.biases": "model-00023.safetensors",
+    "model.layers.21.mlp.shared_experts.up_proj.weight": "model-00023.safetensors",
+    "model.layers.21.mlp.shared_experts.up_proj.scales": "model-00023.safetensors",
+    "model.layers.21.mlp.shared_experts.up_proj.biases": "model-00023.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00023.safetensors",
+    "model.layers.21.self_attn.kv_a_layernorm.weight": "model-00023.safetensors",
+    "model.layers.21.self_attn.kv_a_proj_with_mqa.weight": "model-00023.safetensors",
+    "model.layers.21.self_attn.kv_a_proj_with_mqa.scales": "model-00023.safetensors",
+    "model.layers.21.self_attn.kv_a_proj_with_mqa.biases": "model-00023.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00023.safetensors",
+    "model.layers.21.self_attn.o_proj.scales": "model-00023.safetensors",
+    "model.layers.21.self_attn.o_proj.biases": "model-00023.safetensors",
+    "model.layers.21.self_attn.q_a_layernorm.weight": "model-00023.safetensors",
+    "model.layers.21.self_attn.q_a_proj.weight": "model-00023.safetensors",
+    "model.layers.21.self_attn.q_a_proj.scales": "model-00023.safetensors",
+    "model.layers.21.self_attn.q_a_proj.biases": "model-00023.safetensors",
+    "model.layers.21.self_attn.q_b_proj.weight": "model-00023.safetensors",
+    "model.layers.21.self_attn.q_b_proj.scales": "model-00023.safetensors",
+    "model.layers.21.self_attn.q_b_proj.biases": "model-00023.safetensors",
+    "model.layers.21.mlp.switch_mlp.gate_proj.weight": "model-00023.safetensors",
+    "model.layers.21.mlp.switch_mlp.gate_proj.scales": "model-00023.safetensors",
+    "model.layers.21.mlp.switch_mlp.gate_proj.biases": "model-00023.safetensors",
+    "model.layers.21.mlp.switch_mlp.down_proj.weight": "model-00023.safetensors",
+    "model.layers.21.mlp.switch_mlp.down_proj.scales": "model-00023.safetensors",
+    "model.layers.21.mlp.switch_mlp.down_proj.biases": "model-00023.safetensors",
+    "model.layers.21.mlp.switch_mlp.up_proj.weight": "model-00023.safetensors",
+    "model.layers.21.mlp.switch_mlp.up_proj.scales": "model-00023.safetensors",
+    "model.layers.21.mlp.switch_mlp.up_proj.biases": "model-00023.safetensors",
+    "model.layers.21.self_attn.embed_q.weight": "model-00023.safetensors",
+    "model.layers.21.self_attn.embed_q.scales": "model-00023.safetensors",
+    "model.layers.21.self_attn.embed_q.biases": "model-00023.safetensors",
+    "model.layers.21.self_attn.unembed_out.weight": "model-00023.safetensors",
+    "model.layers.21.self_attn.unembed_out.scales": "model-00023.safetensors",
+    "model.layers.21.self_attn.unembed_out.biases": "model-00023.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00024.safetensors",
+    "model.layers.22.mlp.gate.e_score_correction_bias": "model-00024.safetensors",
+    "model.layers.22.mlp.gate.weight": "model-00024.safetensors",
+    "model.layers.22.mlp.shared_experts.down_proj.weight": "model-00024.safetensors",
+    "model.layers.22.mlp.shared_experts.down_proj.scales": "model-00024.safetensors",
+    "model.layers.22.mlp.shared_experts.down_proj.biases": "model-00024.safetensors",
+    "model.layers.22.mlp.shared_experts.gate_proj.weight": "model-00024.safetensors",
+    "model.layers.22.mlp.shared_experts.gate_proj.scales": "model-00024.safetensors",
+    "model.layers.22.mlp.shared_experts.gate_proj.biases": "model-00024.safetensors",
+    "model.layers.22.mlp.shared_experts.up_proj.weight": "model-00024.safetensors",
+    "model.layers.22.mlp.shared_experts.up_proj.scales": "model-00024.safetensors",
+    "model.layers.22.mlp.shared_experts.up_proj.biases": "model-00024.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00024.safetensors",
+    "model.layers.22.self_attn.indexer.k_norm.bias": "model-00024.safetensors",
+    "model.layers.22.self_attn.indexer.k_norm.weight": "model-00024.safetensors",
+    "model.layers.22.self_attn.indexer.weights_proj.weight": "model-00024.safetensors",
+    "model.layers.22.self_attn.indexer.wk.weight": "model-00024.safetensors",
+    "model.layers.22.self_attn.indexer.wq_b.weight": "model-00024.safetensors",
+    "model.layers.22.self_attn.kv_a_layernorm.weight": "model-00024.safetensors",
+    "model.layers.22.self_attn.kv_a_proj_with_mqa.weight": "model-00024.safetensors",
+    "model.layers.22.self_attn.kv_a_proj_with_mqa.scales": "model-00024.safetensors",
+    "model.layers.22.self_attn.kv_a_proj_with_mqa.biases": "model-00024.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00024.safetensors",
+    "model.layers.22.self_attn.o_proj.scales": "model-00024.safetensors",
+    "model.layers.22.self_attn.o_proj.biases": "model-00024.safetensors",
+    "model.layers.22.self_attn.q_a_layernorm.weight": "model-00024.safetensors",
+    "model.layers.22.self_attn.q_a_proj.weight": "model-00024.safetensors",
+    "model.layers.22.self_attn.q_a_proj.scales": "model-00024.safetensors",
+    "model.layers.22.self_attn.q_a_proj.biases": "model-00024.safetensors",
+    "model.layers.22.self_attn.q_b_proj.weight": "model-00024.safetensors",
+    "model.layers.22.self_attn.q_b_proj.scales": "model-00024.safetensors",
+    "model.layers.22.self_attn.q_b_proj.biases": "model-00024.safetensors",
+    "model.layers.22.mlp.switch_mlp.gate_proj.weight": "model-00024.safetensors",
+    "model.layers.22.mlp.switch_mlp.gate_proj.scales": "model-00024.safetensors",
+    "model.layers.22.mlp.switch_mlp.gate_proj.biases": "model-00024.safetensors",
+    "model.layers.22.mlp.switch_mlp.down_proj.weight": "model-00024.safetensors",
+    "model.layers.22.mlp.switch_mlp.down_proj.scales": "model-00024.safetensors",
+    "model.layers.22.mlp.switch_mlp.down_proj.biases": "model-00024.safetensors",
+    "model.layers.22.mlp.switch_mlp.up_proj.weight": "model-00024.safetensors",
+    "model.layers.22.mlp.switch_mlp.up_proj.scales": "model-00024.safetensors",
+    "model.layers.22.mlp.switch_mlp.up_proj.biases": "model-00024.safetensors",
+    "model.layers.22.self_attn.embed_q.weight": "model-00024.safetensors",
+    "model.layers.22.self_attn.embed_q.scales": "model-00024.safetensors",
+    "model.layers.22.self_attn.embed_q.biases": "model-00024.safetensors",
+    "model.layers.22.self_attn.unembed_out.weight": "model-00024.safetensors",
+    "model.layers.22.self_attn.unembed_out.scales": "model-00024.safetensors",
+    "model.layers.22.self_attn.unembed_out.biases": "model-00024.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00025.safetensors",
+    "model.layers.23.mlp.gate.e_score_correction_bias": "model-00025.safetensors",
+    "model.layers.23.mlp.gate.weight": "model-00025.safetensors",
+    "model.layers.23.mlp.shared_experts.down_proj.weight": "model-00025.safetensors",
+    "model.layers.23.mlp.shared_experts.down_proj.scales": "model-00025.safetensors",
+    "model.layers.23.mlp.shared_experts.down_proj.biases": "model-00025.safetensors",
+    "model.layers.23.mlp.shared_experts.gate_proj.weight": "model-00025.safetensors",
+    "model.layers.23.mlp.shared_experts.gate_proj.scales": "model-00025.safetensors",
+    "model.layers.23.mlp.shared_experts.gate_proj.biases": "model-00025.safetensors",
+    "model.layers.23.mlp.shared_experts.up_proj.weight": "model-00025.safetensors",
+    "model.layers.23.mlp.shared_experts.up_proj.scales": "model-00025.safetensors",
+    "model.layers.23.mlp.shared_experts.up_proj.biases": "model-00025.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00025.safetensors",
+    "model.layers.23.self_attn.kv_a_layernorm.weight": "model-00025.safetensors",
+    "model.layers.23.self_attn.kv_a_proj_with_mqa.weight": "model-00025.safetensors",
+    "model.layers.23.self_attn.kv_a_proj_with_mqa.scales": "model-00025.safetensors",
+    "model.layers.23.self_attn.kv_a_proj_with_mqa.biases": "model-00025.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00025.safetensors",
+    "model.layers.23.self_attn.o_proj.scales": "model-00025.safetensors",
+    "model.layers.23.self_attn.o_proj.biases": "model-00025.safetensors",
+    "model.layers.23.self_attn.q_a_layernorm.weight": "model-00025.safetensors",
+    "model.layers.23.self_attn.q_a_proj.weight": "model-00025.safetensors",
+    "model.layers.23.self_attn.q_a_proj.scales": "model-00025.safetensors",
+    "model.layers.23.self_attn.q_a_proj.biases": "model-00025.safetensors",
+    "model.layers.23.self_attn.q_b_proj.weight": "model-00025.safetensors",
+    "model.layers.23.self_attn.q_b_proj.scales": "model-00025.safetensors",
+    "model.layers.23.self_attn.q_b_proj.biases": "model-00025.safetensors",
+    "model.layers.23.mlp.switch_mlp.gate_proj.weight": "model-00025.safetensors",
+    "model.layers.23.mlp.switch_mlp.gate_proj.scales": "model-00025.safetensors",
+    "model.layers.23.mlp.switch_mlp.gate_proj.biases": "model-00025.safetensors",
+    "model.layers.23.mlp.switch_mlp.down_proj.weight": "model-00025.safetensors",
+    "model.layers.23.mlp.switch_mlp.down_proj.scales": "model-00025.safetensors",
+    "model.layers.23.mlp.switch_mlp.down_proj.biases": "model-00025.safetensors",
+    "model.layers.23.mlp.switch_mlp.up_proj.weight": "model-00025.safetensors",
+    "model.layers.23.mlp.switch_mlp.up_proj.scales": "model-00025.safetensors",
+    "model.layers.23.mlp.switch_mlp.up_proj.biases": "model-00025.safetensors",
+    "model.layers.23.self_attn.embed_q.weight": "model-00025.safetensors",
+    "model.layers.23.self_attn.embed_q.scales": "model-00025.safetensors",
+    "model.layers.23.self_attn.embed_q.biases": "model-00025.safetensors",
+    "model.layers.23.self_attn.unembed_out.weight": "model-00025.safetensors",
+    "model.layers.23.self_attn.unembed_out.scales": "model-00025.safetensors",
+    "model.layers.23.self_attn.unembed_out.biases": "model-00025.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00026.safetensors",
+    "model.layers.24.mlp.gate.e_score_correction_bias": "model-00026.safetensors",
+    "model.layers.24.mlp.gate.weight": "model-00026.safetensors",
+    "model.layers.24.mlp.shared_experts.down_proj.weight": "model-00026.safetensors",
+    "model.layers.24.mlp.shared_experts.down_proj.scales": "model-00026.safetensors",
+    "model.layers.24.mlp.shared_experts.down_proj.biases": "model-00026.safetensors",
+    "model.layers.24.mlp.shared_experts.gate_proj.weight": "model-00026.safetensors",
+    "model.layers.24.mlp.shared_experts.gate_proj.scales": "model-00026.safetensors",
+    "model.layers.24.mlp.shared_experts.gate_proj.biases": "model-00026.safetensors",
+    "model.layers.24.mlp.shared_experts.up_proj.weight": "model-00026.safetensors",
+    "model.layers.24.mlp.shared_experts.up_proj.scales": "model-00026.safetensors",
+    "model.layers.24.mlp.shared_experts.up_proj.biases": "model-00026.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00026.safetensors",
+    "model.layers.24.self_attn.kv_a_layernorm.weight": "model-00026.safetensors",
+    "model.layers.24.self_attn.kv_a_proj_with_mqa.weight": "model-00026.safetensors",
+    "model.layers.24.self_attn.kv_a_proj_with_mqa.scales": "model-00026.safetensors",
+    "model.layers.24.self_attn.kv_a_proj_with_mqa.biases": "model-00026.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00026.safetensors",
+    "model.layers.24.self_attn.o_proj.scales": "model-00026.safetensors",
+    "model.layers.24.self_attn.o_proj.biases": "model-00026.safetensors",
+    "model.layers.24.self_attn.q_a_layernorm.weight": "model-00026.safetensors",
+    "model.layers.24.self_attn.q_a_proj.weight": "model-00026.safetensors",
+    "model.layers.24.self_attn.q_a_proj.scales": "model-00026.safetensors",
+    "model.layers.24.self_attn.q_a_proj.biases": "model-00026.safetensors",
+    "model.layers.24.self_attn.q_b_proj.weight": "model-00026.safetensors",
+    "model.layers.24.self_attn.q_b_proj.scales": "model-00026.safetensors",
+    "model.layers.24.self_attn.q_b_proj.biases": "model-00026.safetensors",
+    "model.layers.24.mlp.switch_mlp.gate_proj.weight": "model-00026.safetensors",
+    "model.layers.24.mlp.switch_mlp.gate_proj.scales": "model-00026.safetensors",
+    "model.layers.24.mlp.switch_mlp.gate_proj.biases": "model-00026.safetensors",
+    "model.layers.24.mlp.switch_mlp.down_proj.weight": "model-00026.safetensors",
+    "model.layers.24.mlp.switch_mlp.down_proj.scales": "model-00026.safetensors",
+    "model.layers.24.mlp.switch_mlp.down_proj.biases": "model-00026.safetensors",
+    "model.layers.24.mlp.switch_mlp.up_proj.weight": "model-00026.safetensors",
+    "model.layers.24.mlp.switch_mlp.up_proj.scales": "model-00026.safetensors",
+    "model.layers.24.mlp.switch_mlp.up_proj.biases": "model-00026.safetensors",
+    "model.layers.24.self_attn.embed_q.weight": "model-00026.safetensors",
+    "model.layers.24.self_attn.embed_q.scales": "model-00026.safetensors",
+    "model.layers.24.self_attn.embed_q.biases": "model-00026.safetensors",
+    "model.layers.24.self_attn.unembed_out.weight": "model-00026.safetensors",
+    "model.layers.24.self_attn.unembed_out.scales": "model-00026.safetensors",
+    "model.layers.24.self_attn.unembed_out.biases": "model-00026.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00027.safetensors",
+    "model.layers.25.mlp.gate.e_score_correction_bias": "model-00027.safetensors",
+    "model.layers.25.mlp.gate.weight": "model-00027.safetensors",
+    "model.layers.25.mlp.shared_experts.down_proj.weight": "model-00027.safetensors",
+    "model.layers.25.mlp.shared_experts.down_proj.scales": "model-00027.safetensors",
+    "model.layers.25.mlp.shared_experts.down_proj.biases": "model-00027.safetensors",
+    "model.layers.25.mlp.shared_experts.gate_proj.weight": "model-00027.safetensors",
+    "model.layers.25.mlp.shared_experts.gate_proj.scales": "model-00027.safetensors",
+    "model.layers.25.mlp.shared_experts.gate_proj.biases": "model-00027.safetensors",
+    "model.layers.25.mlp.shared_experts.up_proj.weight": "model-00027.safetensors",
+    "model.layers.25.mlp.shared_experts.up_proj.scales": "model-00027.safetensors",
+    "model.layers.25.mlp.shared_experts.up_proj.biases": "model-00027.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00027.safetensors",
+    "model.layers.25.self_attn.kv_a_layernorm.weight": "model-00027.safetensors",
+    "model.layers.25.self_attn.kv_a_proj_with_mqa.weight": "model-00027.safetensors",
+    "model.layers.25.self_attn.kv_a_proj_with_mqa.scales": "model-00027.safetensors",
+    "model.layers.25.self_attn.kv_a_proj_with_mqa.biases": "model-00027.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00027.safetensors",
+    "model.layers.25.self_attn.o_proj.scales": "model-00027.safetensors",
+    "model.layers.25.self_attn.o_proj.biases": "model-00027.safetensors",
+    "model.layers.25.self_attn.q_a_layernorm.weight": "model-00027.safetensors",
+    "model.layers.25.self_attn.q_a_proj.weight": "model-00027.safetensors",
+    "model.layers.25.self_attn.q_a_proj.scales": "model-00027.safetensors",
+    "model.layers.25.self_attn.q_a_proj.biases": "model-00027.safetensors",
+    "model.layers.25.self_attn.q_b_proj.weight": "model-00027.safetensors",
+    "model.layers.25.self_attn.q_b_proj.scales": "model-00027.safetensors",
+    "model.layers.25.self_attn.q_b_proj.biases": "model-00027.safetensors",
+    "model.layers.25.mlp.switch_mlp.gate_proj.weight": "model-00027.safetensors",
+    "model.layers.25.mlp.switch_mlp.gate_proj.scales": "model-00027.safetensors",
+    "model.layers.25.mlp.switch_mlp.gate_proj.biases": "model-00027.safetensors",
+    "model.layers.25.mlp.switch_mlp.down_proj.weight": "model-00027.safetensors",
+    "model.layers.25.mlp.switch_mlp.down_proj.scales": "model-00027.safetensors",
+    "model.layers.25.mlp.switch_mlp.down_proj.biases": "model-00027.safetensors",
+    "model.layers.25.mlp.switch_mlp.up_proj.weight": "model-00027.safetensors",
+    "model.layers.25.mlp.switch_mlp.up_proj.scales": "model-00027.safetensors",
+    "model.layers.25.mlp.switch_mlp.up_proj.biases": "model-00027.safetensors",
+    "model.layers.25.self_attn.embed_q.weight": "model-00027.safetensors",
+    "model.layers.25.self_attn.embed_q.scales": "model-00027.safetensors",
+    "model.layers.25.self_attn.embed_q.biases": "model-00027.safetensors",
+    "model.layers.25.self_attn.unembed_out.weight": "model-00027.safetensors",
+    "model.layers.25.self_attn.unembed_out.scales": "model-00027.safetensors",
+    "model.layers.25.self_attn.unembed_out.biases": "model-00027.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-00028.safetensors",
+    "model.layers.26.mlp.gate.e_score_correction_bias": "model-00028.safetensors",
+    "model.layers.26.mlp.gate.weight": "model-00028.safetensors",
+    "model.layers.26.mlp.shared_experts.down_proj.weight": "model-00028.safetensors",
+    "model.layers.26.mlp.shared_experts.down_proj.scales": "model-00028.safetensors",
+    "model.layers.26.mlp.shared_experts.down_proj.biases": "model-00028.safetensors",
+    "model.layers.26.mlp.shared_experts.gate_proj.weight": "model-00028.safetensors",
+    "model.layers.26.mlp.shared_experts.gate_proj.scales": "model-00028.safetensors",
+    "model.layers.26.mlp.shared_experts.gate_proj.biases": "model-00028.safetensors",
+    "model.layers.26.mlp.shared_experts.up_proj.weight": "model-00028.safetensors",
+    "model.layers.26.mlp.shared_experts.up_proj.scales": "model-00028.safetensors",
+    "model.layers.26.mlp.shared_experts.up_proj.biases": "model-00028.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00028.safetensors",
+    "model.layers.26.self_attn.indexer.k_norm.bias": "model-00028.safetensors",
+    "model.layers.26.self_attn.indexer.k_norm.weight": "model-00028.safetensors",
+    "model.layers.26.self_attn.indexer.weights_proj.weight": "model-00028.safetensors",
+    "model.layers.26.self_attn.indexer.wk.weight": "model-00028.safetensors",
+    "model.layers.26.self_attn.indexer.wq_b.weight": "model-00028.safetensors",
+    "model.layers.26.self_attn.kv_a_layernorm.weight": "model-00028.safetensors",
+    "model.layers.26.self_attn.kv_a_proj_with_mqa.weight": "model-00028.safetensors",
+    "model.layers.26.self_attn.kv_a_proj_with_mqa.scales": "model-00028.safetensors",
+    "model.layers.26.self_attn.kv_a_proj_with_mqa.biases": "model-00028.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00028.safetensors",
+    "model.layers.26.self_attn.o_proj.scales": "model-00028.safetensors",
+    "model.layers.26.self_attn.o_proj.biases": "model-00028.safetensors",
+    "model.layers.26.self_attn.q_a_layernorm.weight": "model-00028.safetensors",
+    "model.layers.26.self_attn.q_a_proj.weight": "model-00028.safetensors",
+    "model.layers.26.self_attn.q_a_proj.scales": "model-00028.safetensors",
+    "model.layers.26.self_attn.q_a_proj.biases": "model-00028.safetensors",
+    "model.layers.26.self_attn.q_b_proj.weight": "model-00028.safetensors",
+    "model.layers.26.self_attn.q_b_proj.scales": "model-00028.safetensors",
+    "model.layers.26.self_attn.q_b_proj.biases": "model-00028.safetensors",
+    "model.layers.26.mlp.switch_mlp.gate_proj.weight": "model-00028.safetensors",
+    "model.layers.26.mlp.switch_mlp.gate_proj.scales": "model-00028.safetensors",
+    "model.layers.26.mlp.switch_mlp.gate_proj.biases": "model-00028.safetensors",
+    "model.layers.26.mlp.switch_mlp.down_proj.weight": "model-00028.safetensors",
+    "model.layers.26.mlp.switch_mlp.down_proj.scales": "model-00028.safetensors",
+    "model.layers.26.mlp.switch_mlp.down_proj.biases": "model-00028.safetensors",
+    "model.layers.26.mlp.switch_mlp.up_proj.weight": "model-00028.safetensors",
+    "model.layers.26.mlp.switch_mlp.up_proj.scales": "model-00028.safetensors",
+    "model.layers.26.mlp.switch_mlp.up_proj.biases": "model-00028.safetensors",
+    "model.layers.26.self_attn.embed_q.weight": "model-00028.safetensors",
+    "model.layers.26.self_attn.embed_q.scales": "model-00028.safetensors",
+    "model.layers.26.self_attn.embed_q.biases": "model-00028.safetensors",
+    "model.layers.26.self_attn.unembed_out.weight": "model-00028.safetensors",
+    "model.layers.26.self_attn.unembed_out.scales": "model-00028.safetensors",
+    "model.layers.26.self_attn.unembed_out.biases": "model-00028.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-00029.safetensors",
+    "model.layers.27.mlp.gate.e_score_correction_bias": "model-00029.safetensors",
+    "model.layers.27.mlp.gate.weight": "model-00029.safetensors",
+    "model.layers.27.mlp.shared_experts.down_proj.weight": "model-00029.safetensors",
+    "model.layers.27.mlp.shared_experts.down_proj.scales": "model-00029.safetensors",
+    "model.layers.27.mlp.shared_experts.down_proj.biases": "model-00029.safetensors",
+    "model.layers.27.mlp.shared_experts.gate_proj.weight": "model-00029.safetensors",
+    "model.layers.27.mlp.shared_experts.gate_proj.scales": "model-00029.safetensors",
+    "model.layers.27.mlp.shared_experts.gate_proj.biases": "model-00029.safetensors",
+    "model.layers.27.mlp.shared_experts.up_proj.weight": "model-00029.safetensors",
+    "model.layers.27.mlp.shared_experts.up_proj.scales": "model-00029.safetensors",
+    "model.layers.27.mlp.shared_experts.up_proj.biases": "model-00029.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00029.safetensors",
+    "model.layers.27.self_attn.kv_a_layernorm.weight": "model-00029.safetensors",
+    "model.layers.27.self_attn.kv_a_proj_with_mqa.weight": "model-00029.safetensors",
+    "model.layers.27.self_attn.kv_a_proj_with_mqa.scales": "model-00029.safetensors",
+    "model.layers.27.self_attn.kv_a_proj_with_mqa.biases": "model-00029.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00029.safetensors",
+    "model.layers.27.self_attn.o_proj.scales": "model-00029.safetensors",
+    "model.layers.27.self_attn.o_proj.biases": "model-00029.safetensors",
+    "model.layers.27.self_attn.q_a_layernorm.weight": "model-00029.safetensors",
+    "model.layers.27.self_attn.q_a_proj.weight": "model-00029.safetensors",
+    "model.layers.27.self_attn.q_a_proj.scales": "model-00029.safetensors",
+    "model.layers.27.self_attn.q_a_proj.biases": "model-00029.safetensors",
+    "model.layers.27.self_attn.q_b_proj.weight": "model-00029.safetensors",
+    "model.layers.27.self_attn.q_b_proj.scales": "model-00029.safetensors",
+    "model.layers.27.self_attn.q_b_proj.biases": "model-00029.safetensors",
+    "model.layers.27.mlp.switch_mlp.gate_proj.weight": "model-00029.safetensors",
+    "model.layers.27.mlp.switch_mlp.gate_proj.scales": "model-00029.safetensors",
+    "model.layers.27.mlp.switch_mlp.gate_proj.biases": "model-00029.safetensors",
+    "model.layers.27.mlp.switch_mlp.down_proj.weight": "model-00029.safetensors",
+    "model.layers.27.mlp.switch_mlp.down_proj.scales": "model-00029.safetensors",
+    "model.layers.27.mlp.switch_mlp.down_proj.biases": "model-00029.safetensors",
+    "model.layers.27.mlp.switch_mlp.up_proj.weight": "model-00029.safetensors",
+    "model.layers.27.mlp.switch_mlp.up_proj.scales": "model-00029.safetensors",
+    "model.layers.27.mlp.switch_mlp.up_proj.biases": "model-00029.safetensors",
+    "model.layers.27.self_attn.embed_q.weight": "model-00029.safetensors",
+    "model.layers.27.self_attn.embed_q.scales": "model-00029.safetensors",
+    "model.layers.27.self_attn.embed_q.biases": "model-00029.safetensors",
+    "model.layers.27.self_attn.unembed_out.weight": "model-00029.safetensors",
+    "model.layers.27.self_attn.unembed_out.scales": "model-00029.safetensors",
+    "model.layers.27.self_attn.unembed_out.biases": "model-00029.safetensors",
+    "model.layers.28.input_layernorm.weight": "model-00030.safetensors",
+    "model.layers.28.mlp.gate.e_score_correction_bias": "model-00030.safetensors",
+    "model.layers.28.mlp.gate.weight": "model-00030.safetensors",
+    "model.layers.28.mlp.shared_experts.down_proj.weight": "model-00030.safetensors",
+    "model.layers.28.mlp.shared_experts.down_proj.scales": "model-00030.safetensors",
+    "model.layers.28.mlp.shared_experts.down_proj.biases": "model-00030.safetensors",
+    "model.layers.28.mlp.shared_experts.gate_proj.weight": "model-00030.safetensors",
+    "model.layers.28.mlp.shared_experts.gate_proj.scales": "model-00030.safetensors",
+    "model.layers.28.mlp.shared_experts.gate_proj.biases": "model-00030.safetensors",
+    "model.layers.28.mlp.shared_experts.up_proj.weight": "model-00030.safetensors",
+    "model.layers.28.mlp.shared_experts.up_proj.scales": "model-00030.safetensors",
+    "model.layers.28.mlp.shared_experts.up_proj.biases": "model-00030.safetensors",
+    "model.layers.28.post_attention_layernorm.weight": "model-00030.safetensors",
+    "model.layers.28.self_attn.kv_a_layernorm.weight": "model-00030.safetensors",
+    "model.layers.28.self_attn.kv_a_proj_with_mqa.weight": "model-00030.safetensors",
+    "model.layers.28.self_attn.kv_a_proj_with_mqa.scales": "model-00030.safetensors",
+    "model.layers.28.self_attn.kv_a_proj_with_mqa.biases": "model-00030.safetensors",
+    "model.layers.28.self_attn.o_proj.weight": "model-00030.safetensors",
+    "model.layers.28.self_attn.o_proj.scales": "model-00030.safetensors",
+    "model.layers.28.self_attn.o_proj.biases": "model-00030.safetensors",
+    "model.layers.28.self_attn.q_a_layernorm.weight": "model-00030.safetensors",
+    "model.layers.28.self_attn.q_a_proj.weight": "model-00030.safetensors",
+    "model.layers.28.self_attn.q_a_proj.scales": "model-00030.safetensors",
+    "model.layers.28.self_attn.q_a_proj.biases": "model-00030.safetensors",
+    "model.layers.28.self_attn.q_b_proj.weight": "model-00030.safetensors",
+    "model.layers.28.self_attn.q_b_proj.scales": "model-00030.safetensors",
+    "model.layers.28.self_attn.q_b_proj.biases": "model-00030.safetensors",
+    "model.layers.28.mlp.switch_mlp.gate_proj.weight": "model-00030.safetensors",
+    "model.layers.28.mlp.switch_mlp.gate_proj.scales": "model-00030.safetensors",
+    "model.layers.28.mlp.switch_mlp.gate_proj.biases": "model-00030.safetensors",
+    "model.layers.28.mlp.switch_mlp.down_proj.weight": "model-00030.safetensors",
+    "model.layers.28.mlp.switch_mlp.down_proj.scales": "model-00030.safetensors",
+    "model.layers.28.mlp.switch_mlp.down_proj.biases": "model-00030.safetensors",
+    "model.layers.28.mlp.switch_mlp.up_proj.weight": "model-00030.safetensors",
+    "model.layers.28.mlp.switch_mlp.up_proj.scales": "model-00030.safetensors",
+    "model.layers.28.mlp.switch_mlp.up_proj.biases": "model-00030.safetensors",
+    "model.layers.28.self_attn.embed_q.weight": "model-00030.safetensors",
+    "model.layers.28.self_attn.embed_q.scales": "model-00030.safetensors",
+    "model.layers.28.self_attn.embed_q.biases": "model-00030.safetensors",
+    "model.layers.28.self_attn.unembed_out.weight": "model-00030.safetensors",
+    "model.layers.28.self_attn.unembed_out.scales": "model-00030.safetensors",
+    "model.layers.28.self_attn.unembed_out.biases": "model-00030.safetensors",
+    "model.layers.29.input_layernorm.weight": "model-00031.safetensors",
+    "model.layers.29.mlp.gate.e_score_correction_bias": "model-00031.safetensors",
+    "model.layers.29.mlp.gate.weight": "model-00031.safetensors",
+    "model.layers.29.mlp.shared_experts.down_proj.weight": "model-00031.safetensors",
+    "model.layers.29.mlp.shared_experts.down_proj.scales": "model-00031.safetensors",
+    "model.layers.29.mlp.shared_experts.down_proj.biases": "model-00031.safetensors",
+    "model.layers.29.mlp.shared_experts.gate_proj.weight": "model-00031.safetensors",
+    "model.layers.29.mlp.shared_experts.gate_proj.scales": "model-00031.safetensors",
+    "model.layers.29.mlp.shared_experts.gate_proj.biases": "model-00031.safetensors",
+    "model.layers.29.mlp.shared_experts.up_proj.weight": "model-00031.safetensors",
+    "model.layers.29.mlp.shared_experts.up_proj.scales": "model-00031.safetensors",
+    "model.layers.29.mlp.shared_experts.up_proj.biases": "model-00031.safetensors",
+    "model.layers.29.post_attention_layernorm.weight": "model-00031.safetensors",
+    "model.layers.29.self_attn.kv_a_layernorm.weight": "model-00031.safetensors",
+    "model.layers.29.self_attn.kv_a_proj_with_mqa.weight": "model-00031.safetensors",
+    "model.layers.29.self_attn.kv_a_proj_with_mqa.scales": "model-00031.safetensors",
+    "model.layers.29.self_attn.kv_a_proj_with_mqa.biases": "model-00031.safetensors",
+    "model.layers.29.self_attn.o_proj.weight": "model-00031.safetensors",
+    "model.layers.29.self_attn.o_proj.scales": "model-00031.safetensors",
+    "model.layers.29.self_attn.o_proj.biases": "model-00031.safetensors",
+    "model.layers.29.self_attn.q_a_layernorm.weight": "model-00031.safetensors",
+    "model.layers.29.self_attn.q_a_proj.weight": "model-00031.safetensors",
+    "model.layers.29.self_attn.q_a_proj.scales": "model-00031.safetensors",
+    "model.layers.29.self_attn.q_a_proj.biases": "model-00031.safetensors",
+    "model.layers.29.self_attn.q_b_proj.weight": "model-00031.safetensors",
+    "model.layers.29.self_attn.q_b_proj.scales": "model-00031.safetensors",
+    "model.layers.29.self_attn.q_b_proj.biases": "model-00031.safetensors",
+    "model.layers.29.mlp.switch_mlp.gate_proj.weight": "model-00031.safetensors",
+    "model.layers.29.mlp.switch_mlp.gate_proj.scales": "model-00031.safetensors",
+    "model.layers.29.mlp.switch_mlp.gate_proj.biases": "model-00031.safetensors",
+    "model.layers.29.mlp.switch_mlp.down_proj.weight": "model-00031.safetensors",
+    "model.layers.29.mlp.switch_mlp.down_proj.scales": "model-00031.safetensors",
+    "model.layers.29.mlp.switch_mlp.down_proj.biases": "model-00031.safetensors",
+    "model.layers.29.mlp.switch_mlp.up_proj.weight": "model-00031.safetensors",
+    "model.layers.29.mlp.switch_mlp.up_proj.scales": "model-00031.safetensors",
+    "model.layers.29.mlp.switch_mlp.up_proj.biases": "model-00031.safetensors",
+    "model.layers.29.self_attn.embed_q.weight": "model-00031.safetensors",
+    "model.layers.29.self_attn.embed_q.scales": "model-00031.safetensors",
+    "model.layers.29.self_attn.embed_q.biases": "model-00031.safetensors",
+    "model.layers.29.self_attn.unembed_out.weight": "model-00031.safetensors",
+    "model.layers.29.self_attn.unembed_out.scales": "model-00031.safetensors",
+    "model.layers.29.self_attn.unembed_out.biases": "model-00031.safetensors",
+    "model.layers.30.input_layernorm.weight": "model-00032.safetensors",
+    "model.layers.30.mlp.gate.e_score_correction_bias": "model-00032.safetensors",
+    "model.layers.30.mlp.gate.weight": "model-00032.safetensors",
+    "model.layers.30.mlp.shared_experts.down_proj.weight": "model-00032.safetensors",
+    "model.layers.30.mlp.shared_experts.down_proj.scales": "model-00032.safetensors",
+    "model.layers.30.mlp.shared_experts.down_proj.biases": "model-00032.safetensors",
+    "model.layers.30.mlp.shared_experts.gate_proj.weight": "model-00032.safetensors",
+    "model.layers.30.mlp.shared_experts.gate_proj.scales": "model-00032.safetensors",
+    "model.layers.30.mlp.shared_experts.gate_proj.biases": "model-00032.safetensors",
+    "model.layers.30.mlp.shared_experts.up_proj.weight": "model-00032.safetensors",
+    "model.layers.30.mlp.shared_experts.up_proj.scales": "model-00032.safetensors",
+    "model.layers.30.mlp.shared_experts.up_proj.biases": "model-00032.safetensors",
+    "model.layers.30.post_attention_layernorm.weight": "model-00032.safetensors",
+    "model.layers.30.self_attn.indexer.k_norm.bias": "model-00032.safetensors",
+    "model.layers.30.self_attn.indexer.k_norm.weight": "model-00032.safetensors",
+    "model.layers.30.self_attn.indexer.weights_proj.weight": "model-00032.safetensors",
+    "model.layers.30.self_attn.indexer.wk.weight": "model-00032.safetensors",
+    "model.layers.30.self_attn.indexer.wq_b.weight": "model-00032.safetensors",
+    "model.layers.30.self_attn.kv_a_layernorm.weight": "model-00032.safetensors",
+    "model.layers.30.self_attn.kv_a_proj_with_mqa.weight": "model-00032.safetensors",
+    "model.layers.30.self_attn.kv_a_proj_with_mqa.scales": "model-00032.safetensors",
+    "model.layers.30.self_attn.kv_a_proj_with_mqa.biases": "model-00032.safetensors",
+    "model.layers.30.self_attn.o_proj.weight": "model-00032.safetensors",
+    "model.layers.30.self_attn.o_proj.scales": "model-00032.safetensors",
+    "model.layers.30.self_attn.o_proj.biases": "model-00032.safetensors",
+    "model.layers.30.self_attn.q_a_layernorm.weight": "model-00032.safetensors",
+    "model.layers.30.self_attn.q_a_proj.weight": "model-00032.safetensors",
+    "model.layers.30.self_attn.q_a_proj.scales": "model-00032.safetensors",
+    "model.layers.30.self_attn.q_a_proj.biases": "model-00032.safetensors",
+    "model.layers.30.self_attn.q_b_proj.weight": "model-00032.safetensors",
+    "model.layers.30.self_attn.q_b_proj.scales": "model-00032.safetensors",
+    "model.layers.30.self_attn.q_b_proj.biases": "model-00032.safetensors",
+    "model.layers.30.mlp.switch_mlp.gate_proj.weight": "model-00032.safetensors",
+    "model.layers.30.mlp.switch_mlp.gate_proj.scales": "model-00032.safetensors",
+    "model.layers.30.mlp.switch_mlp.gate_proj.biases": "model-00032.safetensors",
+    "model.layers.30.mlp.switch_mlp.down_proj.weight": "model-00032.safetensors",
+    "model.layers.30.mlp.switch_mlp.down_proj.scales": "model-00032.safetensors",
+    "model.layers.30.mlp.switch_mlp.down_proj.biases": "model-00032.safetensors",
+    "model.layers.30.mlp.switch_mlp.up_proj.weight": "model-00032.safetensors",
+    "model.layers.30.mlp.switch_mlp.up_proj.scales": "model-00032.safetensors",
+    "model.layers.30.mlp.switch_mlp.up_proj.biases": "model-00032.safetensors",
+    "model.layers.30.self_attn.embed_q.weight": "model-00032.safetensors",
+    "model.layers.30.self_attn.embed_q.scales": "model-00032.safetensors",
+    "model.layers.30.self_attn.embed_q.biases": "model-00032.safetensors",
+    "model.layers.30.self_attn.unembed_out.weight": "model-00032.safetensors",
+    "model.layers.30.self_attn.unembed_out.scales": "model-00032.safetensors",
+    "model.layers.30.self_attn.unembed_out.biases": "model-00032.safetensors",
+    "model.layers.31.input_layernorm.weight": "model-00033.safetensors",
+    "model.layers.31.mlp.gate.e_score_correction_bias": "model-00033.safetensors",
+    "model.layers.31.mlp.gate.weight": "model-00033.safetensors",
+    "model.layers.31.mlp.shared_experts.down_proj.weight": "model-00033.safetensors",
+    "model.layers.31.mlp.shared_experts.down_proj.scales": "model-00033.safetensors",
+    "model.layers.31.mlp.shared_experts.down_proj.biases": "model-00033.safetensors",
+    "model.layers.31.mlp.shared_experts.gate_proj.weight": "model-00033.safetensors",
+    "model.layers.31.mlp.shared_experts.gate_proj.scales": "model-00033.safetensors",
+    "model.layers.31.mlp.shared_experts.gate_proj.biases": "model-00033.safetensors",
+    "model.layers.31.mlp.shared_experts.up_proj.weight": "model-00033.safetensors",
+    "model.layers.31.mlp.shared_experts.up_proj.scales": "model-00033.safetensors",
+    "model.layers.31.mlp.shared_experts.up_proj.biases": "model-00033.safetensors",
+    "model.layers.31.post_attention_layernorm.weight": "model-00033.safetensors",
+    "model.layers.31.self_attn.kv_a_layernorm.weight": "model-00033.safetensors",
+    "model.layers.31.self_attn.kv_a_proj_with_mqa.weight": "model-00033.safetensors",
+    "model.layers.31.self_attn.kv_a_proj_with_mqa.scales": "model-00033.safetensors",
+    "model.layers.31.self_attn.kv_a_proj_with_mqa.biases": "model-00033.safetensors",
+    "model.layers.31.self_attn.o_proj.weight": "model-00033.safetensors",
+    "model.layers.31.self_attn.o_proj.scales": "model-00033.safetensors",
+    "model.layers.31.self_attn.o_proj.biases": "model-00033.safetensors",
+    "model.layers.31.self_attn.q_a_layernorm.weight": "model-00033.safetensors",
+    "model.layers.31.self_attn.q_a_proj.weight": "model-00033.safetensors",
+    "model.layers.31.self_attn.q_a_proj.scales": "model-00033.safetensors",
+    "model.layers.31.self_attn.q_a_proj.biases": "model-00033.safetensors",
+    "model.layers.31.self_attn.q_b_proj.weight": "model-00033.safetensors",
+    "model.layers.31.self_attn.q_b_proj.scales": "model-00033.safetensors",
+    "model.layers.31.self_attn.q_b_proj.biases": "model-00033.safetensors",
+    "model.layers.31.mlp.switch_mlp.gate_proj.weight": "model-00033.safetensors",
+    "model.layers.31.mlp.switch_mlp.gate_proj.scales": "model-00033.safetensors",
+    "model.layers.31.mlp.switch_mlp.gate_proj.biases": "model-00033.safetensors",
+    "model.layers.31.mlp.switch_mlp.down_proj.weight": "model-00033.safetensors",
+    "model.layers.31.mlp.switch_mlp.down_proj.scales": "model-00033.safetensors",
+    "model.layers.31.mlp.switch_mlp.down_proj.biases": "model-00033.safetensors",
+    "model.layers.31.mlp.switch_mlp.up_proj.weight": "model-00033.safetensors",
+    "model.layers.31.mlp.switch_mlp.up_proj.scales": "model-00033.safetensors",
+    "model.layers.31.mlp.switch_mlp.up_proj.biases": "model-00033.safetensors",
+    "model.layers.31.self_attn.embed_q.weight": "model-00033.safetensors",
+    "model.layers.31.self_attn.embed_q.scales": "model-00033.safetensors",
+    "model.layers.31.self_attn.embed_q.biases": "model-00033.safetensors",
+    "model.layers.31.self_attn.unembed_out.weight": "model-00033.safetensors",
+    "model.layers.31.self_attn.unembed_out.scales": "model-00033.safetensors",
+    "model.layers.31.self_attn.unembed_out.biases": "model-00033.safetensors",
+    "model.layers.32.input_layernorm.weight": "model-00034.safetensors",
+    "model.layers.32.mlp.gate.e_score_correction_bias": "model-00034.safetensors",
+    "model.layers.32.mlp.gate.weight": "model-00034.safetensors",
+    "model.layers.32.mlp.shared_experts.down_proj.weight": "model-00034.safetensors",
+    "model.layers.32.mlp.shared_experts.down_proj.scales": "model-00034.safetensors",
+    "model.layers.32.mlp.shared_experts.down_proj.biases": "model-00034.safetensors",
+    "model.layers.32.mlp.shared_experts.gate_proj.weight": "model-00034.safetensors",
+    "model.layers.32.mlp.shared_experts.gate_proj.scales": "model-00034.safetensors",
+    "model.layers.32.mlp.shared_experts.gate_proj.biases": "model-00034.safetensors",
+    "model.layers.32.mlp.shared_experts.up_proj.weight": "model-00034.safetensors",
+    "model.layers.32.mlp.shared_experts.up_proj.scales": "model-00034.safetensors",
+    "model.layers.32.mlp.shared_experts.up_proj.biases": "model-00034.safetensors",
+    "model.layers.32.post_attention_layernorm.weight": "model-00034.safetensors",
+    "model.layers.32.self_attn.kv_a_layernorm.weight": "model-00034.safetensors",
+    "model.layers.32.self_attn.kv_a_proj_with_mqa.weight": "model-00034.safetensors",
+    "model.layers.32.self_attn.kv_a_proj_with_mqa.scales": "model-00034.safetensors",
+    "model.layers.32.self_attn.kv_a_proj_with_mqa.biases": "model-00034.safetensors",
+    "model.layers.32.self_attn.o_proj.weight": "model-00034.safetensors",
+    "model.layers.32.self_attn.o_proj.scales": "model-00034.safetensors",
+    "model.layers.32.self_attn.o_proj.biases": "model-00034.safetensors",
+    "model.layers.32.self_attn.q_a_layernorm.weight": "model-00034.safetensors",
+    "model.layers.32.self_attn.q_a_proj.weight": "model-00034.safetensors",
+    "model.layers.32.self_attn.q_a_proj.scales": "model-00034.safetensors",
+    "model.layers.32.self_attn.q_a_proj.biases": "model-00034.safetensors",
+    "model.layers.32.self_attn.q_b_proj.weight": "model-00034.safetensors",
+    "model.layers.32.self_attn.q_b_proj.scales": "model-00034.safetensors",
+    "model.layers.32.self_attn.q_b_proj.biases": "model-00034.safetensors",
+    "model.layers.32.mlp.switch_mlp.gate_proj.weight": "model-00034.safetensors",
+    "model.layers.32.mlp.switch_mlp.gate_proj.scales": "model-00034.safetensors",
+    "model.layers.32.mlp.switch_mlp.gate_proj.biases": "model-00034.safetensors",
+    "model.layers.32.mlp.switch_mlp.down_proj.weight": "model-00034.safetensors",
+    "model.layers.32.mlp.switch_mlp.down_proj.scales": "model-00034.safetensors",
+    "model.layers.32.mlp.switch_mlp.down_proj.biases": "model-00034.safetensors",
+    "model.layers.32.mlp.switch_mlp.up_proj.weight": "model-00034.safetensors",
+    "model.layers.32.mlp.switch_mlp.up_proj.scales": "model-00034.safetensors",
+    "model.layers.32.mlp.switch_mlp.up_proj.biases": "model-00034.safetensors",
+    "model.layers.32.self_attn.embed_q.weight": "model-00034.safetensors",
+    "model.layers.32.self_attn.embed_q.scales": "model-00034.safetensors",
+    "model.layers.32.self_attn.embed_q.biases": "model-00034.safetensors",
+    "model.layers.32.self_attn.unembed_out.weight": "model-00034.safetensors",
+    "model.layers.32.self_attn.unembed_out.scales": "model-00034.safetensors",
+    "model.layers.32.self_attn.unembed_out.biases": "model-00034.safetensors",
+    "model.layers.33.input_layernorm.weight": "model-00035.safetensors",
+    "model.layers.33.mlp.gate.e_score_correction_bias": "model-00035.safetensors",
+    "model.layers.33.mlp.gate.weight": "model-00035.safetensors",
+    "model.layers.33.mlp.shared_experts.down_proj.weight": "model-00035.safetensors",
+    "model.layers.33.mlp.shared_experts.down_proj.scales": "model-00035.safetensors",
+    "model.layers.33.mlp.shared_experts.down_proj.biases": "model-00035.safetensors",
+    "model.layers.33.mlp.shared_experts.gate_proj.weight": "model-00035.safetensors",
+    "model.layers.33.mlp.shared_experts.gate_proj.scales": "model-00035.safetensors",
+    "model.layers.33.mlp.shared_experts.gate_proj.biases": "model-00035.safetensors",
+    "model.layers.33.mlp.shared_experts.up_proj.weight": "model-00035.safetensors",
+    "model.layers.33.mlp.shared_experts.up_proj.scales": "model-00035.safetensors",
+    "model.layers.33.mlp.shared_experts.up_proj.biases": "model-00035.safetensors",
+    "model.layers.33.post_attention_layernorm.weight": "model-00035.safetensors",
+    "model.layers.33.self_attn.kv_a_layernorm.weight": "model-00035.safetensors",
+    "model.layers.33.self_attn.kv_a_proj_with_mqa.weight": "model-00035.safetensors",
+    "model.layers.33.self_attn.kv_a_proj_with_mqa.scales": "model-00035.safetensors",
+    "model.layers.33.self_attn.kv_a_proj_with_mqa.biases": "model-00035.safetensors",
+    "model.layers.33.self_attn.o_proj.weight": "model-00035.safetensors",
+    "model.layers.33.self_attn.o_proj.scales": "model-00035.safetensors",
+    "model.layers.33.self_attn.o_proj.biases": "model-00035.safetensors",
+    "model.layers.33.self_attn.q_a_layernorm.weight": "model-00035.safetensors",
+    "model.layers.33.self_attn.q_a_proj.weight": "model-00035.safetensors",
+    "model.layers.33.self_attn.q_a_proj.scales": "model-00035.safetensors",
+    "model.layers.33.self_attn.q_a_proj.biases": "model-00035.safetensors",
+    "model.layers.33.self_attn.q_b_proj.weight": "model-00035.safetensors",
+    "model.layers.33.self_attn.q_b_proj.scales": "model-00035.safetensors",
+    "model.layers.33.self_attn.q_b_proj.biases": "model-00035.safetensors",
+    "model.layers.33.mlp.switch_mlp.gate_proj.weight": "model-00035.safetensors",
+    "model.layers.33.mlp.switch_mlp.gate_proj.scales": "model-00035.safetensors",
+    "model.layers.33.mlp.switch_mlp.gate_proj.biases": "model-00035.safetensors",
+    "model.layers.33.mlp.switch_mlp.down_proj.weight": "model-00035.safetensors",
+    "model.layers.33.mlp.switch_mlp.down_proj.scales": "model-00035.safetensors",
+    "model.layers.33.mlp.switch_mlp.down_proj.biases": "model-00035.safetensors",
+    "model.layers.33.mlp.switch_mlp.up_proj.weight": "model-00035.safetensors",
+    "model.layers.33.mlp.switch_mlp.up_proj.scales": "model-00035.safetensors",
+    "model.layers.33.mlp.switch_mlp.up_proj.biases": "model-00035.safetensors",
+    "model.layers.33.self_attn.embed_q.weight": "model-00035.safetensors",
+    "model.layers.33.self_attn.embed_q.scales": "model-00035.safetensors",
+    "model.layers.33.self_attn.embed_q.biases": "model-00035.safetensors",
+    "model.layers.33.self_attn.unembed_out.weight": "model-00035.safetensors",
+    "model.layers.33.self_attn.unembed_out.scales": "model-00035.safetensors",
+    "model.layers.33.self_attn.unembed_out.biases": "model-00035.safetensors",
+    "model.layers.34.input_layernorm.weight": "model-00036.safetensors",
+    "model.layers.34.mlp.gate.e_score_correction_bias": "model-00036.safetensors",
+    "model.layers.34.mlp.gate.weight": "model-00036.safetensors",
+    "model.layers.34.mlp.shared_experts.down_proj.weight": "model-00036.safetensors",
+    "model.layers.34.mlp.shared_experts.down_proj.scales": "model-00036.safetensors",
+    "model.layers.34.mlp.shared_experts.down_proj.biases": "model-00036.safetensors",
+    "model.layers.34.mlp.shared_experts.gate_proj.weight": "model-00036.safetensors",
+    "model.layers.34.mlp.shared_experts.gate_proj.scales": "model-00036.safetensors",
+    "model.layers.34.mlp.shared_experts.gate_proj.biases": "model-00036.safetensors",
+    "model.layers.34.mlp.shared_experts.up_proj.weight": "model-00036.safetensors",
+    "model.layers.34.mlp.shared_experts.up_proj.scales": "model-00036.safetensors",
+    "model.layers.34.mlp.shared_experts.up_proj.biases": "model-00036.safetensors",
+    "model.layers.34.post_attention_layernorm.weight": "model-00036.safetensors",
+    "model.layers.34.self_attn.indexer.k_norm.bias": "model-00036.safetensors",
+    "model.layers.34.self_attn.indexer.k_norm.weight": "model-00036.safetensors",
+    "model.layers.34.self_attn.indexer.weights_proj.weight": "model-00036.safetensors",
+    "model.layers.34.self_attn.indexer.wk.weight": "model-00036.safetensors",
+    "model.layers.34.self_attn.indexer.wq_b.weight": "model-00036.safetensors",
+    "model.layers.34.self_attn.kv_a_layernorm.weight": "model-00036.safetensors",
+    "model.layers.34.self_attn.kv_a_proj_with_mqa.weight": "model-00036.safetensors",
+    "model.layers.34.self_attn.kv_a_proj_with_mqa.scales": "model-00036.safetensors",
+    "model.layers.34.self_attn.kv_a_proj_with_mqa.biases": "model-00036.safetensors",
+    "model.layers.34.self_attn.o_proj.weight": "model-00036.safetensors",
+    "model.layers.34.self_attn.o_proj.scales": "model-00036.safetensors",
+    "model.layers.34.self_attn.o_proj.biases": "model-00036.safetensors",
+    "model.layers.34.self_attn.q_a_layernorm.weight": "model-00036.safetensors",
+    "model.layers.34.self_attn.q_a_proj.weight": "model-00036.safetensors",
+    "model.layers.34.self_attn.q_a_proj.scales": "model-00036.safetensors",
+    "model.layers.34.self_attn.q_a_proj.biases": "model-00036.safetensors",
+    "model.layers.34.self_attn.q_b_proj.weight": "model-00036.safetensors",
+    "model.layers.34.self_attn.q_b_proj.scales": "model-00036.safetensors",
+    "model.layers.34.self_attn.q_b_proj.biases": "model-00036.safetensors",
+    "model.layers.34.mlp.switch_mlp.gate_proj.weight": "model-00036.safetensors",
+    "model.layers.34.mlp.switch_mlp.gate_proj.scales": "model-00036.safetensors",
+    "model.layers.34.mlp.switch_mlp.gate_proj.biases": "model-00036.safetensors",
+    "model.layers.34.mlp.switch_mlp.down_proj.weight": "model-00036.safetensors",
+    "model.layers.34.mlp.switch_mlp.down_proj.scales": "model-00036.safetensors",
+    "model.layers.34.mlp.switch_mlp.down_proj.biases": "model-00036.safetensors",
+    "model.layers.34.mlp.switch_mlp.up_proj.weight": "model-00036.safetensors",
+    "model.layers.34.mlp.switch_mlp.up_proj.scales": "model-00036.safetensors",
+    "model.layers.34.mlp.switch_mlp.up_proj.biases": "model-00036.safetensors",
+    "model.layers.34.self_attn.embed_q.weight": "model-00036.safetensors",
+    "model.layers.34.self_attn.embed_q.scales": "model-00036.safetensors",
+    "model.layers.34.self_attn.embed_q.biases": "model-00036.safetensors",
+    "model.layers.34.self_attn.unembed_out.weight": "model-00036.safetensors",
+    "model.layers.34.self_attn.unembed_out.scales": "model-00036.safetensors",
+    "model.layers.34.self_attn.unembed_out.biases": "model-00036.safetensors",
+    "model.layers.35.input_layernorm.weight": "model-00037.safetensors",
+    "model.layers.35.mlp.gate.e_score_correction_bias": "model-00037.safetensors",
+    "model.layers.35.mlp.gate.weight": "model-00037.safetensors",
+    "model.layers.35.mlp.shared_experts.down_proj.weight": "model-00037.safetensors",
+    "model.layers.35.mlp.shared_experts.down_proj.scales": "model-00037.safetensors",
+    "model.layers.35.mlp.shared_experts.down_proj.biases": "model-00037.safetensors",
+    "model.layers.35.mlp.shared_experts.gate_proj.weight": "model-00037.safetensors",
+    "model.layers.35.mlp.shared_experts.gate_proj.scales": "model-00037.safetensors",
+    "model.layers.35.mlp.shared_experts.gate_proj.biases": "model-00037.safetensors",
+    "model.layers.35.mlp.shared_experts.up_proj.weight": "model-00037.safetensors",
+    "model.layers.35.mlp.shared_experts.up_proj.scales": "model-00037.safetensors",
+    "model.layers.35.mlp.shared_experts.up_proj.biases": "model-00037.safetensors",
+    "model.layers.35.post_attention_layernorm.weight": "model-00037.safetensors",
+    "model.layers.35.self_attn.kv_a_layernorm.weight": "model-00037.safetensors",
+    "model.layers.35.self_attn.kv_a_proj_with_mqa.weight": "model-00037.safetensors",
+    "model.layers.35.self_attn.kv_a_proj_with_mqa.scales": "model-00037.safetensors",
+    "model.layers.35.self_attn.kv_a_proj_with_mqa.biases": "model-00037.safetensors",
+    "model.layers.35.self_attn.o_proj.weight": "model-00037.safetensors",
+    "model.layers.35.self_attn.o_proj.scales": "model-00037.safetensors",
+    "model.layers.35.self_attn.o_proj.biases": "model-00037.safetensors",
+    "model.layers.35.self_attn.q_a_layernorm.weight": "model-00037.safetensors",
+    "model.layers.35.self_attn.q_a_proj.weight": "model-00037.safetensors",
+    "model.layers.35.self_attn.q_a_proj.scales": "model-00037.safetensors",
+    "model.layers.35.self_attn.q_a_proj.biases": "model-00037.safetensors",
+    "model.layers.35.self_attn.q_b_proj.weight": "model-00037.safetensors",
+    "model.layers.35.self_attn.q_b_proj.scales": "model-00037.safetensors",
+    "model.layers.35.self_attn.q_b_proj.biases": "model-00037.safetensors",
+    "model.layers.35.mlp.switch_mlp.gate_proj.weight": "model-00037.safetensors",
+    "model.layers.35.mlp.switch_mlp.gate_proj.scales": "model-00037.safetensors",
+    "model.layers.35.mlp.switch_mlp.gate_proj.biases": "model-00037.safetensors",
+    "model.layers.35.mlp.switch_mlp.down_proj.weight": "model-00037.safetensors",
+    "model.layers.35.mlp.switch_mlp.down_proj.scales": "model-00037.safetensors",
+    "model.layers.35.mlp.switch_mlp.down_proj.biases": "model-00037.safetensors",
+    "model.layers.35.mlp.switch_mlp.up_proj.weight": "model-00037.safetensors",
+    "model.layers.35.mlp.switch_mlp.up_proj.scales": "model-00037.safetensors",
+    "model.layers.35.mlp.switch_mlp.up_proj.biases": "model-00037.safetensors",
+    "model.layers.35.self_attn.embed_q.weight": "model-00037.safetensors",
+    "model.layers.35.self_attn.embed_q.scales": "model-00037.safetensors",
+    "model.layers.35.self_attn.embed_q.biases": "model-00037.safetensors",
+    "model.layers.35.self_attn.unembed_out.weight": "model-00037.safetensors",
+    "model.layers.35.self_attn.unembed_out.scales": "model-00037.safetensors",
+    "model.layers.35.self_attn.unembed_out.biases": "model-00037.safetensors",
+    "model.layers.36.input_layernorm.weight": "model-00038.safetensors",
+    "model.layers.36.mlp.gate.e_score_correction_bias": "model-00038.safetensors",
+    "model.layers.36.mlp.gate.weight": "model-00038.safetensors",
+    "model.layers.36.mlp.shared_experts.down_proj.weight": "model-00038.safetensors",
+    "model.layers.36.mlp.shared_experts.down_proj.scales": "model-00038.safetensors",
+    "model.layers.36.mlp.shared_experts.down_proj.biases": "model-00038.safetensors",
+    "model.layers.36.mlp.shared_experts.gate_proj.weight": "model-00038.safetensors",
+    "model.layers.36.mlp.shared_experts.gate_proj.scales": "model-00038.safetensors",
+    "model.layers.36.mlp.shared_experts.gate_proj.biases": "model-00038.safetensors",
+    "model.layers.36.mlp.shared_experts.up_proj.weight": "model-00038.safetensors",
+    "model.layers.36.mlp.shared_experts.up_proj.scales": "model-00038.safetensors",
+    "model.layers.36.mlp.shared_experts.up_proj.biases": "model-00038.safetensors",
+    "model.layers.36.post_attention_layernorm.weight": "model-00038.safetensors",
+    "model.layers.36.self_attn.kv_a_layernorm.weight": "model-00038.safetensors",
+    "model.layers.36.self_attn.kv_a_proj_with_mqa.weight": "model-00038.safetensors",
+    "model.layers.36.self_attn.kv_a_proj_with_mqa.scales": "model-00038.safetensors",
+    "model.layers.36.self_attn.kv_a_proj_with_mqa.biases": "model-00038.safetensors",
+    "model.layers.36.self_attn.o_proj.weight": "model-00038.safetensors",
+    "model.layers.36.self_attn.o_proj.scales": "model-00038.safetensors",
+    "model.layers.36.self_attn.o_proj.biases": "model-00038.safetensors",
+    "model.layers.36.self_attn.q_a_layernorm.weight": "model-00038.safetensors",
+    "model.layers.36.self_attn.q_a_proj.weight": "model-00038.safetensors",
+    "model.layers.36.self_attn.q_a_proj.scales": "model-00038.safetensors",
+    "model.layers.36.self_attn.q_a_proj.biases": "model-00038.safetensors",
+    "model.layers.36.self_attn.q_b_proj.weight": "model-00038.safetensors",
+    "model.layers.36.self_attn.q_b_proj.scales": "model-00038.safetensors",
+    "model.layers.36.self_attn.q_b_proj.biases": "model-00038.safetensors",
+    "model.layers.36.mlp.switch_mlp.gate_proj.weight": "model-00038.safetensors",
+    "model.layers.36.mlp.switch_mlp.gate_proj.scales": "model-00038.safetensors",
+    "model.layers.36.mlp.switch_mlp.gate_proj.biases": "model-00038.safetensors",
+    "model.layers.36.mlp.switch_mlp.down_proj.weight": "model-00038.safetensors",
+    "model.layers.36.mlp.switch_mlp.down_proj.scales": "model-00038.safetensors",
+    "model.layers.36.mlp.switch_mlp.down_proj.biases": "model-00038.safetensors",
+    "model.layers.36.mlp.switch_mlp.up_proj.weight": "model-00038.safetensors",
+    "model.layers.36.mlp.switch_mlp.up_proj.scales": "model-00038.safetensors",
+    "model.layers.36.mlp.switch_mlp.up_proj.biases": "model-00038.safetensors",
+    "model.layers.36.self_attn.embed_q.weight": "model-00038.safetensors",
+    "model.layers.36.self_attn.embed_q.scales": "model-00038.safetensors",
+    "model.layers.36.self_attn.embed_q.biases": "model-00038.safetensors",
+    "model.layers.36.self_attn.unembed_out.weight": "model-00038.safetensors",
+    "model.layers.36.self_attn.unembed_out.scales": "model-00038.safetensors",
+    "model.layers.36.self_attn.unembed_out.biases": "model-00038.safetensors",
+    "model.layers.37.input_layernorm.weight": "model-00039.safetensors",
+    "model.layers.37.mlp.gate.e_score_correction_bias": "model-00039.safetensors",
+    "model.layers.37.mlp.gate.weight": "model-00039.safetensors",
+    "model.layers.37.mlp.shared_experts.down_proj.weight": "model-00039.safetensors",
+    "model.layers.37.mlp.shared_experts.down_proj.scales": "model-00039.safetensors",
+    "model.layers.37.mlp.shared_experts.down_proj.biases": "model-00039.safetensors",
+    "model.layers.37.mlp.shared_experts.gate_proj.weight": "model-00039.safetensors",
+    "model.layers.37.mlp.shared_experts.gate_proj.scales": "model-00039.safetensors",
+    "model.layers.37.mlp.shared_experts.gate_proj.biases": "model-00039.safetensors",
+    "model.layers.37.mlp.shared_experts.up_proj.weight": "model-00039.safetensors",
+    "model.layers.37.mlp.shared_experts.up_proj.scales": "model-00039.safetensors",
+    "model.layers.37.mlp.shared_experts.up_proj.biases": "model-00039.safetensors",
+    "model.layers.37.post_attention_layernorm.weight": "model-00039.safetensors",
+    "model.layers.37.self_attn.kv_a_layernorm.weight": "model-00039.safetensors",
+    "model.layers.37.self_attn.kv_a_proj_with_mqa.weight": "model-00039.safetensors",
+    "model.layers.37.self_attn.kv_a_proj_with_mqa.scales": "model-00039.safetensors",
+    "model.layers.37.self_attn.kv_a_proj_with_mqa.biases": "model-00039.safetensors",
+    "model.layers.37.self_attn.o_proj.weight": "model-00039.safetensors",
+    "model.layers.37.self_attn.o_proj.scales": "model-00039.safetensors",
+    "model.layers.37.self_attn.o_proj.biases": "model-00039.safetensors",
+    "model.layers.37.self_attn.q_a_layernorm.weight": "model-00039.safetensors",
+    "model.layers.37.self_attn.q_a_proj.weight": "model-00039.safetensors",
+    "model.layers.37.self_attn.q_a_proj.scales": "model-00039.safetensors",
+    "model.layers.37.self_attn.q_a_proj.biases": "model-00039.safetensors",
+    "model.layers.37.self_attn.q_b_proj.weight": "model-00039.safetensors",
+    "model.layers.37.self_attn.q_b_proj.scales": "model-00039.safetensors",
+    "model.layers.37.self_attn.q_b_proj.biases": "model-00039.safetensors",
+    "model.layers.37.mlp.switch_mlp.gate_proj.weight": "model-00039.safetensors",
+    "model.layers.37.mlp.switch_mlp.gate_proj.scales": "model-00039.safetensors",
+    "model.layers.37.mlp.switch_mlp.gate_proj.biases": "model-00039.safetensors",
+    "model.layers.37.mlp.switch_mlp.down_proj.weight": "model-00039.safetensors",
+    "model.layers.37.mlp.switch_mlp.down_proj.scales": "model-00039.safetensors",
+    "model.layers.37.mlp.switch_mlp.down_proj.biases": "model-00039.safetensors",
+    "model.layers.37.mlp.switch_mlp.up_proj.weight": "model-00039.safetensors",
+    "model.layers.37.mlp.switch_mlp.up_proj.scales": "model-00039.safetensors",
+    "model.layers.37.mlp.switch_mlp.up_proj.biases": "model-00039.safetensors",
+    "model.layers.37.self_attn.embed_q.weight": "model-00039.safetensors",
+    "model.layers.37.self_attn.embed_q.scales": "model-00039.safetensors",
+    "model.layers.37.self_attn.embed_q.biases": "model-00039.safetensors",
+    "model.layers.37.self_attn.unembed_out.weight": "model-00039.safetensors",
+    "model.layers.37.self_attn.unembed_out.scales": "model-00039.safetensors",
+    "model.layers.37.self_attn.unembed_out.biases": "model-00039.safetensors",
+    "model.layers.38.input_layernorm.weight": "model-00040.safetensors",
+    "model.layers.38.mlp.gate.e_score_correction_bias": "model-00040.safetensors",
+    "model.layers.38.mlp.gate.weight": "model-00040.safetensors",
+    "model.layers.38.mlp.shared_experts.down_proj.weight": "model-00040.safetensors",
+    "model.layers.38.mlp.shared_experts.down_proj.scales": "model-00040.safetensors",
+    "model.layers.38.mlp.shared_experts.down_proj.biases": "model-00040.safetensors",
+    "model.layers.38.mlp.shared_experts.gate_proj.weight": "model-00040.safetensors",
+    "model.layers.38.mlp.shared_experts.gate_proj.scales": "model-00040.safetensors",
+    "model.layers.38.mlp.shared_experts.gate_proj.biases": "model-00040.safetensors",
+    "model.layers.38.mlp.shared_experts.up_proj.weight": "model-00040.safetensors",
+    "model.layers.38.mlp.shared_experts.up_proj.scales": "model-00040.safetensors",
+    "model.layers.38.mlp.shared_experts.up_proj.biases": "model-00040.safetensors",
+    "model.layers.38.post_attention_layernorm.weight": "model-00040.safetensors",
+    "model.layers.38.self_attn.indexer.k_norm.bias": "model-00040.safetensors",
+    "model.layers.38.self_attn.indexer.k_norm.weight": "model-00040.safetensors",
+    "model.layers.38.self_attn.indexer.weights_proj.weight": "model-00040.safetensors",
+    "model.layers.38.self_attn.indexer.wk.weight": "model-00040.safetensors",
+    "model.layers.38.self_attn.indexer.wq_b.weight": "model-00040.safetensors",
+    "model.layers.38.self_attn.kv_a_layernorm.weight": "model-00040.safetensors",
+    "model.layers.38.self_attn.kv_a_proj_with_mqa.weight": "model-00040.safetensors",
+    "model.layers.38.self_attn.kv_a_proj_with_mqa.scales": "model-00040.safetensors",
+    "model.layers.38.self_attn.kv_a_proj_with_mqa.biases": "model-00040.safetensors",
+    "model.layers.38.self_attn.o_proj.weight": "model-00040.safetensors",
+    "model.layers.38.self_attn.o_proj.scales": "model-00040.safetensors",
+    "model.layers.38.self_attn.o_proj.biases": "model-00040.safetensors",
+    "model.layers.38.self_attn.q_a_layernorm.weight": "model-00040.safetensors",
+    "model.layers.38.self_attn.q_a_proj.weight": "model-00040.safetensors",
+    "model.layers.38.self_attn.q_a_proj.scales": "model-00040.safetensors",
+    "model.layers.38.self_attn.q_a_proj.biases": "model-00040.safetensors",
+    "model.layers.38.self_attn.q_b_proj.weight": "model-00040.safetensors",
+    "model.layers.38.self_attn.q_b_proj.scales": "model-00040.safetensors",
+    "model.layers.38.self_attn.q_b_proj.biases": "model-00040.safetensors",
+    "model.layers.38.mlp.switch_mlp.gate_proj.weight": "model-00040.safetensors",
+    "model.layers.38.mlp.switch_mlp.gate_proj.scales": "model-00040.safetensors",
+    "model.layers.38.mlp.switch_mlp.gate_proj.biases": "model-00040.safetensors",
+    "model.layers.38.mlp.switch_mlp.down_proj.weight": "model-00040.safetensors",
+    "model.layers.38.mlp.switch_mlp.down_proj.scales": "model-00040.safetensors",
+    "model.layers.38.mlp.switch_mlp.down_proj.biases": "model-00040.safetensors",
+    "model.layers.38.mlp.switch_mlp.up_proj.weight": "model-00040.safetensors",
+    "model.layers.38.mlp.switch_mlp.up_proj.scales": "model-00040.safetensors",
+    "model.layers.38.mlp.switch_mlp.up_proj.biases": "model-00040.safetensors",
+    "model.layers.38.self_attn.embed_q.weight": "model-00040.safetensors",
+    "model.layers.38.self_attn.embed_q.scales": "model-00040.safetensors",
+    "model.layers.38.self_attn.embed_q.biases": "model-00040.safetensors",
+    "model.layers.38.self_attn.unembed_out.weight": "model-00040.safetensors",
+    "model.layers.38.self_attn.unembed_out.scales": "model-00040.safetensors",
+    "model.layers.38.self_attn.unembed_out.biases": "model-00040.safetensors",
+    "model.layers.39.input_layernorm.weight": "model-00041.safetensors",
+    "model.layers.39.mlp.gate.e_score_correction_bias": "model-00041.safetensors",
+    "model.layers.39.mlp.gate.weight": "model-00041.safetensors",
+    "model.layers.39.mlp.shared_experts.down_proj.weight": "model-00041.safetensors",
+    "model.layers.39.mlp.shared_experts.down_proj.scales": "model-00041.safetensors",
+    "model.layers.39.mlp.shared_experts.down_proj.biases": "model-00041.safetensors",
+    "model.layers.39.mlp.shared_experts.gate_proj.weight": "model-00041.safetensors",
+    "model.layers.39.mlp.shared_experts.gate_proj.scales": "model-00041.safetensors",
+    "model.layers.39.mlp.shared_experts.gate_proj.biases": "model-00041.safetensors",
+    "model.layers.39.mlp.shared_experts.up_proj.weight": "model-00041.safetensors",
+    "model.layers.39.mlp.shared_experts.up_proj.scales": "model-00041.safetensors",
+    "model.layers.39.mlp.shared_experts.up_proj.biases": "model-00041.safetensors",
+    "model.layers.39.post_attention_layernorm.weight": "model-00041.safetensors",
+    "model.layers.39.self_attn.kv_a_layernorm.weight": "model-00041.safetensors",
+    "model.layers.39.self_attn.kv_a_proj_with_mqa.weight": "model-00041.safetensors",
+    "model.layers.39.self_attn.kv_a_proj_with_mqa.scales": "model-00041.safetensors",
+    "model.layers.39.self_attn.kv_a_proj_with_mqa.biases": "model-00041.safetensors",
+    "model.layers.39.self_attn.o_proj.weight": "model-00041.safetensors",
+    "model.layers.39.self_attn.o_proj.scales": "model-00041.safetensors",
+    "model.layers.39.self_attn.o_proj.biases": "model-00041.safetensors",
+    "model.layers.39.self_attn.q_a_layernorm.weight": "model-00041.safetensors",
+    "model.layers.39.self_attn.q_a_proj.weight": "model-00041.safetensors",
+    "model.layers.39.self_attn.q_a_proj.scales": "model-00041.safetensors",
+    "model.layers.39.self_attn.q_a_proj.biases": "model-00041.safetensors",
+    "model.layers.39.self_attn.q_b_proj.weight": "model-00041.safetensors",
+    "model.layers.39.self_attn.q_b_proj.scales": "model-00041.safetensors",
+    "model.layers.39.self_attn.q_b_proj.biases": "model-00041.safetensors",
+    "model.layers.39.mlp.switch_mlp.gate_proj.weight": "model-00041.safetensors",
+    "model.layers.39.mlp.switch_mlp.gate_proj.scales": "model-00041.safetensors",
+    "model.layers.39.mlp.switch_mlp.gate_proj.biases": "model-00041.safetensors",
+    "model.layers.39.mlp.switch_mlp.down_proj.weight": "model-00041.safetensors",
+    "model.layers.39.mlp.switch_mlp.down_proj.scales": "model-00041.safetensors",
+    "model.layers.39.mlp.switch_mlp.down_proj.biases": "model-00041.safetensors",
+    "model.layers.39.mlp.switch_mlp.up_proj.weight": "model-00041.safetensors",
+    "model.layers.39.mlp.switch_mlp.up_proj.scales": "model-00041.safetensors",
+    "model.layers.39.mlp.switch_mlp.up_proj.biases": "model-00041.safetensors",
+    "model.layers.39.self_attn.embed_q.weight": "model-00041.safetensors",
+    "model.layers.39.self_attn.embed_q.scales": "model-00041.safetensors",
+    "model.layers.39.self_attn.embed_q.biases": "model-00041.safetensors",
+    "model.layers.39.self_attn.unembed_out.weight": "model-00041.safetensors",
+    "model.layers.39.self_attn.unembed_out.scales": "model-00041.safetensors",
+    "model.layers.39.self_attn.unembed_out.biases": "model-00041.safetensors",
+    "model.layers.40.input_layernorm.weight": "model-00042.safetensors",
+    "model.layers.40.mlp.gate.e_score_correction_bias": "model-00042.safetensors",
+    "model.layers.40.mlp.gate.weight": "model-00042.safetensors",
+    "model.layers.40.mlp.shared_experts.down_proj.weight": "model-00042.safetensors",
+    "model.layers.40.mlp.shared_experts.down_proj.scales": "model-00042.safetensors",
+    "model.layers.40.mlp.shared_experts.down_proj.biases": "model-00042.safetensors",
+    "model.layers.40.mlp.shared_experts.gate_proj.weight": "model-00042.safetensors",
+    "model.layers.40.mlp.shared_experts.gate_proj.scales": "model-00042.safetensors",
+    "model.layers.40.mlp.shared_experts.gate_proj.biases": "model-00042.safetensors",
+    "model.layers.40.mlp.shared_experts.up_proj.weight": "model-00042.safetensors",
+    "model.layers.40.mlp.shared_experts.up_proj.scales": "model-00042.safetensors",
+    "model.layers.40.mlp.shared_experts.up_proj.biases": "model-00042.safetensors",
+    "model.layers.40.post_attention_layernorm.weight": "model-00042.safetensors",
+    "model.layers.40.self_attn.kv_a_layernorm.weight": "model-00042.safetensors",
+    "model.layers.40.self_attn.kv_a_proj_with_mqa.weight": "model-00042.safetensors",
+    "model.layers.40.self_attn.kv_a_proj_with_mqa.scales": "model-00042.safetensors",
+    "model.layers.40.self_attn.kv_a_proj_with_mqa.biases": "model-00042.safetensors",
+    "model.layers.40.self_attn.o_proj.weight": "model-00042.safetensors",
+    "model.layers.40.self_attn.o_proj.scales": "model-00042.safetensors",
+    "model.layers.40.self_attn.o_proj.biases": "model-00042.safetensors",
+    "model.layers.40.self_attn.q_a_layernorm.weight": "model-00042.safetensors",
+    "model.layers.40.self_attn.q_a_proj.weight": "model-00042.safetensors",
+    "model.layers.40.self_attn.q_a_proj.scales": "model-00042.safetensors",
+    "model.layers.40.self_attn.q_a_proj.biases": "model-00042.safetensors",
+    "model.layers.40.self_attn.q_b_proj.weight": "model-00042.safetensors",
+    "model.layers.40.self_attn.q_b_proj.scales": "model-00042.safetensors",
+    "model.layers.40.self_attn.q_b_proj.biases": "model-00042.safetensors",
+    "model.layers.40.mlp.switch_mlp.gate_proj.weight": "model-00042.safetensors",
+    "model.layers.40.mlp.switch_mlp.gate_proj.scales": "model-00042.safetensors",
+    "model.layers.40.mlp.switch_mlp.gate_proj.biases": "model-00042.safetensors",
+    "model.layers.40.mlp.switch_mlp.down_proj.weight": "model-00042.safetensors",
+    "model.layers.40.mlp.switch_mlp.down_proj.scales": "model-00042.safetensors",
+    "model.layers.40.mlp.switch_mlp.down_proj.biases": "model-00042.safetensors",
+    "model.layers.40.mlp.switch_mlp.up_proj.weight": "model-00042.safetensors",
+    "model.layers.40.mlp.switch_mlp.up_proj.scales": "model-00042.safetensors",
+    "model.layers.40.mlp.switch_mlp.up_proj.biases": "model-00042.safetensors",
+    "model.layers.40.self_attn.embed_q.weight": "model-00042.safetensors",
+    "model.layers.40.self_attn.embed_q.scales": "model-00042.safetensors",
+    "model.layers.40.self_attn.embed_q.biases": "model-00042.safetensors",
+    "model.layers.40.self_attn.unembed_out.weight": "model-00042.safetensors",
+    "model.layers.40.self_attn.unembed_out.scales": "model-00042.safetensors",
+    "model.layers.40.self_attn.unembed_out.biases": "model-00042.safetensors",
+    "model.layers.41.input_layernorm.weight": "model-00043.safetensors",
+    "model.layers.41.mlp.gate.e_score_correction_bias": "model-00043.safetensors",
+    "model.layers.41.mlp.gate.weight": "model-00043.safetensors",
+    "model.layers.41.mlp.shared_experts.down_proj.weight": "model-00043.safetensors",
+    "model.layers.41.mlp.shared_experts.down_proj.scales": "model-00043.safetensors",
+    "model.layers.41.mlp.shared_experts.down_proj.biases": "model-00043.safetensors",
+    "model.layers.41.mlp.shared_experts.gate_proj.weight": "model-00043.safetensors",
+    "model.layers.41.mlp.shared_experts.gate_proj.scales": "model-00043.safetensors",
+    "model.layers.41.mlp.shared_experts.gate_proj.biases": "model-00043.safetensors",
+    "model.layers.41.mlp.shared_experts.up_proj.weight": "model-00043.safetensors",
+    "model.layers.41.mlp.shared_experts.up_proj.scales": "model-00043.safetensors",
+    "model.layers.41.mlp.shared_experts.up_proj.biases": "model-00043.safetensors",
+    "model.layers.41.post_attention_layernorm.weight": "model-00043.safetensors",
+    "model.layers.41.self_attn.kv_a_layernorm.weight": "model-00043.safetensors",
+    "model.layers.41.self_attn.kv_a_proj_with_mqa.weight": "model-00043.safetensors",
+    "model.layers.41.self_attn.kv_a_proj_with_mqa.scales": "model-00043.safetensors",
+    "model.layers.41.self_attn.kv_a_proj_with_mqa.biases": "model-00043.safetensors",
+    "model.layers.41.self_attn.o_proj.weight": "model-00043.safetensors",
+    "model.layers.41.self_attn.o_proj.scales": "model-00043.safetensors",
+    "model.layers.41.self_attn.o_proj.biases": "model-00043.safetensors",
+    "model.layers.41.self_attn.q_a_layernorm.weight": "model-00043.safetensors",
+    "model.layers.41.self_attn.q_a_proj.weight": "model-00043.safetensors",
+    "model.layers.41.self_attn.q_a_proj.scales": "model-00043.safetensors",
+    "model.layers.41.self_attn.q_a_proj.biases": "model-00043.safetensors",
+    "model.layers.41.self_attn.q_b_proj.weight": "model-00043.safetensors",
+    "model.layers.41.self_attn.q_b_proj.scales": "model-00043.safetensors",
+    "model.layers.41.self_attn.q_b_proj.biases": "model-00043.safetensors",
+    "model.layers.41.mlp.switch_mlp.gate_proj.weight": "model-00043.safetensors",
+    "model.layers.41.mlp.switch_mlp.gate_proj.scales": "model-00043.safetensors",
+    "model.layers.41.mlp.switch_mlp.gate_proj.biases": "model-00043.safetensors",
+    "model.layers.41.mlp.switch_mlp.down_proj.weight": "model-00043.safetensors",
+    "model.layers.41.mlp.switch_mlp.down_proj.scales": "model-00043.safetensors",
+    "model.layers.41.mlp.switch_mlp.down_proj.biases": "model-00043.safetensors",
+    "model.layers.41.mlp.switch_mlp.up_proj.weight": "model-00043.safetensors",
+    "model.layers.41.mlp.switch_mlp.up_proj.scales": "model-00043.safetensors",
+    "model.layers.41.mlp.switch_mlp.up_proj.biases": "model-00043.safetensors",
+    "model.layers.41.self_attn.embed_q.weight": "model-00043.safetensors",
+    "model.layers.41.self_attn.embed_q.scales": "model-00043.safetensors",
+    "model.layers.41.self_attn.embed_q.biases": "model-00043.safetensors",
+    "model.layers.41.self_attn.unembed_out.weight": "model-00043.safetensors",
+    "model.layers.41.self_attn.unembed_out.scales": "model-00043.safetensors",
+    "model.layers.41.self_attn.unembed_out.biases": "model-00043.safetensors",
+    "model.layers.42.input_layernorm.weight": "model-00044.safetensors",
+    "model.layers.42.mlp.gate.e_score_correction_bias": "model-00044.safetensors",
+    "model.layers.42.mlp.gate.weight": "model-00044.safetensors",
+    "model.layers.42.mlp.shared_experts.down_proj.weight": "model-00044.safetensors",
+    "model.layers.42.mlp.shared_experts.down_proj.scales": "model-00044.safetensors",
+    "model.layers.42.mlp.shared_experts.down_proj.biases": "model-00044.safetensors",
+    "model.layers.42.mlp.shared_experts.gate_proj.weight": "model-00044.safetensors",
+    "model.layers.42.mlp.shared_experts.gate_proj.scales": "model-00044.safetensors",
+    "model.layers.42.mlp.shared_experts.gate_proj.biases": "model-00044.safetensors",
+    "model.layers.42.mlp.shared_experts.up_proj.weight": "model-00044.safetensors",
+    "model.layers.42.mlp.shared_experts.up_proj.scales": "model-00044.safetensors",
+    "model.layers.42.mlp.shared_experts.up_proj.biases": "model-00044.safetensors",
+    "model.layers.42.post_attention_layernorm.weight": "model-00044.safetensors",
+    "model.layers.42.self_attn.indexer.k_norm.bias": "model-00044.safetensors",
+    "model.layers.42.self_attn.indexer.k_norm.weight": "model-00044.safetensors",
+    "model.layers.42.self_attn.indexer.weights_proj.weight": "model-00044.safetensors",
+    "model.layers.42.self_attn.indexer.wk.weight": "model-00044.safetensors",
+    "model.layers.42.self_attn.indexer.wq_b.weight": "model-00044.safetensors",
+    "model.layers.42.self_attn.kv_a_layernorm.weight": "model-00044.safetensors",
+    "model.layers.42.self_attn.kv_a_proj_with_mqa.weight": "model-00044.safetensors",
+    "model.layers.42.self_attn.kv_a_proj_with_mqa.scales": "model-00044.safetensors",
+    "model.layers.42.self_attn.kv_a_proj_with_mqa.biases": "model-00044.safetensors",
+    "model.layers.42.self_attn.o_proj.weight": "model-00044.safetensors",
+    "model.layers.42.self_attn.o_proj.scales": "model-00044.safetensors",
+    "model.layers.42.self_attn.o_proj.biases": "model-00044.safetensors",
+    "model.layers.42.self_attn.q_a_layernorm.weight": "model-00044.safetensors",
+    "model.layers.42.self_attn.q_a_proj.weight": "model-00044.safetensors",
+    "model.layers.42.self_attn.q_a_proj.scales": "model-00044.safetensors",
+    "model.layers.42.self_attn.q_a_proj.biases": "model-00044.safetensors",
+    "model.layers.42.self_attn.q_b_proj.weight": "model-00044.safetensors",
+    "model.layers.42.self_attn.q_b_proj.scales": "model-00044.safetensors",
+    "model.layers.42.self_attn.q_b_proj.biases": "model-00044.safetensors",
+    "model.layers.42.mlp.switch_mlp.gate_proj.weight": "model-00044.safetensors",
+    "model.layers.42.mlp.switch_mlp.gate_proj.scales": "model-00044.safetensors",
+    "model.layers.42.mlp.switch_mlp.gate_proj.biases": "model-00044.safetensors",
+    "model.layers.42.mlp.switch_mlp.down_proj.weight": "model-00044.safetensors",
+    "model.layers.42.mlp.switch_mlp.down_proj.scales": "model-00044.safetensors",
+    "model.layers.42.mlp.switch_mlp.down_proj.biases": "model-00044.safetensors",
+    "model.layers.42.mlp.switch_mlp.up_proj.weight": "model-00044.safetensors",
+    "model.layers.42.mlp.switch_mlp.up_proj.scales": "model-00044.safetensors",
+    "model.layers.42.mlp.switch_mlp.up_proj.biases": "model-00044.safetensors",
+    "model.layers.42.self_attn.embed_q.weight": "model-00044.safetensors",
+    "model.layers.42.self_attn.embed_q.scales": "model-00044.safetensors",
+    "model.layers.42.self_attn.embed_q.biases": "model-00044.safetensors",
+    "model.layers.42.self_attn.unembed_out.weight": "model-00044.safetensors",
+    "model.layers.42.self_attn.unembed_out.scales": "model-00044.safetensors",
+    "model.layers.42.self_attn.unembed_out.biases": "model-00044.safetensors",
+    "model.layers.43.input_layernorm.weight": "model-00045.safetensors",
+    "model.layers.43.mlp.gate.e_score_correction_bias": "model-00045.safetensors",
+    "model.layers.43.mlp.gate.weight": "model-00045.safetensors",
+    "model.layers.43.mlp.shared_experts.down_proj.weight": "model-00045.safetensors",
+    "model.layers.43.mlp.shared_experts.down_proj.scales": "model-00045.safetensors",
+    "model.layers.43.mlp.shared_experts.down_proj.biases": "model-00045.safetensors",
+    "model.layers.43.mlp.shared_experts.gate_proj.weight": "model-00045.safetensors",
+    "model.layers.43.mlp.shared_experts.gate_proj.scales": "model-00045.safetensors",
+    "model.layers.43.mlp.shared_experts.gate_proj.biases": "model-00045.safetensors",
+    "model.layers.43.mlp.shared_experts.up_proj.weight": "model-00045.safetensors",
+    "model.layers.43.mlp.shared_experts.up_proj.scales": "model-00045.safetensors",
+    "model.layers.43.mlp.shared_experts.up_proj.biases": "model-00045.safetensors",
+    "model.layers.43.post_attention_layernorm.weight": "model-00045.safetensors",
+    "model.layers.43.self_attn.kv_a_layernorm.weight": "model-00045.safetensors",
+    "model.layers.43.self_attn.kv_a_proj_with_mqa.weight": "model-00045.safetensors",
+    "model.layers.43.self_attn.kv_a_proj_with_mqa.scales": "model-00045.safetensors",
+    "model.layers.43.self_attn.kv_a_proj_with_mqa.biases": "model-00045.safetensors",
+    "model.layers.43.self_attn.o_proj.weight": "model-00045.safetensors",
+    "model.layers.43.self_attn.o_proj.scales": "model-00045.safetensors",
+    "model.layers.43.self_attn.o_proj.biases": "model-00045.safetensors",
+    "model.layers.43.self_attn.q_a_layernorm.weight": "model-00045.safetensors",
+    "model.layers.43.self_attn.q_a_proj.weight": "model-00045.safetensors",
+    "model.layers.43.self_attn.q_a_proj.scales": "model-00045.safetensors",
+    "model.layers.43.self_attn.q_a_proj.biases": "model-00045.safetensors",
+    "model.layers.43.self_attn.q_b_proj.weight": "model-00045.safetensors",
+    "model.layers.43.self_attn.q_b_proj.scales": "model-00045.safetensors",
+    "model.layers.43.self_attn.q_b_proj.biases": "model-00045.safetensors",
+    "model.layers.43.mlp.switch_mlp.gate_proj.weight": "model-00045.safetensors",
+    "model.layers.43.mlp.switch_mlp.gate_proj.scales": "model-00045.safetensors",
+    "model.layers.43.mlp.switch_mlp.gate_proj.biases": "model-00045.safetensors",
+    "model.layers.43.mlp.switch_mlp.down_proj.weight": "model-00045.safetensors",
+    "model.layers.43.mlp.switch_mlp.down_proj.scales": "model-00045.safetensors",
+    "model.layers.43.mlp.switch_mlp.down_proj.biases": "model-00045.safetensors",
+    "model.layers.43.mlp.switch_mlp.up_proj.weight": "model-00045.safetensors",
+    "model.layers.43.mlp.switch_mlp.up_proj.scales": "model-00045.safetensors",
+    "model.layers.43.mlp.switch_mlp.up_proj.biases": "model-00045.safetensors",
+    "model.layers.43.self_attn.embed_q.weight": "model-00045.safetensors",
+    "model.layers.43.self_attn.embed_q.scales": "model-00045.safetensors",
+    "model.layers.43.self_attn.embed_q.biases": "model-00045.safetensors",
+    "model.layers.43.self_attn.unembed_out.weight": "model-00045.safetensors",
+    "model.layers.43.self_attn.unembed_out.scales": "model-00045.safetensors",
+    "model.layers.43.self_attn.unembed_out.biases": "model-00045.safetensors",
+    "model.layers.44.input_layernorm.weight": "model-00046.safetensors",
+    "model.layers.44.mlp.gate.e_score_correction_bias": "model-00046.safetensors",
+    "model.layers.44.mlp.gate.weight": "model-00046.safetensors",
+    "model.layers.44.mlp.shared_experts.down_proj.weight": "model-00046.safetensors",
+    "model.layers.44.mlp.shared_experts.down_proj.scales": "model-00046.safetensors",
+    "model.layers.44.mlp.shared_experts.down_proj.biases": "model-00046.safetensors",
+    "model.layers.44.mlp.shared_experts.gate_proj.weight": "model-00046.safetensors",
+    "model.layers.44.mlp.shared_experts.gate_proj.scales": "model-00046.safetensors",
+    "model.layers.44.mlp.shared_experts.gate_proj.biases": "model-00046.safetensors",
+    "model.layers.44.mlp.shared_experts.up_proj.weight": "model-00046.safetensors",
+    "model.layers.44.mlp.shared_experts.up_proj.scales": "model-00046.safetensors",
+    "model.layers.44.mlp.shared_experts.up_proj.biases": "model-00046.safetensors",
+    "model.layers.44.post_attention_layernorm.weight": "model-00046.safetensors",
+    "model.layers.44.self_attn.kv_a_layernorm.weight": "model-00046.safetensors",
+    "model.layers.44.self_attn.kv_a_proj_with_mqa.weight": "model-00046.safetensors",
+    "model.layers.44.self_attn.kv_a_proj_with_mqa.scales": "model-00046.safetensors",
+    "model.layers.44.self_attn.kv_a_proj_with_mqa.biases": "model-00046.safetensors",
+    "model.layers.44.self_attn.o_proj.weight": "model-00046.safetensors",
+    "model.layers.44.self_attn.o_proj.scales": "model-00046.safetensors",
+    "model.layers.44.self_attn.o_proj.biases": "model-00046.safetensors",
+    "model.layers.44.self_attn.q_a_layernorm.weight": "model-00046.safetensors",
+    "model.layers.44.self_attn.q_a_proj.weight": "model-00046.safetensors",
+    "model.layers.44.self_attn.q_a_proj.scales": "model-00046.safetensors",
+    "model.layers.44.self_attn.q_a_proj.biases": "model-00046.safetensors",
+    "model.layers.44.self_attn.q_b_proj.weight": "model-00046.safetensors",
+    "model.layers.44.self_attn.q_b_proj.scales": "model-00046.safetensors",
+    "model.layers.44.self_attn.q_b_proj.biases": "model-00046.safetensors",
+    "model.layers.44.mlp.switch_mlp.gate_proj.weight": "model-00046.safetensors",
+    "model.layers.44.mlp.switch_mlp.gate_proj.scales": "model-00046.safetensors",
+    "model.layers.44.mlp.switch_mlp.gate_proj.biases": "model-00046.safetensors",
+    "model.layers.44.mlp.switch_mlp.down_proj.weight": "model-00046.safetensors",
+    "model.layers.44.mlp.switch_mlp.down_proj.scales": "model-00046.safetensors",
+    "model.layers.44.mlp.switch_mlp.down_proj.biases": "model-00046.safetensors",
+    "model.layers.44.mlp.switch_mlp.up_proj.weight": "model-00046.safetensors",
+    "model.layers.44.mlp.switch_mlp.up_proj.scales": "model-00046.safetensors",
+    "model.layers.44.mlp.switch_mlp.up_proj.biases": "model-00046.safetensors",
+    "model.layers.44.self_attn.embed_q.weight": "model-00046.safetensors",
+    "model.layers.44.self_attn.embed_q.scales": "model-00046.safetensors",
+    "model.layers.44.self_attn.embed_q.biases": "model-00046.safetensors",
+    "model.layers.44.self_attn.unembed_out.weight": "model-00046.safetensors",
+    "model.layers.44.self_attn.unembed_out.scales": "model-00046.safetensors",
+    "model.layers.44.self_attn.unembed_out.biases": "model-00046.safetensors",
+    "model.layers.45.input_layernorm.weight": "model-00047.safetensors",
+    "model.layers.45.mlp.gate.e_score_correction_bias": "model-00047.safetensors",
+    "model.layers.45.mlp.gate.weight": "model-00047.safetensors",
+    "model.layers.45.mlp.shared_experts.down_proj.weight": "model-00047.safetensors",
+    "model.layers.45.mlp.shared_experts.down_proj.scales": "model-00047.safetensors",
+    "model.layers.45.mlp.shared_experts.down_proj.biases": "model-00047.safetensors",
+    "model.layers.45.mlp.shared_experts.gate_proj.weight": "model-00047.safetensors",
+    "model.layers.45.mlp.shared_experts.gate_proj.scales": "model-00047.safetensors",
+    "model.layers.45.mlp.shared_experts.gate_proj.biases": "model-00047.safetensors",
+    "model.layers.45.mlp.shared_experts.up_proj.weight": "model-00047.safetensors",
+    "model.layers.45.mlp.shared_experts.up_proj.scales": "model-00047.safetensors",
+    "model.layers.45.mlp.shared_experts.up_proj.biases": "model-00047.safetensors",
+    "model.layers.45.post_attention_layernorm.weight": "model-00047.safetensors",
+    "model.layers.45.self_attn.kv_a_layernorm.weight": "model-00047.safetensors",
+    "model.layers.45.self_attn.kv_a_proj_with_mqa.weight": "model-00047.safetensors",
+    "model.layers.45.self_attn.kv_a_proj_with_mqa.scales": "model-00047.safetensors",
+    "model.layers.45.self_attn.kv_a_proj_with_mqa.biases": "model-00047.safetensors",
+    "model.layers.45.self_attn.o_proj.weight": "model-00047.safetensors",
+    "model.layers.45.self_attn.o_proj.scales": "model-00047.safetensors",
+    "model.layers.45.self_attn.o_proj.biases": "model-00047.safetensors",
+    "model.layers.45.self_attn.q_a_layernorm.weight": "model-00047.safetensors",
+    "model.layers.45.self_attn.q_a_proj.weight": "model-00047.safetensors",
+    "model.layers.45.self_attn.q_a_proj.scales": "model-00047.safetensors",
+    "model.layers.45.self_attn.q_a_proj.biases": "model-00047.safetensors",
+    "model.layers.45.self_attn.q_b_proj.weight": "model-00047.safetensors",
+    "model.layers.45.self_attn.q_b_proj.scales": "model-00047.safetensors",
+    "model.layers.45.self_attn.q_b_proj.biases": "model-00047.safetensors",
+    "model.layers.45.mlp.switch_mlp.gate_proj.weight": "model-00047.safetensors",
+    "model.layers.45.mlp.switch_mlp.gate_proj.scales": "model-00047.safetensors",
+    "model.layers.45.mlp.switch_mlp.gate_proj.biases": "model-00047.safetensors",
+    "model.layers.45.mlp.switch_mlp.down_proj.weight": "model-00047.safetensors",
+    "model.layers.45.mlp.switch_mlp.down_proj.scales": "model-00047.safetensors",
+    "model.layers.45.mlp.switch_mlp.down_proj.biases": "model-00047.safetensors",
+    "model.layers.45.mlp.switch_mlp.up_proj.weight": "model-00047.safetensors",
+    "model.layers.45.mlp.switch_mlp.up_proj.scales": "model-00047.safetensors",
+    "model.layers.45.mlp.switch_mlp.up_proj.biases": "model-00047.safetensors",
+    "model.layers.45.self_attn.embed_q.weight": "model-00047.safetensors",
+    "model.layers.45.self_attn.embed_q.scales": "model-00047.safetensors",
+    "model.layers.45.self_attn.embed_q.biases": "model-00047.safetensors",
+    "model.layers.45.self_attn.unembed_out.weight": "model-00047.safetensors",
+    "model.layers.45.self_attn.unembed_out.scales": "model-00047.safetensors",
+    "model.layers.45.self_attn.unembed_out.biases": "model-00047.safetensors",
+    "model.layers.46.input_layernorm.weight": "model-00048.safetensors",
+    "model.layers.46.mlp.gate.e_score_correction_bias": "model-00048.safetensors",
+    "model.layers.46.mlp.gate.weight": "model-00048.safetensors",
+    "model.layers.46.mlp.shared_experts.down_proj.weight": "model-00048.safetensors",
+    "model.layers.46.mlp.shared_experts.down_proj.scales": "model-00048.safetensors",
+    "model.layers.46.mlp.shared_experts.down_proj.biases": "model-00048.safetensors",
+    "model.layers.46.mlp.shared_experts.gate_proj.weight": "model-00048.safetensors",
+    "model.layers.46.mlp.shared_experts.gate_proj.scales": "model-00048.safetensors",
+    "model.layers.46.mlp.shared_experts.gate_proj.biases": "model-00048.safetensors",
+    "model.layers.46.mlp.shared_experts.up_proj.weight": "model-00048.safetensors",
+    "model.layers.46.mlp.shared_experts.up_proj.scales": "model-00048.safetensors",
+    "model.layers.46.mlp.shared_experts.up_proj.biases": "model-00048.safetensors",
+    "model.layers.46.post_attention_layernorm.weight": "model-00048.safetensors",
+    "model.layers.46.self_attn.indexer.k_norm.bias": "model-00048.safetensors",
+    "model.layers.46.self_attn.indexer.k_norm.weight": "model-00048.safetensors",
+    "model.layers.46.self_attn.indexer.weights_proj.weight": "model-00048.safetensors",
+    "model.layers.46.self_attn.indexer.wk.weight": "model-00048.safetensors",
+    "model.layers.46.self_attn.indexer.wq_b.weight": "model-00048.safetensors",
+    "model.layers.46.self_attn.kv_a_layernorm.weight": "model-00048.safetensors",
+    "model.layers.46.self_attn.kv_a_proj_with_mqa.weight": "model-00048.safetensors",
+    "model.layers.46.self_attn.kv_a_proj_with_mqa.scales": "model-00048.safetensors",
+    "model.layers.46.self_attn.kv_a_proj_with_mqa.biases": "model-00048.safetensors",
+    "model.layers.46.self_attn.o_proj.weight": "model-00048.safetensors",
+    "model.layers.46.self_attn.o_proj.scales": "model-00048.safetensors",
+    "model.layers.46.self_attn.o_proj.biases": "model-00048.safetensors",
+    "model.layers.46.self_attn.q_a_layernorm.weight": "model-00048.safetensors",
+    "model.layers.46.self_attn.q_a_proj.weight": "model-00048.safetensors",
+    "model.layers.46.self_attn.q_a_proj.scales": "model-00048.safetensors",
+    "model.layers.46.self_attn.q_a_proj.biases": "model-00048.safetensors",
+    "model.layers.46.self_attn.q_b_proj.weight": "model-00048.safetensors",
+    "model.layers.46.self_attn.q_b_proj.scales": "model-00048.safetensors",
+    "model.layers.46.self_attn.q_b_proj.biases": "model-00048.safetensors",
+    "model.layers.46.mlp.switch_mlp.gate_proj.weight": "model-00048.safetensors",
+    "model.layers.46.mlp.switch_mlp.gate_proj.scales": "model-00048.safetensors",
+    "model.layers.46.mlp.switch_mlp.gate_proj.biases": "model-00048.safetensors",
+    "model.layers.46.mlp.switch_mlp.down_proj.weight": "model-00048.safetensors",
+    "model.layers.46.mlp.switch_mlp.down_proj.scales": "model-00048.safetensors",
+    "model.layers.46.mlp.switch_mlp.down_proj.biases": "model-00048.safetensors",
+    "model.layers.46.mlp.switch_mlp.up_proj.weight": "model-00048.safetensors",
+    "model.layers.46.mlp.switch_mlp.up_proj.scales": "model-00048.safetensors",
+    "model.layers.46.mlp.switch_mlp.up_proj.biases": "model-00048.safetensors",
+    "model.layers.46.self_attn.embed_q.weight": "model-00048.safetensors",
+    "model.layers.46.self_attn.embed_q.scales": "model-00048.safetensors",
+    "model.layers.46.self_attn.embed_q.biases": "model-00048.safetensors",
+    "model.layers.46.self_attn.unembed_out.weight": "model-00048.safetensors",
+    "model.layers.46.self_attn.unembed_out.scales": "model-00048.safetensors",
+    "model.layers.46.self_attn.unembed_out.biases": "model-00048.safetensors",
+    "model.layers.47.input_layernorm.weight": "model-00049.safetensors",
+    "model.layers.47.mlp.gate.e_score_correction_bias": "model-00049.safetensors",
+    "model.layers.47.mlp.gate.weight": "model-00049.safetensors",
+    "model.layers.47.mlp.shared_experts.down_proj.weight": "model-00049.safetensors",
+    "model.layers.47.mlp.shared_experts.down_proj.scales": "model-00049.safetensors",
+    "model.layers.47.mlp.shared_experts.down_proj.biases": "model-00049.safetensors",
+    "model.layers.47.mlp.shared_experts.gate_proj.weight": "model-00049.safetensors",
+    "model.layers.47.mlp.shared_experts.gate_proj.scales": "model-00049.safetensors",
+    "model.layers.47.mlp.shared_experts.gate_proj.biases": "model-00049.safetensors",
+    "model.layers.47.mlp.shared_experts.up_proj.weight": "model-00049.safetensors",
+    "model.layers.47.mlp.shared_experts.up_proj.scales": "model-00049.safetensors",
+    "model.layers.47.mlp.shared_experts.up_proj.biases": "model-00049.safetensors",
+    "model.layers.47.post_attention_layernorm.weight": "model-00049.safetensors",
+    "model.layers.47.self_attn.kv_a_layernorm.weight": "model-00049.safetensors",
+    "model.layers.47.self_attn.kv_a_proj_with_mqa.weight": "model-00049.safetensors",
+    "model.layers.47.self_attn.kv_a_proj_with_mqa.scales": "model-00049.safetensors",
+    "model.layers.47.self_attn.kv_a_proj_with_mqa.biases": "model-00049.safetensors",
+    "model.layers.47.self_attn.o_proj.weight": "model-00049.safetensors",
+    "model.layers.47.self_attn.o_proj.scales": "model-00049.safetensors",
+    "model.layers.47.self_attn.o_proj.biases": "model-00049.safetensors",
+    "model.layers.47.self_attn.q_a_layernorm.weight": "model-00049.safetensors",
+    "model.layers.47.self_attn.q_a_proj.weight": "model-00049.safetensors",
+    "model.layers.47.self_attn.q_a_proj.scales": "model-00049.safetensors",
+    "model.layers.47.self_attn.q_a_proj.biases": "model-00049.safetensors",
+    "model.layers.47.self_attn.q_b_proj.weight": "model-00049.safetensors",
+    "model.layers.47.self_attn.q_b_proj.scales": "model-00049.safetensors",
+    "model.layers.47.self_attn.q_b_proj.biases": "model-00049.safetensors",
+    "model.layers.47.mlp.switch_mlp.gate_proj.weight": "model-00049.safetensors",
+    "model.layers.47.mlp.switch_mlp.gate_proj.scales": "model-00049.safetensors",
+    "model.layers.47.mlp.switch_mlp.gate_proj.biases": "model-00049.safetensors",
+    "model.layers.47.mlp.switch_mlp.down_proj.weight": "model-00049.safetensors",
+    "model.layers.47.mlp.switch_mlp.down_proj.scales": "model-00049.safetensors",
+    "model.layers.47.mlp.switch_mlp.down_proj.biases": "model-00049.safetensors",
+    "model.layers.47.mlp.switch_mlp.up_proj.weight": "model-00049.safetensors",
+    "model.layers.47.mlp.switch_mlp.up_proj.scales": "model-00049.safetensors",
+    "model.layers.47.mlp.switch_mlp.up_proj.biases": "model-00049.safetensors",
+    "model.layers.47.self_attn.embed_q.weight": "model-00049.safetensors",
+    "model.layers.47.self_attn.embed_q.scales": "model-00049.safetensors",
+    "model.layers.47.self_attn.embed_q.biases": "model-00049.safetensors",
+    "model.layers.47.self_attn.unembed_out.weight": "model-00049.safetensors",
+    "model.layers.47.self_attn.unembed_out.scales": "model-00049.safetensors",
+    "model.layers.47.self_attn.unembed_out.biases": "model-00049.safetensors",
+    "model.layers.48.input_layernorm.weight": "model-00050.safetensors",
+    "model.layers.48.mlp.gate.e_score_correction_bias": "model-00050.safetensors",
+    "model.layers.48.mlp.gate.weight": "model-00050.safetensors",
+    "model.layers.48.mlp.shared_experts.down_proj.weight": "model-00050.safetensors",
+    "model.layers.48.mlp.shared_experts.down_proj.scales": "model-00050.safetensors",
+    "model.layers.48.mlp.shared_experts.down_proj.biases": "model-00050.safetensors",
+    "model.layers.48.mlp.shared_experts.gate_proj.weight": "model-00050.safetensors",
+    "model.layers.48.mlp.shared_experts.gate_proj.scales": "model-00050.safetensors",
+    "model.layers.48.mlp.shared_experts.gate_proj.biases": "model-00050.safetensors",
+    "model.layers.48.mlp.shared_experts.up_proj.weight": "model-00050.safetensors",
+    "model.layers.48.mlp.shared_experts.up_proj.scales": "model-00050.safetensors",
+    "model.layers.48.mlp.shared_experts.up_proj.biases": "model-00050.safetensors",
+    "model.layers.48.post_attention_layernorm.weight": "model-00050.safetensors",
+    "model.layers.48.self_attn.kv_a_layernorm.weight": "model-00050.safetensors",
+    "model.layers.48.self_attn.kv_a_proj_with_mqa.weight": "model-00050.safetensors",
+    "model.layers.48.self_attn.kv_a_proj_with_mqa.scales": "model-00050.safetensors",
+    "model.layers.48.self_attn.kv_a_proj_with_mqa.biases": "model-00050.safetensors",
+    "model.layers.48.self_attn.o_proj.weight": "model-00050.safetensors",
+    "model.layers.48.self_attn.o_proj.scales": "model-00050.safetensors",
+    "model.layers.48.self_attn.o_proj.biases": "model-00050.safetensors",
+    "model.layers.48.self_attn.q_a_layernorm.weight": "model-00050.safetensors",
+    "model.layers.48.self_attn.q_a_proj.weight": "model-00050.safetensors",
+    "model.layers.48.self_attn.q_a_proj.scales": "model-00050.safetensors",
+    "model.layers.48.self_attn.q_a_proj.biases": "model-00050.safetensors",
+    "model.layers.48.self_attn.q_b_proj.weight": "model-00050.safetensors",
+    "model.layers.48.self_attn.q_b_proj.scales": "model-00050.safetensors",
+    "model.layers.48.self_attn.q_b_proj.biases": "model-00050.safetensors",
+    "model.layers.48.mlp.switch_mlp.gate_proj.weight": "model-00050.safetensors",
+    "model.layers.48.mlp.switch_mlp.gate_proj.scales": "model-00050.safetensors",
+    "model.layers.48.mlp.switch_mlp.gate_proj.biases": "model-00050.safetensors",
+    "model.layers.48.mlp.switch_mlp.down_proj.weight": "model-00050.safetensors",
+    "model.layers.48.mlp.switch_mlp.down_proj.scales": "model-00050.safetensors",
+    "model.layers.48.mlp.switch_mlp.down_proj.biases": "model-00050.safetensors",
+    "model.layers.48.mlp.switch_mlp.up_proj.weight": "model-00050.safetensors",
+    "model.layers.48.mlp.switch_mlp.up_proj.scales": "model-00050.safetensors",
+    "model.layers.48.mlp.switch_mlp.up_proj.biases": "model-00050.safetensors",
+    "model.layers.48.self_attn.embed_q.weight": "model-00050.safetensors",
+    "model.layers.48.self_attn.embed_q.scales": "model-00050.safetensors",
+    "model.layers.48.self_attn.embed_q.biases": "model-00050.safetensors",
+    "model.layers.48.self_attn.unembed_out.weight": "model-00050.safetensors",
+    "model.layers.48.self_attn.unembed_out.scales": "model-00050.safetensors",
+    "model.layers.48.self_attn.unembed_out.biases": "model-00050.safetensors",
+    "model.layers.49.input_layernorm.weight": "model-00051.safetensors",
+    "model.layers.49.mlp.gate.e_score_correction_bias": "model-00051.safetensors",
+    "model.layers.49.mlp.gate.weight": "model-00051.safetensors",
+    "model.layers.49.mlp.shared_experts.down_proj.weight": "model-00051.safetensors",
+    "model.layers.49.mlp.shared_experts.down_proj.scales": "model-00051.safetensors",
+    "model.layers.49.mlp.shared_experts.down_proj.biases": "model-00051.safetensors",
+    "model.layers.49.mlp.shared_experts.gate_proj.weight": "model-00051.safetensors",
+    "model.layers.49.mlp.shared_experts.gate_proj.scales": "model-00051.safetensors",
+    "model.layers.49.mlp.shared_experts.gate_proj.biases": "model-00051.safetensors",
+    "model.layers.49.mlp.shared_experts.up_proj.weight": "model-00051.safetensors",
+    "model.layers.49.mlp.shared_experts.up_proj.scales": "model-00051.safetensors",
+    "model.layers.49.mlp.shared_experts.up_proj.biases": "model-00051.safetensors",
+    "model.layers.49.post_attention_layernorm.weight": "model-00051.safetensors",
+    "model.layers.49.self_attn.kv_a_layernorm.weight": "model-00051.safetensors",
+    "model.layers.49.self_attn.kv_a_proj_with_mqa.weight": "model-00051.safetensors",
+    "model.layers.49.self_attn.kv_a_proj_with_mqa.scales": "model-00051.safetensors",
+    "model.layers.49.self_attn.kv_a_proj_with_mqa.biases": "model-00051.safetensors",
+    "model.layers.49.self_attn.o_proj.weight": "model-00051.safetensors",
+    "model.layers.49.self_attn.o_proj.scales": "model-00051.safetensors",
+    "model.layers.49.self_attn.o_proj.biases": "model-00051.safetensors",
+    "model.layers.49.self_attn.q_a_layernorm.weight": "model-00051.safetensors",
+    "model.layers.49.self_attn.q_a_proj.weight": "model-00051.safetensors",
+    "model.layers.49.self_attn.q_a_proj.scales": "model-00051.safetensors",
+    "model.layers.49.self_attn.q_a_proj.biases": "model-00051.safetensors",
+    "model.layers.49.self_attn.q_b_proj.weight": "model-00051.safetensors",
+    "model.layers.49.self_attn.q_b_proj.scales": "model-00051.safetensors",
+    "model.layers.49.self_attn.q_b_proj.biases": "model-00051.safetensors",
+    "model.layers.49.mlp.switch_mlp.gate_proj.weight": "model-00051.safetensors",
+    "model.layers.49.mlp.switch_mlp.gate_proj.scales": "model-00051.safetensors",
+    "model.layers.49.mlp.switch_mlp.gate_proj.biases": "model-00051.safetensors",
+    "model.layers.49.mlp.switch_mlp.down_proj.weight": "model-00051.safetensors",
+    "model.layers.49.mlp.switch_mlp.down_proj.scales": "model-00051.safetensors",
+    "model.layers.49.mlp.switch_mlp.down_proj.biases": "model-00051.safetensors",
+    "model.layers.49.mlp.switch_mlp.up_proj.weight": "model-00051.safetensors",
+    "model.layers.49.mlp.switch_mlp.up_proj.scales": "model-00051.safetensors",
+    "model.layers.49.mlp.switch_mlp.up_proj.biases": "model-00051.safetensors",
+    "model.layers.49.self_attn.embed_q.weight": "model-00051.safetensors",
+    "model.layers.49.self_attn.embed_q.scales": "model-00051.safetensors",
+    "model.layers.49.self_attn.embed_q.biases": "model-00051.safetensors",
+    "model.layers.49.self_attn.unembed_out.weight": "model-00051.safetensors",
+    "model.layers.49.self_attn.unembed_out.scales": "model-00051.safetensors",
+    "model.layers.49.self_attn.unembed_out.biases": "model-00051.safetensors",
+    "model.layers.50.input_layernorm.weight": "model-00052.safetensors",
+    "model.layers.50.mlp.gate.e_score_correction_bias": "model-00052.safetensors",
+    "model.layers.50.mlp.gate.weight": "model-00052.safetensors",
+    "model.layers.50.mlp.shared_experts.down_proj.weight": "model-00052.safetensors",
+    "model.layers.50.mlp.shared_experts.down_proj.scales": "model-00052.safetensors",
+    "model.layers.50.mlp.shared_experts.down_proj.biases": "model-00052.safetensors",
+    "model.layers.50.mlp.shared_experts.gate_proj.weight": "model-00052.safetensors",
+    "model.layers.50.mlp.shared_experts.gate_proj.scales": "model-00052.safetensors",
+    "model.layers.50.mlp.shared_experts.gate_proj.biases": "model-00052.safetensors",
+    "model.layers.50.mlp.shared_experts.up_proj.weight": "model-00052.safetensors",
+    "model.layers.50.mlp.shared_experts.up_proj.scales": "model-00052.safetensors",
+    "model.layers.50.mlp.shared_experts.up_proj.biases": "model-00052.safetensors",
+    "model.layers.50.post_attention_layernorm.weight": "model-00052.safetensors",
+    "model.layers.50.self_attn.indexer.k_norm.bias": "model-00052.safetensors",
+    "model.layers.50.self_attn.indexer.k_norm.weight": "model-00052.safetensors",
+    "model.layers.50.self_attn.indexer.weights_proj.weight": "model-00052.safetensors",
+    "model.layers.50.self_attn.indexer.wk.weight": "model-00052.safetensors",
+    "model.layers.50.self_attn.indexer.wq_b.weight": "model-00052.safetensors",
+    "model.layers.50.self_attn.kv_a_layernorm.weight": "model-00052.safetensors",
+    "model.layers.50.self_attn.kv_a_proj_with_mqa.weight": "model-00052.safetensors",
+    "model.layers.50.self_attn.kv_a_proj_with_mqa.scales": "model-00052.safetensors",
+    "model.layers.50.self_attn.kv_a_proj_with_mqa.biases": "model-00052.safetensors",
+    "model.layers.50.self_attn.o_proj.weight": "model-00052.safetensors",
+    "model.layers.50.self_attn.o_proj.scales": "model-00052.safetensors",
+    "model.layers.50.self_attn.o_proj.biases": "model-00052.safetensors",
+    "model.layers.50.self_attn.q_a_layernorm.weight": "model-00052.safetensors",
+    "model.layers.50.self_attn.q_a_proj.weight": "model-00052.safetensors",
+    "model.layers.50.self_attn.q_a_proj.scales": "model-00052.safetensors",
+    "model.layers.50.self_attn.q_a_proj.biases": "model-00052.safetensors",
+    "model.layers.50.self_attn.q_b_proj.weight": "model-00052.safetensors",
+    "model.layers.50.self_attn.q_b_proj.scales": "model-00052.safetensors",
+    "model.layers.50.self_attn.q_b_proj.biases": "model-00052.safetensors",
+    "model.layers.50.mlp.switch_mlp.gate_proj.weight": "model-00052.safetensors",
+    "model.layers.50.mlp.switch_mlp.gate_proj.scales": "model-00052.safetensors",
+    "model.layers.50.mlp.switch_mlp.gate_proj.biases": "model-00052.safetensors",
+    "model.layers.50.mlp.switch_mlp.down_proj.weight": "model-00052.safetensors",
+    "model.layers.50.mlp.switch_mlp.down_proj.scales": "model-00052.safetensors",
+    "model.layers.50.mlp.switch_mlp.down_proj.biases": "model-00052.safetensors",
+    "model.layers.50.mlp.switch_mlp.up_proj.weight": "model-00052.safetensors",
+    "model.layers.50.mlp.switch_mlp.up_proj.scales": "model-00052.safetensors",
+    "model.layers.50.mlp.switch_mlp.up_proj.biases": "model-00052.safetensors",
+    "model.layers.50.self_attn.embed_q.weight": "model-00052.safetensors",
+    "model.layers.50.self_attn.embed_q.scales": "model-00052.safetensors",
+    "model.layers.50.self_attn.embed_q.biases": "model-00052.safetensors",
+    "model.layers.50.self_attn.unembed_out.weight": "model-00052.safetensors",
+    "model.layers.50.self_attn.unembed_out.scales": "model-00052.safetensors",
+    "model.layers.50.self_attn.unembed_out.biases": "model-00052.safetensors",
+    "model.layers.51.input_layernorm.weight": "model-00053.safetensors",
+    "model.layers.51.mlp.gate.e_score_correction_bias": "model-00053.safetensors",
+    "model.layers.51.mlp.gate.weight": "model-00053.safetensors",
+    "model.layers.51.mlp.shared_experts.down_proj.weight": "model-00053.safetensors",
+    "model.layers.51.mlp.shared_experts.down_proj.scales": "model-00053.safetensors",
+    "model.layers.51.mlp.shared_experts.down_proj.biases": "model-00053.safetensors",
+    "model.layers.51.mlp.shared_experts.gate_proj.weight": "model-00053.safetensors",
+    "model.layers.51.mlp.shared_experts.gate_proj.scales": "model-00053.safetensors",
+    "model.layers.51.mlp.shared_experts.gate_proj.biases": "model-00053.safetensors",
+    "model.layers.51.mlp.shared_experts.up_proj.weight": "model-00053.safetensors",
+    "model.layers.51.mlp.shared_experts.up_proj.scales": "model-00053.safetensors",
+    "model.layers.51.mlp.shared_experts.up_proj.biases": "model-00053.safetensors",
+    "model.layers.51.post_attention_layernorm.weight": "model-00053.safetensors",
+    "model.layers.51.self_attn.kv_a_layernorm.weight": "model-00053.safetensors",
+    "model.layers.51.self_attn.kv_a_proj_with_mqa.weight": "model-00053.safetensors",
+    "model.layers.51.self_attn.kv_a_proj_with_mqa.scales": "model-00053.safetensors",
+    "model.layers.51.self_attn.kv_a_proj_with_mqa.biases": "model-00053.safetensors",
+    "model.layers.51.self_attn.o_proj.weight": "model-00053.safetensors",
+    "model.layers.51.self_attn.o_proj.scales": "model-00053.safetensors",
+    "model.layers.51.self_attn.o_proj.biases": "model-00053.safetensors",
+    "model.layers.51.self_attn.q_a_layernorm.weight": "model-00053.safetensors",
+    "model.layers.51.self_attn.q_a_proj.weight": "model-00053.safetensors",
+    "model.layers.51.self_attn.q_a_proj.scales": "model-00053.safetensors",
+    "model.layers.51.self_attn.q_a_proj.biases": "model-00053.safetensors",
+    "model.layers.51.self_attn.q_b_proj.weight": "model-00053.safetensors",
+    "model.layers.51.self_attn.q_b_proj.scales": "model-00053.safetensors",
+    "model.layers.51.self_attn.q_b_proj.biases": "model-00053.safetensors",
+    "model.layers.51.mlp.switch_mlp.gate_proj.weight": "model-00053.safetensors",
+    "model.layers.51.mlp.switch_mlp.gate_proj.scales": "model-00053.safetensors",
+    "model.layers.51.mlp.switch_mlp.gate_proj.biases": "model-00053.safetensors",
+    "model.layers.51.mlp.switch_mlp.down_proj.weight": "model-00053.safetensors",
+    "model.layers.51.mlp.switch_mlp.down_proj.scales": "model-00053.safetensors",
+    "model.layers.51.mlp.switch_mlp.down_proj.biases": "model-00053.safetensors",
+    "model.layers.51.mlp.switch_mlp.up_proj.weight": "model-00053.safetensors",
+    "model.layers.51.mlp.switch_mlp.up_proj.scales": "model-00053.safetensors",
+    "model.layers.51.mlp.switch_mlp.up_proj.biases": "model-00053.safetensors",
+    "model.layers.51.self_attn.embed_q.weight": "model-00053.safetensors",
+    "model.layers.51.self_attn.embed_q.scales": "model-00053.safetensors",
+    "model.layers.51.self_attn.embed_q.biases": "model-00053.safetensors",
+    "model.layers.51.self_attn.unembed_out.weight": "model-00053.safetensors",
+    "model.layers.51.self_attn.unembed_out.scales": "model-00053.safetensors",
+    "model.layers.51.self_attn.unembed_out.biases": "model-00053.safetensors",
+    "model.layers.52.input_layernorm.weight": "model-00054.safetensors",
+    "model.layers.52.mlp.gate.e_score_correction_bias": "model-00054.safetensors",
+    "model.layers.52.mlp.gate.weight": "model-00054.safetensors",
+    "model.layers.52.mlp.shared_experts.down_proj.weight": "model-00054.safetensors",
+    "model.layers.52.mlp.shared_experts.down_proj.scales": "model-00054.safetensors",
+    "model.layers.52.mlp.shared_experts.down_proj.biases": "model-00054.safetensors",
+    "model.layers.52.mlp.shared_experts.gate_proj.weight": "model-00054.safetensors",
+    "model.layers.52.mlp.shared_experts.gate_proj.scales": "model-00054.safetensors",
+    "model.layers.52.mlp.shared_experts.gate_proj.biases": "model-00054.safetensors",
+    "model.layers.52.mlp.shared_experts.up_proj.weight": "model-00054.safetensors",
+    "model.layers.52.mlp.shared_experts.up_proj.scales": "model-00054.safetensors",
+    "model.layers.52.mlp.shared_experts.up_proj.biases": "model-00054.safetensors",
+    "model.layers.52.post_attention_layernorm.weight": "model-00054.safetensors",
+    "model.layers.52.self_attn.kv_a_layernorm.weight": "model-00054.safetensors",
+    "model.layers.52.self_attn.kv_a_proj_with_mqa.weight": "model-00054.safetensors",
+    "model.layers.52.self_attn.kv_a_proj_with_mqa.scales": "model-00054.safetensors",
+    "model.layers.52.self_attn.kv_a_proj_with_mqa.biases": "model-00054.safetensors",
+    "model.layers.52.self_attn.o_proj.weight": "model-00054.safetensors",
+    "model.layers.52.self_attn.o_proj.scales": "model-00054.safetensors",
+    "model.layers.52.self_attn.o_proj.biases": "model-00054.safetensors",
+    "model.layers.52.self_attn.q_a_layernorm.weight": "model-00054.safetensors",
+    "model.layers.52.self_attn.q_a_proj.weight": "model-00054.safetensors",
+    "model.layers.52.self_attn.q_a_proj.scales": "model-00054.safetensors",
+    "model.layers.52.self_attn.q_a_proj.biases": "model-00054.safetensors",
+    "model.layers.52.self_attn.q_b_proj.weight": "model-00054.safetensors",
+    "model.layers.52.self_attn.q_b_proj.scales": "model-00054.safetensors",
+    "model.layers.52.self_attn.q_b_proj.biases": "model-00054.safetensors",
+    "model.layers.52.mlp.switch_mlp.gate_proj.weight": "model-00054.safetensors",
+    "model.layers.52.mlp.switch_mlp.gate_proj.scales": "model-00054.safetensors",
+    "model.layers.52.mlp.switch_mlp.gate_proj.biases": "model-00054.safetensors",
+    "model.layers.52.mlp.switch_mlp.down_proj.weight": "model-00054.safetensors",
+    "model.layers.52.mlp.switch_mlp.down_proj.scales": "model-00054.safetensors",
+    "model.layers.52.mlp.switch_mlp.down_proj.biases": "model-00054.safetensors",
+    "model.layers.52.mlp.switch_mlp.up_proj.weight": "model-00054.safetensors",
+    "model.layers.52.mlp.switch_mlp.up_proj.scales": "model-00054.safetensors",
+    "model.layers.52.mlp.switch_mlp.up_proj.biases": "model-00054.safetensors",
+    "model.layers.52.self_attn.embed_q.weight": "model-00054.safetensors",
+    "model.layers.52.self_attn.embed_q.scales": "model-00054.safetensors",
+    "model.layers.52.self_attn.embed_q.biases": "model-00054.safetensors",
+    "model.layers.52.self_attn.unembed_out.weight": "model-00054.safetensors",
+    "model.layers.52.self_attn.unembed_out.scales": "model-00054.safetensors",
+    "model.layers.52.self_attn.unembed_out.biases": "model-00054.safetensors",
+    "model.layers.53.input_layernorm.weight": "model-00055.safetensors",
+    "model.layers.53.mlp.gate.e_score_correction_bias": "model-00055.safetensors",
+    "model.layers.53.mlp.gate.weight": "model-00055.safetensors",
+    "model.layers.53.mlp.shared_experts.down_proj.weight": "model-00055.safetensors",
+    "model.layers.53.mlp.shared_experts.down_proj.scales": "model-00055.safetensors",
+    "model.layers.53.mlp.shared_experts.down_proj.biases": "model-00055.safetensors",
+    "model.layers.53.mlp.shared_experts.gate_proj.weight": "model-00055.safetensors",
+    "model.layers.53.mlp.shared_experts.gate_proj.scales": "model-00055.safetensors",
+    "model.layers.53.mlp.shared_experts.gate_proj.biases": "model-00055.safetensors",
+    "model.layers.53.mlp.shared_experts.up_proj.weight": "model-00055.safetensors",
+    "model.layers.53.mlp.shared_experts.up_proj.scales": "model-00055.safetensors",
+    "model.layers.53.mlp.shared_experts.up_proj.biases": "model-00055.safetensors",
+    "model.layers.53.post_attention_layernorm.weight": "model-00055.safetensors",
+    "model.layers.53.self_attn.kv_a_layernorm.weight": "model-00055.safetensors",
+    "model.layers.53.self_attn.kv_a_proj_with_mqa.weight": "model-00055.safetensors",
+    "model.layers.53.self_attn.kv_a_proj_with_mqa.scales": "model-00055.safetensors",
+    "model.layers.53.self_attn.kv_a_proj_with_mqa.biases": "model-00055.safetensors",
+    "model.layers.53.self_attn.o_proj.weight": "model-00055.safetensors",
+    "model.layers.53.self_attn.o_proj.scales": "model-00055.safetensors",
+    "model.layers.53.self_attn.o_proj.biases": "model-00055.safetensors",
+    "model.layers.53.self_attn.q_a_layernorm.weight": "model-00055.safetensors",
+    "model.layers.53.self_attn.q_a_proj.weight": "model-00055.safetensors",
+    "model.layers.53.self_attn.q_a_proj.scales": "model-00055.safetensors",
+    "model.layers.53.self_attn.q_a_proj.biases": "model-00055.safetensors",
+    "model.layers.53.self_attn.q_b_proj.weight": "model-00055.safetensors",
+    "model.layers.53.self_attn.q_b_proj.scales": "model-00055.safetensors",
+    "model.layers.53.self_attn.q_b_proj.biases": "model-00055.safetensors",
+    "model.layers.53.mlp.switch_mlp.gate_proj.weight": "model-00055.safetensors",
+    "model.layers.53.mlp.switch_mlp.gate_proj.scales": "model-00055.safetensors",
+    "model.layers.53.mlp.switch_mlp.gate_proj.biases": "model-00055.safetensors",
+    "model.layers.53.mlp.switch_mlp.down_proj.weight": "model-00055.safetensors",
+    "model.layers.53.mlp.switch_mlp.down_proj.scales": "model-00055.safetensors",
+    "model.layers.53.mlp.switch_mlp.down_proj.biases": "model-00055.safetensors",
+    "model.layers.53.mlp.switch_mlp.up_proj.weight": "model-00055.safetensors",
+    "model.layers.53.mlp.switch_mlp.up_proj.scales": "model-00055.safetensors",
+    "model.layers.53.mlp.switch_mlp.up_proj.biases": "model-00055.safetensors",
+    "model.layers.53.self_attn.embed_q.weight": "model-00055.safetensors",
+    "model.layers.53.self_attn.embed_q.scales": "model-00055.safetensors",
+    "model.layers.53.self_attn.embed_q.biases": "model-00055.safetensors",
+    "model.layers.53.self_attn.unembed_out.weight": "model-00055.safetensors",
+    "model.layers.53.self_attn.unembed_out.scales": "model-00055.safetensors",
+    "model.layers.53.self_attn.unembed_out.biases": "model-00055.safetensors",
+    "model.layers.54.input_layernorm.weight": "model-00056.safetensors",
+    "model.layers.54.mlp.gate.e_score_correction_bias": "model-00056.safetensors",
+    "model.layers.54.mlp.gate.weight": "model-00056.safetensors",
+    "model.layers.54.mlp.shared_experts.down_proj.weight": "model-00056.safetensors",
+    "model.layers.54.mlp.shared_experts.down_proj.scales": "model-00056.safetensors",
+    "model.layers.54.mlp.shared_experts.down_proj.biases": "model-00056.safetensors",
+    "model.layers.54.mlp.shared_experts.gate_proj.weight": "model-00056.safetensors",
+    "model.layers.54.mlp.shared_experts.gate_proj.scales": "model-00056.safetensors",
+    "model.layers.54.mlp.shared_experts.gate_proj.biases": "model-00056.safetensors",
+    "model.layers.54.mlp.shared_experts.up_proj.weight": "model-00056.safetensors",
+    "model.layers.54.mlp.shared_experts.up_proj.scales": "model-00056.safetensors",
+    "model.layers.54.mlp.shared_experts.up_proj.biases": "model-00056.safetensors",
+    "model.layers.54.post_attention_layernorm.weight": "model-00056.safetensors",
+    "model.layers.54.self_attn.indexer.k_norm.bias": "model-00056.safetensors",
+    "model.layers.54.self_attn.indexer.k_norm.weight": "model-00056.safetensors",
+    "model.layers.54.self_attn.indexer.weights_proj.weight": "model-00056.safetensors",
+    "model.layers.54.self_attn.indexer.wk.weight": "model-00056.safetensors",
+    "model.layers.54.self_attn.indexer.wq_b.weight": "model-00056.safetensors",
+    "model.layers.54.self_attn.kv_a_layernorm.weight": "model-00056.safetensors",
+    "model.layers.54.self_attn.kv_a_proj_with_mqa.weight": "model-00056.safetensors",
+    "model.layers.54.self_attn.kv_a_proj_with_mqa.scales": "model-00056.safetensors",
+    "model.layers.54.self_attn.kv_a_proj_with_mqa.biases": "model-00056.safetensors",
+    "model.layers.54.self_attn.o_proj.weight": "model-00056.safetensors",
+    "model.layers.54.self_attn.o_proj.scales": "model-00056.safetensors",
+    "model.layers.54.self_attn.o_proj.biases": "model-00056.safetensors",
+    "model.layers.54.self_attn.q_a_layernorm.weight": "model-00056.safetensors",
+    "model.layers.54.self_attn.q_a_proj.weight": "model-00056.safetensors",
+    "model.layers.54.self_attn.q_a_proj.scales": "model-00056.safetensors",
+    "model.layers.54.self_attn.q_a_proj.biases": "model-00056.safetensors",
+    "model.layers.54.self_attn.q_b_proj.weight": "model-00056.safetensors",
+    "model.layers.54.self_attn.q_b_proj.scales": "model-00056.safetensors",
+    "model.layers.54.self_attn.q_b_proj.biases": "model-00056.safetensors",
+    "model.layers.54.mlp.switch_mlp.gate_proj.weight": "model-00056.safetensors",
+    "model.layers.54.mlp.switch_mlp.gate_proj.scales": "model-00056.safetensors",
+    "model.layers.54.mlp.switch_mlp.gate_proj.biases": "model-00056.safetensors",
+    "model.layers.54.mlp.switch_mlp.down_proj.weight": "model-00056.safetensors",
+    "model.layers.54.mlp.switch_mlp.down_proj.scales": "model-00056.safetensors",
+    "model.layers.54.mlp.switch_mlp.down_proj.biases": "model-00056.safetensors",
+    "model.layers.54.mlp.switch_mlp.up_proj.weight": "model-00056.safetensors",
+    "model.layers.54.mlp.switch_mlp.up_proj.scales": "model-00056.safetensors",
+    "model.layers.54.mlp.switch_mlp.up_proj.biases": "model-00056.safetensors",
+    "model.layers.54.self_attn.embed_q.weight": "model-00056.safetensors",
+    "model.layers.54.self_attn.embed_q.scales": "model-00056.safetensors",
+    "model.layers.54.self_attn.embed_q.biases": "model-00056.safetensors",
+    "model.layers.54.self_attn.unembed_out.weight": "model-00056.safetensors",
+    "model.layers.54.self_attn.unembed_out.scales": "model-00056.safetensors",
+    "model.layers.54.self_attn.unembed_out.biases": "model-00056.safetensors",
+    "model.layers.55.input_layernorm.weight": "model-00057.safetensors",
+    "model.layers.55.mlp.gate.e_score_correction_bias": "model-00057.safetensors",
+    "model.layers.55.mlp.gate.weight": "model-00057.safetensors",
+    "model.layers.55.mlp.shared_experts.down_proj.weight": "model-00057.safetensors",
+    "model.layers.55.mlp.shared_experts.down_proj.scales": "model-00057.safetensors",
+    "model.layers.55.mlp.shared_experts.down_proj.biases": "model-00057.safetensors",
+    "model.layers.55.mlp.shared_experts.gate_proj.weight": "model-00057.safetensors",
+    "model.layers.55.mlp.shared_experts.gate_proj.scales": "model-00057.safetensors",
+    "model.layers.55.mlp.shared_experts.gate_proj.biases": "model-00057.safetensors",
+    "model.layers.55.mlp.shared_experts.up_proj.weight": "model-00057.safetensors",
+    "model.layers.55.mlp.shared_experts.up_proj.scales": "model-00057.safetensors",
+    "model.layers.55.mlp.shared_experts.up_proj.biases": "model-00057.safetensors",
+    "model.layers.55.post_attention_layernorm.weight": "model-00057.safetensors",
+    "model.layers.55.self_attn.kv_a_layernorm.weight": "model-00057.safetensors",
+    "model.layers.55.self_attn.kv_a_proj_with_mqa.weight": "model-00057.safetensors",
+    "model.layers.55.self_attn.kv_a_proj_with_mqa.scales": "model-00057.safetensors",
+    "model.layers.55.self_attn.kv_a_proj_with_mqa.biases": "model-00057.safetensors",
+    "model.layers.55.self_attn.o_proj.weight": "model-00057.safetensors",
+    "model.layers.55.self_attn.o_proj.scales": "model-00057.safetensors",
+    "model.layers.55.self_attn.o_proj.biases": "model-00057.safetensors",
+    "model.layers.55.self_attn.q_a_layernorm.weight": "model-00057.safetensors",
+    "model.layers.55.self_attn.q_a_proj.weight": "model-00057.safetensors",
+    "model.layers.55.self_attn.q_a_proj.scales": "model-00057.safetensors",
+    "model.layers.55.self_attn.q_a_proj.biases": "model-00057.safetensors",
+    "model.layers.55.self_attn.q_b_proj.weight": "model-00057.safetensors",
+    "model.layers.55.self_attn.q_b_proj.scales": "model-00057.safetensors",
+    "model.layers.55.self_attn.q_b_proj.biases": "model-00057.safetensors",
+    "model.layers.55.mlp.switch_mlp.gate_proj.weight": "model-00057.safetensors",
+    "model.layers.55.mlp.switch_mlp.gate_proj.scales": "model-00057.safetensors",
+    "model.layers.55.mlp.switch_mlp.gate_proj.biases": "model-00057.safetensors",
+    "model.layers.55.mlp.switch_mlp.down_proj.weight": "model-00057.safetensors",
+    "model.layers.55.mlp.switch_mlp.down_proj.scales": "model-00057.safetensors",
+    "model.layers.55.mlp.switch_mlp.down_proj.biases": "model-00057.safetensors",
+    "model.layers.55.mlp.switch_mlp.up_proj.weight": "model-00057.safetensors",
+    "model.layers.55.mlp.switch_mlp.up_proj.scales": "model-00057.safetensors",
+    "model.layers.55.mlp.switch_mlp.up_proj.biases": "model-00057.safetensors",
+    "model.layers.55.self_attn.embed_q.weight": "model-00057.safetensors",
+    "model.layers.55.self_attn.embed_q.scales": "model-00057.safetensors",
+    "model.layers.55.self_attn.embed_q.biases": "model-00057.safetensors",
+    "model.layers.55.self_attn.unembed_out.weight": "model-00057.safetensors",
+    "model.layers.55.self_attn.unembed_out.scales": "model-00057.safetensors",
+    "model.layers.55.self_attn.unembed_out.biases": "model-00057.safetensors",
+    "model.layers.56.input_layernorm.weight": "model-00058.safetensors",
+    "model.layers.56.mlp.gate.e_score_correction_bias": "model-00058.safetensors",
+    "model.layers.56.mlp.gate.weight": "model-00058.safetensors",
+    "model.layers.56.mlp.shared_experts.down_proj.weight": "model-00058.safetensors",
+    "model.layers.56.mlp.shared_experts.down_proj.scales": "model-00058.safetensors",
+    "model.layers.56.mlp.shared_experts.down_proj.biases": "model-00058.safetensors",
+    "model.layers.56.mlp.shared_experts.gate_proj.weight": "model-00058.safetensors",
+    "model.layers.56.mlp.shared_experts.gate_proj.scales": "model-00058.safetensors",
+    "model.layers.56.mlp.shared_experts.gate_proj.biases": "model-00058.safetensors",
+    "model.layers.56.mlp.shared_experts.up_proj.weight": "model-00058.safetensors",
+    "model.layers.56.mlp.shared_experts.up_proj.scales": "model-00058.safetensors",
+    "model.layers.56.mlp.shared_experts.up_proj.biases": "model-00058.safetensors",
+    "model.layers.56.post_attention_layernorm.weight": "model-00058.safetensors",
+    "model.layers.56.self_attn.kv_a_layernorm.weight": "model-00058.safetensors",
+    "model.layers.56.self_attn.kv_a_proj_with_mqa.weight": "model-00058.safetensors",
+    "model.layers.56.self_attn.kv_a_proj_with_mqa.scales": "model-00058.safetensors",
+    "model.layers.56.self_attn.kv_a_proj_with_mqa.biases": "model-00058.safetensors",
+    "model.layers.56.self_attn.o_proj.weight": "model-00058.safetensors",
+    "model.layers.56.self_attn.o_proj.scales": "model-00058.safetensors",
+    "model.layers.56.self_attn.o_proj.biases": "model-00058.safetensors",
+    "model.layers.56.self_attn.q_a_layernorm.weight": "model-00058.safetensors",
+    "model.layers.56.self_attn.q_a_proj.weight": "model-00058.safetensors",
+    "model.layers.56.self_attn.q_a_proj.scales": "model-00058.safetensors",
+    "model.layers.56.self_attn.q_a_proj.biases": "model-00058.safetensors",
+    "model.layers.56.self_attn.q_b_proj.weight": "model-00058.safetensors",
+    "model.layers.56.self_attn.q_b_proj.scales": "model-00058.safetensors",
+    "model.layers.56.self_attn.q_b_proj.biases": "model-00058.safetensors",
+    "model.layers.56.mlp.switch_mlp.gate_proj.weight": "model-00058.safetensors",
+    "model.layers.56.mlp.switch_mlp.gate_proj.scales": "model-00058.safetensors",
+    "model.layers.56.mlp.switch_mlp.gate_proj.biases": "model-00058.safetensors",
+    "model.layers.56.mlp.switch_mlp.down_proj.weight": "model-00058.safetensors",
+    "model.layers.56.mlp.switch_mlp.down_proj.scales": "model-00058.safetensors",
+    "model.layers.56.mlp.switch_mlp.down_proj.biases": "model-00058.safetensors",
+    "model.layers.56.mlp.switch_mlp.up_proj.weight": "model-00058.safetensors",
+    "model.layers.56.mlp.switch_mlp.up_proj.scales": "model-00058.safetensors",
+    "model.layers.56.mlp.switch_mlp.up_proj.biases": "model-00058.safetensors",
+    "model.layers.56.self_attn.embed_q.weight": "model-00058.safetensors",
+    "model.layers.56.self_attn.embed_q.scales": "model-00058.safetensors",
+    "model.layers.56.self_attn.embed_q.biases": "model-00058.safetensors",
+    "model.layers.56.self_attn.unembed_out.weight": "model-00058.safetensors",
+    "model.layers.56.self_attn.unembed_out.scales": "model-00058.safetensors",
+    "model.layers.56.self_attn.unembed_out.biases": "model-00058.safetensors",
+    "model.layers.57.input_layernorm.weight": "model-00059.safetensors",
+    "model.layers.57.mlp.gate.e_score_correction_bias": "model-00059.safetensors",
+    "model.layers.57.mlp.gate.weight": "model-00059.safetensors",
+    "model.layers.57.mlp.shared_experts.down_proj.weight": "model-00059.safetensors",
+    "model.layers.57.mlp.shared_experts.down_proj.scales": "model-00059.safetensors",
+    "model.layers.57.mlp.shared_experts.down_proj.biases": "model-00059.safetensors",
+    "model.layers.57.mlp.shared_experts.gate_proj.weight": "model-00059.safetensors",
+    "model.layers.57.mlp.shared_experts.gate_proj.scales": "model-00059.safetensors",
+    "model.layers.57.mlp.shared_experts.gate_proj.biases": "model-00059.safetensors",
+    "model.layers.57.mlp.shared_experts.up_proj.weight": "model-00059.safetensors",
+    "model.layers.57.mlp.shared_experts.up_proj.scales": "model-00059.safetensors",
+    "model.layers.57.mlp.shared_experts.up_proj.biases": "model-00059.safetensors",
+    "model.layers.57.post_attention_layernorm.weight": "model-00059.safetensors",
+    "model.layers.57.self_attn.kv_a_layernorm.weight": "model-00059.safetensors",
+    "model.layers.57.self_attn.kv_a_proj_with_mqa.weight": "model-00059.safetensors",
+    "model.layers.57.self_attn.kv_a_proj_with_mqa.scales": "model-00059.safetensors",
+    "model.layers.57.self_attn.kv_a_proj_with_mqa.biases": "model-00059.safetensors",
+    "model.layers.57.self_attn.o_proj.weight": "model-00059.safetensors",
+    "model.layers.57.self_attn.o_proj.scales": "model-00059.safetensors",
+    "model.layers.57.self_attn.o_proj.biases": "model-00059.safetensors",
+    "model.layers.57.self_attn.q_a_layernorm.weight": "model-00059.safetensors",
+    "model.layers.57.self_attn.q_a_proj.weight": "model-00059.safetensors",
+    "model.layers.57.self_attn.q_a_proj.scales": "model-00059.safetensors",
+    "model.layers.57.self_attn.q_a_proj.biases": "model-00059.safetensors",
+    "model.layers.57.self_attn.q_b_proj.weight": "model-00059.safetensors",
+    "model.layers.57.self_attn.q_b_proj.scales": "model-00059.safetensors",
+    "model.layers.57.self_attn.q_b_proj.biases": "model-00059.safetensors",
+    "model.layers.57.mlp.switch_mlp.gate_proj.weight": "model-00059.safetensors",
+    "model.layers.57.mlp.switch_mlp.gate_proj.scales": "model-00059.safetensors",
+    "model.layers.57.mlp.switch_mlp.gate_proj.biases": "model-00059.safetensors",
+    "model.layers.57.mlp.switch_mlp.down_proj.weight": "model-00059.safetensors",
+    "model.layers.57.mlp.switch_mlp.down_proj.scales": "model-00059.safetensors",
+    "model.layers.57.mlp.switch_mlp.down_proj.biases": "model-00059.safetensors",
+    "model.layers.57.mlp.switch_mlp.up_proj.weight": "model-00059.safetensors",
+    "model.layers.57.mlp.switch_mlp.up_proj.scales": "model-00059.safetensors",
+    "model.layers.57.mlp.switch_mlp.up_proj.biases": "model-00059.safetensors",
+    "model.layers.57.self_attn.embed_q.weight": "model-00059.safetensors",
+    "model.layers.57.self_attn.embed_q.scales": "model-00059.safetensors",
+    "model.layers.57.self_attn.embed_q.biases": "model-00059.safetensors",
+    "model.layers.57.self_attn.unembed_out.weight": "model-00059.safetensors",
+    "model.layers.57.self_attn.unembed_out.scales": "model-00059.safetensors",
+    "model.layers.57.self_attn.unembed_out.biases": "model-00059.safetensors",
+    "model.layers.58.input_layernorm.weight": "model-00060.safetensors",
+    "model.layers.58.mlp.gate.e_score_correction_bias": "model-00060.safetensors",
+    "model.layers.58.mlp.gate.weight": "model-00060.safetensors",
+    "model.layers.58.mlp.shared_experts.down_proj.weight": "model-00060.safetensors",
+    "model.layers.58.mlp.shared_experts.down_proj.scales": "model-00060.safetensors",
+    "model.layers.58.mlp.shared_experts.down_proj.biases": "model-00060.safetensors",
+    "model.layers.58.mlp.shared_experts.gate_proj.weight": "model-00060.safetensors",
+    "model.layers.58.mlp.shared_experts.gate_proj.scales": "model-00060.safetensors",
+    "model.layers.58.mlp.shared_experts.gate_proj.biases": "model-00060.safetensors",
+    "model.layers.58.mlp.shared_experts.up_proj.weight": "model-00060.safetensors",
+    "model.layers.58.mlp.shared_experts.up_proj.scales": "model-00060.safetensors",
+    "model.layers.58.mlp.shared_experts.up_proj.biases": "model-00060.safetensors",
+    "model.layers.58.post_attention_layernorm.weight": "model-00060.safetensors",
+    "model.layers.58.self_attn.indexer.k_norm.bias": "model-00060.safetensors",
+    "model.layers.58.self_attn.indexer.k_norm.weight": "model-00060.safetensors",
+    "model.layers.58.self_attn.indexer.weights_proj.weight": "model-00060.safetensors",
+    "model.layers.58.self_attn.indexer.wk.weight": "model-00060.safetensors",
+    "model.layers.58.self_attn.indexer.wq_b.weight": "model-00060.safetensors",
+    "model.layers.58.self_attn.kv_a_layernorm.weight": "model-00060.safetensors",
+    "model.layers.58.self_attn.kv_a_proj_with_mqa.weight": "model-00060.safetensors",
+    "model.layers.58.self_attn.kv_a_proj_with_mqa.scales": "model-00060.safetensors",
+    "model.layers.58.self_attn.kv_a_proj_with_mqa.biases": "model-00060.safetensors",
+    "model.layers.58.self_attn.o_proj.weight": "model-00060.safetensors",
+    "model.layers.58.self_attn.o_proj.scales": "model-00060.safetensors",
+    "model.layers.58.self_attn.o_proj.biases": "model-00060.safetensors",
+    "model.layers.58.self_attn.q_a_layernorm.weight": "model-00060.safetensors",
+    "model.layers.58.self_attn.q_a_proj.weight": "model-00060.safetensors",
+    "model.layers.58.self_attn.q_a_proj.scales": "model-00060.safetensors",
+    "model.layers.58.self_attn.q_a_proj.biases": "model-00060.safetensors",
+    "model.layers.58.self_attn.q_b_proj.weight": "model-00060.safetensors",
+    "model.layers.58.self_attn.q_b_proj.scales": "model-00060.safetensors",
+    "model.layers.58.self_attn.q_b_proj.biases": "model-00060.safetensors",
+    "model.layers.58.mlp.switch_mlp.gate_proj.weight": "model-00060.safetensors",
+    "model.layers.58.mlp.switch_mlp.gate_proj.scales": "model-00060.safetensors",
+    "model.layers.58.mlp.switch_mlp.gate_proj.biases": "model-00060.safetensors",
+    "model.layers.58.mlp.switch_mlp.down_proj.weight": "model-00060.safetensors",
+    "model.layers.58.mlp.switch_mlp.down_proj.scales": "model-00060.safetensors",
+    "model.layers.58.mlp.switch_mlp.down_proj.biases": "model-00060.safetensors",
+    "model.layers.58.mlp.switch_mlp.up_proj.weight": "model-00060.safetensors",
+    "model.layers.58.mlp.switch_mlp.up_proj.scales": "model-00060.safetensors",
+    "model.layers.58.mlp.switch_mlp.up_proj.biases": "model-00060.safetensors",
+    "model.layers.58.self_attn.embed_q.weight": "model-00060.safetensors",
+    "model.layers.58.self_attn.embed_q.scales": "model-00060.safetensors",
+    "model.layers.58.self_attn.embed_q.biases": "model-00060.safetensors",
+    "model.layers.58.self_attn.unembed_out.weight": "model-00060.safetensors",
+    "model.layers.58.self_attn.unembed_out.scales": "model-00060.safetensors",
+    "model.layers.58.self_attn.unembed_out.biases": "model-00060.safetensors",
+    "model.layers.59.input_layernorm.weight": "model-00061.safetensors",
+    "model.layers.59.mlp.gate.e_score_correction_bias": "model-00061.safetensors",
+    "model.layers.59.mlp.gate.weight": "model-00061.safetensors",
+    "model.layers.59.mlp.shared_experts.down_proj.weight": "model-00061.safetensors",
+    "model.layers.59.mlp.shared_experts.down_proj.scales": "model-00061.safetensors",
+    "model.layers.59.mlp.shared_experts.down_proj.biases": "model-00061.safetensors",
+    "model.layers.59.mlp.shared_experts.gate_proj.weight": "model-00061.safetensors",
+    "model.layers.59.mlp.shared_experts.gate_proj.scales": "model-00061.safetensors",
+    "model.layers.59.mlp.shared_experts.gate_proj.biases": "model-00061.safetensors",
+    "model.layers.59.mlp.shared_experts.up_proj.weight": "model-00061.safetensors",
+    "model.layers.59.mlp.shared_experts.up_proj.scales": "model-00061.safetensors",
+    "model.layers.59.mlp.shared_experts.up_proj.biases": "model-00061.safetensors",
+    "model.layers.59.post_attention_layernorm.weight": "model-00061.safetensors",
+    "model.layers.59.self_attn.kv_a_layernorm.weight": "model-00061.safetensors",
+    "model.layers.59.self_attn.kv_a_proj_with_mqa.weight": "model-00061.safetensors",
+    "model.layers.59.self_attn.kv_a_proj_with_mqa.scales": "model-00061.safetensors",
+    "model.layers.59.self_attn.kv_a_proj_with_mqa.biases": "model-00061.safetensors",
+    "model.layers.59.self_attn.o_proj.weight": "model-00061.safetensors",
+    "model.layers.59.self_attn.o_proj.scales": "model-00061.safetensors",
+    "model.layers.59.self_attn.o_proj.biases": "model-00061.safetensors",
+    "model.layers.59.self_attn.q_a_layernorm.weight": "model-00061.safetensors",
+    "model.layers.59.self_attn.q_a_proj.weight": "model-00061.safetensors",
+    "model.layers.59.self_attn.q_a_proj.scales": "model-00061.safetensors",
+    "model.layers.59.self_attn.q_a_proj.biases": "model-00061.safetensors",
+    "model.layers.59.self_attn.q_b_proj.weight": "model-00061.safetensors",
+    "model.layers.59.self_attn.q_b_proj.scales": "model-00061.safetensors",
+    "model.layers.59.self_attn.q_b_proj.biases": "model-00061.safetensors",
+    "model.layers.59.mlp.switch_mlp.gate_proj.weight": "model-00061.safetensors",
+    "model.layers.59.mlp.switch_mlp.gate_proj.scales": "model-00061.safetensors",
+    "model.layers.59.mlp.switch_mlp.gate_proj.biases": "model-00061.safetensors",
+    "model.layers.59.mlp.switch_mlp.down_proj.weight": "model-00061.safetensors",
+    "model.layers.59.mlp.switch_mlp.down_proj.scales": "model-00061.safetensors",
+    "model.layers.59.mlp.switch_mlp.down_proj.biases": "model-00061.safetensors",
+    "model.layers.59.mlp.switch_mlp.up_proj.weight": "model-00061.safetensors",
+    "model.layers.59.mlp.switch_mlp.up_proj.scales": "model-00061.safetensors",
+    "model.layers.59.mlp.switch_mlp.up_proj.biases": "model-00061.safetensors",
+    "model.layers.59.self_attn.embed_q.weight": "model-00061.safetensors",
+    "model.layers.59.self_attn.embed_q.scales": "model-00061.safetensors",
+    "model.layers.59.self_attn.embed_q.biases": "model-00061.safetensors",
+    "model.layers.59.self_attn.unembed_out.weight": "model-00061.safetensors",
+    "model.layers.59.self_attn.unembed_out.scales": "model-00061.safetensors",
+    "model.layers.59.self_attn.unembed_out.biases": "model-00061.safetensors",
+    "model.layers.60.input_layernorm.weight": "model-00062.safetensors",
+    "model.layers.60.mlp.gate.e_score_correction_bias": "model-00062.safetensors",
+    "model.layers.60.mlp.gate.weight": "model-00062.safetensors",
+    "model.layers.60.mlp.shared_experts.down_proj.weight": "model-00062.safetensors",
+    "model.layers.60.mlp.shared_experts.down_proj.scales": "model-00062.safetensors",
+    "model.layers.60.mlp.shared_experts.down_proj.biases": "model-00062.safetensors",
+    "model.layers.60.mlp.shared_experts.gate_proj.weight": "model-00062.safetensors",
+    "model.layers.60.mlp.shared_experts.gate_proj.scales": "model-00062.safetensors",
+    "model.layers.60.mlp.shared_experts.gate_proj.biases": "model-00062.safetensors",
+    "model.layers.60.mlp.shared_experts.up_proj.weight": "model-00062.safetensors",
+    "model.layers.60.mlp.shared_experts.up_proj.scales": "model-00062.safetensors",
+    "model.layers.60.mlp.shared_experts.up_proj.biases": "model-00062.safetensors",
+    "model.layers.60.post_attention_layernorm.weight": "model-00062.safetensors",
+    "model.layers.60.self_attn.kv_a_layernorm.weight": "model-00062.safetensors",
+    "model.layers.60.self_attn.kv_a_proj_with_mqa.weight": "model-00062.safetensors",
+    "model.layers.60.self_attn.kv_a_proj_with_mqa.scales": "model-00062.safetensors",
+    "model.layers.60.self_attn.kv_a_proj_with_mqa.biases": "model-00062.safetensors",
+    "model.layers.60.self_attn.o_proj.weight": "model-00062.safetensors",
+    "model.layers.60.self_attn.o_proj.scales": "model-00062.safetensors",
+    "model.layers.60.self_attn.o_proj.biases": "model-00062.safetensors",
+    "model.layers.60.self_attn.q_a_layernorm.weight": "model-00062.safetensors",
+    "model.layers.60.self_attn.q_a_proj.weight": "model-00062.safetensors",
+    "model.layers.60.self_attn.q_a_proj.scales": "model-00062.safetensors",
+    "model.layers.60.self_attn.q_a_proj.biases": "model-00062.safetensors",
+    "model.layers.60.self_attn.q_b_proj.weight": "model-00062.safetensors",
+    "model.layers.60.self_attn.q_b_proj.scales": "model-00062.safetensors",
+    "model.layers.60.self_attn.q_b_proj.biases": "model-00062.safetensors",
+    "model.layers.60.mlp.switch_mlp.gate_proj.weight": "model-00062.safetensors",
+    "model.layers.60.mlp.switch_mlp.gate_proj.scales": "model-00062.safetensors",
+    "model.layers.60.mlp.switch_mlp.gate_proj.biases": "model-00062.safetensors",
+    "model.layers.60.mlp.switch_mlp.down_proj.weight": "model-00062.safetensors",
+    "model.layers.60.mlp.switch_mlp.down_proj.scales": "model-00062.safetensors",
+    "model.layers.60.mlp.switch_mlp.down_proj.biases": "model-00062.safetensors",
+    "model.layers.60.mlp.switch_mlp.up_proj.weight": "model-00062.safetensors",
+    "model.layers.60.mlp.switch_mlp.up_proj.scales": "model-00062.safetensors",
+    "model.layers.60.mlp.switch_mlp.up_proj.biases": "model-00062.safetensors",
+    "model.layers.60.self_attn.embed_q.weight": "model-00062.safetensors",
+    "model.layers.60.self_attn.embed_q.scales": "model-00062.safetensors",
+    "model.layers.60.self_attn.embed_q.biases": "model-00062.safetensors",
+    "model.layers.60.self_attn.unembed_out.weight": "model-00062.safetensors",
+    "model.layers.60.self_attn.unembed_out.scales": "model-00062.safetensors",
+    "model.layers.60.self_attn.unembed_out.biases": "model-00062.safetensors",
+    "model.layers.61.input_layernorm.weight": "model-00063.safetensors",
+    "model.layers.61.mlp.gate.e_score_correction_bias": "model-00063.safetensors",
+    "model.layers.61.mlp.gate.weight": "model-00063.safetensors",
+    "model.layers.61.mlp.shared_experts.down_proj.weight": "model-00063.safetensors",
+    "model.layers.61.mlp.shared_experts.down_proj.scales": "model-00063.safetensors",
+    "model.layers.61.mlp.shared_experts.down_proj.biases": "model-00063.safetensors",
+    "model.layers.61.mlp.shared_experts.gate_proj.weight": "model-00063.safetensors",
+    "model.layers.61.mlp.shared_experts.gate_proj.scales": "model-00063.safetensors",
+    "model.layers.61.mlp.shared_experts.gate_proj.biases": "model-00063.safetensors",
+    "model.layers.61.mlp.shared_experts.up_proj.weight": "model-00063.safetensors",
+    "model.layers.61.mlp.shared_experts.up_proj.scales": "model-00063.safetensors",
+    "model.layers.61.mlp.shared_experts.up_proj.biases": "model-00063.safetensors",
+    "model.layers.61.post_attention_layernorm.weight": "model-00063.safetensors",
+    "model.layers.61.self_attn.kv_a_layernorm.weight": "model-00063.safetensors",
+    "model.layers.61.self_attn.kv_a_proj_with_mqa.weight": "model-00063.safetensors",
+    "model.layers.61.self_attn.kv_a_proj_with_mqa.scales": "model-00063.safetensors",
+    "model.layers.61.self_attn.kv_a_proj_with_mqa.biases": "model-00063.safetensors",
+    "model.layers.61.self_attn.o_proj.weight": "model-00063.safetensors",
+    "model.layers.61.self_attn.o_proj.scales": "model-00063.safetensors",
+    "model.layers.61.self_attn.o_proj.biases": "model-00063.safetensors",
+    "model.layers.61.self_attn.q_a_layernorm.weight": "model-00063.safetensors",
+    "model.layers.61.self_attn.q_a_proj.weight": "model-00063.safetensors",
+    "model.layers.61.self_attn.q_a_proj.scales": "model-00063.safetensors",
+    "model.layers.61.self_attn.q_a_proj.biases": "model-00063.safetensors",
+    "model.layers.61.self_attn.q_b_proj.weight": "model-00063.safetensors",
+    "model.layers.61.self_attn.q_b_proj.scales": "model-00063.safetensors",
+    "model.layers.61.self_attn.q_b_proj.biases": "model-00063.safetensors",
+    "model.layers.61.mlp.switch_mlp.gate_proj.weight": "model-00063.safetensors",
+    "model.layers.61.mlp.switch_mlp.gate_proj.scales": "model-00063.safetensors",
+    "model.layers.61.mlp.switch_mlp.gate_proj.biases": "model-00063.safetensors",
+    "model.layers.61.mlp.switch_mlp.down_proj.weight": "model-00063.safetensors",
+    "model.layers.61.mlp.switch_mlp.down_proj.scales": "model-00063.safetensors",
+    "model.layers.61.mlp.switch_mlp.down_proj.biases": "model-00063.safetensors",
+    "model.layers.61.mlp.switch_mlp.up_proj.weight": "model-00063.safetensors",
+    "model.layers.61.mlp.switch_mlp.up_proj.scales": "model-00063.safetensors",
+    "model.layers.61.mlp.switch_mlp.up_proj.biases": "model-00063.safetensors",
+    "model.layers.61.self_attn.embed_q.weight": "model-00063.safetensors",
+    "model.layers.61.self_attn.embed_q.scales": "model-00063.safetensors",
+    "model.layers.61.self_attn.embed_q.biases": "model-00063.safetensors",
+    "model.layers.61.self_attn.unembed_out.weight": "model-00063.safetensors",
+    "model.layers.61.self_attn.unembed_out.scales": "model-00063.safetensors",
+    "model.layers.61.self_attn.unembed_out.biases": "model-00063.safetensors",
+    "model.layers.62.input_layernorm.weight": "model-00064.safetensors",
+    "model.layers.62.mlp.gate.e_score_correction_bias": "model-00064.safetensors",
+    "model.layers.62.mlp.gate.weight": "model-00064.safetensors",
+    "model.layers.62.mlp.shared_experts.down_proj.weight": "model-00064.safetensors",
+    "model.layers.62.mlp.shared_experts.down_proj.scales": "model-00064.safetensors",
+    "model.layers.62.mlp.shared_experts.down_proj.biases": "model-00064.safetensors",
+    "model.layers.62.mlp.shared_experts.gate_proj.weight": "model-00064.safetensors",
+    "model.layers.62.mlp.shared_experts.gate_proj.scales": "model-00064.safetensors",
+    "model.layers.62.mlp.shared_experts.gate_proj.biases": "model-00064.safetensors",
+    "model.layers.62.mlp.shared_experts.up_proj.weight": "model-00064.safetensors",
+    "model.layers.62.mlp.shared_experts.up_proj.scales": "model-00064.safetensors",
+    "model.layers.62.mlp.shared_experts.up_proj.biases": "model-00064.safetensors",
+    "model.layers.62.post_attention_layernorm.weight": "model-00064.safetensors",
+    "model.layers.62.self_attn.indexer.k_norm.bias": "model-00064.safetensors",
+    "model.layers.62.self_attn.indexer.k_norm.weight": "model-00064.safetensors",
+    "model.layers.62.self_attn.indexer.weights_proj.weight": "model-00064.safetensors",
+    "model.layers.62.self_attn.indexer.wk.weight": "model-00064.safetensors",
+    "model.layers.62.self_attn.indexer.wq_b.weight": "model-00064.safetensors",
+    "model.layers.62.self_attn.kv_a_layernorm.weight": "model-00064.safetensors",
+    "model.layers.62.self_attn.kv_a_proj_with_mqa.weight": "model-00064.safetensors",
+    "model.layers.62.self_attn.kv_a_proj_with_mqa.scales": "model-00064.safetensors",
+    "model.layers.62.self_attn.kv_a_proj_with_mqa.biases": "model-00064.safetensors",
+    "model.layers.62.self_attn.o_proj.weight": "model-00064.safetensors",
+    "model.layers.62.self_attn.o_proj.scales": "model-00064.safetensors",
+    "model.layers.62.self_attn.o_proj.biases": "model-00064.safetensors",
+    "model.layers.62.self_attn.q_a_layernorm.weight": "model-00064.safetensors",
+    "model.layers.62.self_attn.q_a_proj.weight": "model-00064.safetensors",
+    "model.layers.62.self_attn.q_a_proj.scales": "model-00064.safetensors",
+    "model.layers.62.self_attn.q_a_proj.biases": "model-00064.safetensors",
+    "model.layers.62.self_attn.q_b_proj.weight": "model-00064.safetensors",
+    "model.layers.62.self_attn.q_b_proj.scales": "model-00064.safetensors",
+    "model.layers.62.self_attn.q_b_proj.biases": "model-00064.safetensors",
+    "model.layers.62.mlp.switch_mlp.gate_proj.weight": "model-00064.safetensors",
+    "model.layers.62.mlp.switch_mlp.gate_proj.scales": "model-00064.safetensors",
+    "model.layers.62.mlp.switch_mlp.gate_proj.biases": "model-00064.safetensors",
+    "model.layers.62.mlp.switch_mlp.down_proj.weight": "model-00064.safetensors",
+    "model.layers.62.mlp.switch_mlp.down_proj.scales": "model-00064.safetensors",
+    "model.layers.62.mlp.switch_mlp.down_proj.biases": "model-00064.safetensors",
+    "model.layers.62.mlp.switch_mlp.up_proj.weight": "model-00064.safetensors",
+    "model.layers.62.mlp.switch_mlp.up_proj.scales": "model-00064.safetensors",
+    "model.layers.62.mlp.switch_mlp.up_proj.biases": "model-00064.safetensors",
+    "model.layers.62.self_attn.embed_q.weight": "model-00064.safetensors",
+    "model.layers.62.self_attn.embed_q.scales": "model-00064.safetensors",
+    "model.layers.62.self_attn.embed_q.biases": "model-00064.safetensors",
+    "model.layers.62.self_attn.unembed_out.weight": "model-00064.safetensors",
+    "model.layers.62.self_attn.unembed_out.scales": "model-00064.safetensors",
+    "model.layers.62.self_attn.unembed_out.biases": "model-00064.safetensors",
+    "model.layers.63.input_layernorm.weight": "model-00065.safetensors",
+    "model.layers.63.mlp.gate.e_score_correction_bias": "model-00065.safetensors",
+    "model.layers.63.mlp.gate.weight": "model-00065.safetensors",
+    "model.layers.63.mlp.shared_experts.down_proj.weight": "model-00065.safetensors",
+    "model.layers.63.mlp.shared_experts.down_proj.scales": "model-00065.safetensors",
+    "model.layers.63.mlp.shared_experts.down_proj.biases": "model-00065.safetensors",
+    "model.layers.63.mlp.shared_experts.gate_proj.weight": "model-00065.safetensors",
+    "model.layers.63.mlp.shared_experts.gate_proj.scales": "model-00065.safetensors",
+    "model.layers.63.mlp.shared_experts.gate_proj.biases": "model-00065.safetensors",
+    "model.layers.63.mlp.shared_experts.up_proj.weight": "model-00065.safetensors",
+    "model.layers.63.mlp.shared_experts.up_proj.scales": "model-00065.safetensors",
+    "model.layers.63.mlp.shared_experts.up_proj.biases": "model-00065.safetensors",
+    "model.layers.63.post_attention_layernorm.weight": "model-00065.safetensors",
+    "model.layers.63.self_attn.kv_a_layernorm.weight": "model-00065.safetensors",
+    "model.layers.63.self_attn.kv_a_proj_with_mqa.weight": "model-00065.safetensors",
+    "model.layers.63.self_attn.kv_a_proj_with_mqa.scales": "model-00065.safetensors",
+    "model.layers.63.self_attn.kv_a_proj_with_mqa.biases": "model-00065.safetensors",
+    "model.layers.63.self_attn.o_proj.weight": "model-00065.safetensors",
+    "model.layers.63.self_attn.o_proj.scales": "model-00065.safetensors",
+    "model.layers.63.self_attn.o_proj.biases": "model-00065.safetensors",
+    "model.layers.63.self_attn.q_a_layernorm.weight": "model-00065.safetensors",
+    "model.layers.63.self_attn.q_a_proj.weight": "model-00065.safetensors",
+    "model.layers.63.self_attn.q_a_proj.scales": "model-00065.safetensors",
+    "model.layers.63.self_attn.q_a_proj.biases": "model-00065.safetensors",
+    "model.layers.63.self_attn.q_b_proj.weight": "model-00065.safetensors",
+    "model.layers.63.self_attn.q_b_proj.scales": "model-00065.safetensors",
+    "model.layers.63.self_attn.q_b_proj.biases": "model-00065.safetensors",
+    "model.layers.63.mlp.switch_mlp.gate_proj.weight": "model-00065.safetensors",
+    "model.layers.63.mlp.switch_mlp.gate_proj.scales": "model-00065.safetensors",
+    "model.layers.63.mlp.switch_mlp.gate_proj.biases": "model-00065.safetensors",
+    "model.layers.63.mlp.switch_mlp.down_proj.weight": "model-00065.safetensors",
+    "model.layers.63.mlp.switch_mlp.down_proj.scales": "model-00065.safetensors",
+    "model.layers.63.mlp.switch_mlp.down_proj.biases": "model-00065.safetensors",
+    "model.layers.63.mlp.switch_mlp.up_proj.weight": "model-00065.safetensors",
+    "model.layers.63.mlp.switch_mlp.up_proj.scales": "model-00065.safetensors",
+    "model.layers.63.mlp.switch_mlp.up_proj.biases": "model-00065.safetensors",
+    "model.layers.63.self_attn.embed_q.weight": "model-00065.safetensors",
+    "model.layers.63.self_attn.embed_q.scales": "model-00065.safetensors",
+    "model.layers.63.self_attn.embed_q.biases": "model-00065.safetensors",
+    "model.layers.63.self_attn.unembed_out.weight": "model-00065.safetensors",
+    "model.layers.63.self_attn.unembed_out.scales": "model-00065.safetensors",
+    "model.layers.63.self_attn.unembed_out.biases": "model-00065.safetensors",
+    "model.layers.64.input_layernorm.weight": "model-00066.safetensors",
+    "model.layers.64.mlp.gate.e_score_correction_bias": "model-00066.safetensors",
+    "model.layers.64.mlp.gate.weight": "model-00066.safetensors",
+    "model.layers.64.mlp.shared_experts.down_proj.weight": "model-00066.safetensors",
+    "model.layers.64.mlp.shared_experts.down_proj.scales": "model-00066.safetensors",
+    "model.layers.64.mlp.shared_experts.down_proj.biases": "model-00066.safetensors",
+    "model.layers.64.mlp.shared_experts.gate_proj.weight": "model-00066.safetensors",
+    "model.layers.64.mlp.shared_experts.gate_proj.scales": "model-00066.safetensors",
+    "model.layers.64.mlp.shared_experts.gate_proj.biases": "model-00066.safetensors",
+    "model.layers.64.mlp.shared_experts.up_proj.weight": "model-00066.safetensors",
+    "model.layers.64.mlp.shared_experts.up_proj.scales": "model-00066.safetensors",
+    "model.layers.64.mlp.shared_experts.up_proj.biases": "model-00066.safetensors",
+    "model.layers.64.post_attention_layernorm.weight": "model-00066.safetensors",
+    "model.layers.64.self_attn.kv_a_layernorm.weight": "model-00066.safetensors",
+    "model.layers.64.self_attn.kv_a_proj_with_mqa.weight": "model-00066.safetensors",
+    "model.layers.64.self_attn.kv_a_proj_with_mqa.scales": "model-00066.safetensors",
+    "model.layers.64.self_attn.kv_a_proj_with_mqa.biases": "model-00066.safetensors",
+    "model.layers.64.self_attn.o_proj.weight": "model-00066.safetensors",
+    "model.layers.64.self_attn.o_proj.scales": "model-00066.safetensors",
+    "model.layers.64.self_attn.o_proj.biases": "model-00066.safetensors",
+    "model.layers.64.self_attn.q_a_layernorm.weight": "model-00066.safetensors",
+    "model.layers.64.self_attn.q_a_proj.weight": "model-00066.safetensors",
+    "model.layers.64.self_attn.q_a_proj.scales": "model-00066.safetensors",
+    "model.layers.64.self_attn.q_a_proj.biases": "model-00066.safetensors",
+    "model.layers.64.self_attn.q_b_proj.weight": "model-00066.safetensors",
+    "model.layers.64.self_attn.q_b_proj.scales": "model-00066.safetensors",
+    "model.layers.64.self_attn.q_b_proj.biases": "model-00066.safetensors",
+    "model.layers.64.mlp.switch_mlp.gate_proj.weight": "model-00066.safetensors",
+    "model.layers.64.mlp.switch_mlp.gate_proj.scales": "model-00066.safetensors",
+    "model.layers.64.mlp.switch_mlp.gate_proj.biases": "model-00066.safetensors",
+    "model.layers.64.mlp.switch_mlp.down_proj.weight": "model-00066.safetensors",
+    "model.layers.64.mlp.switch_mlp.down_proj.scales": "model-00066.safetensors",
+    "model.layers.64.mlp.switch_mlp.down_proj.biases": "model-00066.safetensors",
+    "model.layers.64.mlp.switch_mlp.up_proj.weight": "model-00066.safetensors",
+    "model.layers.64.mlp.switch_mlp.up_proj.scales": "model-00066.safetensors",
+    "model.layers.64.mlp.switch_mlp.up_proj.biases": "model-00066.safetensors",
+    "model.layers.64.self_attn.embed_q.weight": "model-00066.safetensors",
+    "model.layers.64.self_attn.embed_q.scales": "model-00066.safetensors",
+    "model.layers.64.self_attn.embed_q.biases": "model-00066.safetensors",
+    "model.layers.64.self_attn.unembed_out.weight": "model-00066.safetensors",
+    "model.layers.64.self_attn.unembed_out.scales": "model-00066.safetensors",
+    "model.layers.64.self_attn.unembed_out.biases": "model-00066.safetensors",
+    "model.layers.65.input_layernorm.weight": "model-00067.safetensors",
+    "model.layers.65.mlp.gate.e_score_correction_bias": "model-00067.safetensors",
+    "model.layers.65.mlp.gate.weight": "model-00067.safetensors",
+    "model.layers.65.mlp.shared_experts.down_proj.weight": "model-00067.safetensors",
+    "model.layers.65.mlp.shared_experts.down_proj.scales": "model-00067.safetensors",
+    "model.layers.65.mlp.shared_experts.down_proj.biases": "model-00067.safetensors",
+    "model.layers.65.mlp.shared_experts.gate_proj.weight": "model-00067.safetensors",
+    "model.layers.65.mlp.shared_experts.gate_proj.scales": "model-00067.safetensors",
+    "model.layers.65.mlp.shared_experts.gate_proj.biases": "model-00067.safetensors",
+    "model.layers.65.mlp.shared_experts.up_proj.weight": "model-00067.safetensors",
+    "model.layers.65.mlp.shared_experts.up_proj.scales": "model-00067.safetensors",
+    "model.layers.65.mlp.shared_experts.up_proj.biases": "model-00067.safetensors",
+    "model.layers.65.post_attention_layernorm.weight": "model-00067.safetensors",
+    "model.layers.65.self_attn.kv_a_layernorm.weight": "model-00067.safetensors",
+    "model.layers.65.self_attn.kv_a_proj_with_mqa.weight": "model-00067.safetensors",
+    "model.layers.65.self_attn.kv_a_proj_with_mqa.scales": "model-00067.safetensors",
+    "model.layers.65.self_attn.kv_a_proj_with_mqa.biases": "model-00067.safetensors",
+    "model.layers.65.self_attn.o_proj.weight": "model-00067.safetensors",
+    "model.layers.65.self_attn.o_proj.scales": "model-00067.safetensors",
+    "model.layers.65.self_attn.o_proj.biases": "model-00067.safetensors",
+    "model.layers.65.self_attn.q_a_layernorm.weight": "model-00067.safetensors",
+    "model.layers.65.self_attn.q_a_proj.weight": "model-00067.safetensors",
+    "model.layers.65.self_attn.q_a_proj.scales": "model-00067.safetensors",
+    "model.layers.65.self_attn.q_a_proj.biases": "model-00067.safetensors",
+    "model.layers.65.self_attn.q_b_proj.weight": "model-00067.safetensors",
+    "model.layers.65.self_attn.q_b_proj.scales": "model-00067.safetensors",
+    "model.layers.65.self_attn.q_b_proj.biases": "model-00067.safetensors",
+    "model.layers.65.mlp.switch_mlp.gate_proj.weight": "model-00067.safetensors",
+    "model.layers.65.mlp.switch_mlp.gate_proj.scales": "model-00067.safetensors",
+    "model.layers.65.mlp.switch_mlp.gate_proj.biases": "model-00067.safetensors",
+    "model.layers.65.mlp.switch_mlp.down_proj.weight": "model-00067.safetensors",
+    "model.layers.65.mlp.switch_mlp.down_proj.scales": "model-00067.safetensors",
+    "model.layers.65.mlp.switch_mlp.down_proj.biases": "model-00067.safetensors",
+    "model.layers.65.mlp.switch_mlp.up_proj.weight": "model-00067.safetensors",
+    "model.layers.65.mlp.switch_mlp.up_proj.scales": "model-00067.safetensors",
+    "model.layers.65.mlp.switch_mlp.up_proj.biases": "model-00067.safetensors",
+    "model.layers.65.self_attn.embed_q.weight": "model-00067.safetensors",
+    "model.layers.65.self_attn.embed_q.scales": "model-00067.safetensors",
+    "model.layers.65.self_attn.embed_q.biases": "model-00067.safetensors",
+    "model.layers.65.self_attn.unembed_out.weight": "model-00067.safetensors",
+    "model.layers.65.self_attn.unembed_out.scales": "model-00067.safetensors",
+    "model.layers.65.self_attn.unembed_out.biases": "model-00067.safetensors",
+    "model.layers.66.input_layernorm.weight": "model-00068.safetensors",
+    "model.layers.66.mlp.gate.e_score_correction_bias": "model-00068.safetensors",
+    "model.layers.66.mlp.gate.weight": "model-00068.safetensors",
+    "model.layers.66.mlp.shared_experts.down_proj.weight": "model-00068.safetensors",
+    "model.layers.66.mlp.shared_experts.down_proj.scales": "model-00068.safetensors",
+    "model.layers.66.mlp.shared_experts.down_proj.biases": "model-00068.safetensors",
+    "model.layers.66.mlp.shared_experts.gate_proj.weight": "model-00068.safetensors",
+    "model.layers.66.mlp.shared_experts.gate_proj.scales": "model-00068.safetensors",
+    "model.layers.66.mlp.shared_experts.gate_proj.biases": "model-00068.safetensors",
+    "model.layers.66.mlp.shared_experts.up_proj.weight": "model-00068.safetensors",
+    "model.layers.66.mlp.shared_experts.up_proj.scales": "model-00068.safetensors",
+    "model.layers.66.mlp.shared_experts.up_proj.biases": "model-00068.safetensors",
+    "model.layers.66.post_attention_layernorm.weight": "model-00068.safetensors",
+    "model.layers.66.self_attn.indexer.k_norm.bias": "model-00068.safetensors",
+    "model.layers.66.self_attn.indexer.k_norm.weight": "model-00068.safetensors",
+    "model.layers.66.self_attn.indexer.weights_proj.weight": "model-00068.safetensors",
+    "model.layers.66.self_attn.indexer.wk.weight": "model-00068.safetensors",
+    "model.layers.66.self_attn.indexer.wq_b.weight": "model-00068.safetensors",
+    "model.layers.66.self_attn.kv_a_layernorm.weight": "model-00068.safetensors",
+    "model.layers.66.self_attn.kv_a_proj_with_mqa.weight": "model-00068.safetensors",
+    "model.layers.66.self_attn.kv_a_proj_with_mqa.scales": "model-00068.safetensors",
+    "model.layers.66.self_attn.kv_a_proj_with_mqa.biases": "model-00068.safetensors",
+    "model.layers.66.self_attn.o_proj.weight": "model-00068.safetensors",
+    "model.layers.66.self_attn.o_proj.scales": "model-00068.safetensors",
+    "model.layers.66.self_attn.o_proj.biases": "model-00068.safetensors",
+    "model.layers.66.self_attn.q_a_layernorm.weight": "model-00068.safetensors",
+    "model.layers.66.self_attn.q_a_proj.weight": "model-00068.safetensors",
+    "model.layers.66.self_attn.q_a_proj.scales": "model-00068.safetensors",
+    "model.layers.66.self_attn.q_a_proj.biases": "model-00068.safetensors",
+    "model.layers.66.self_attn.q_b_proj.weight": "model-00068.safetensors",
+    "model.layers.66.self_attn.q_b_proj.scales": "model-00068.safetensors",
+    "model.layers.66.self_attn.q_b_proj.biases": "model-00068.safetensors",
+    "model.layers.66.mlp.switch_mlp.gate_proj.weight": "model-00068.safetensors",
+    "model.layers.66.mlp.switch_mlp.gate_proj.scales": "model-00068.safetensors",
+    "model.layers.66.mlp.switch_mlp.gate_proj.biases": "model-00068.safetensors",
+    "model.layers.66.mlp.switch_mlp.down_proj.weight": "model-00068.safetensors",
+    "model.layers.66.mlp.switch_mlp.down_proj.scales": "model-00068.safetensors",
+    "model.layers.66.mlp.switch_mlp.down_proj.biases": "model-00068.safetensors",
+    "model.layers.66.mlp.switch_mlp.up_proj.weight": "model-00068.safetensors",
+    "model.layers.66.mlp.switch_mlp.up_proj.scales": "model-00068.safetensors",
+    "model.layers.66.mlp.switch_mlp.up_proj.biases": "model-00068.safetensors",
+    "model.layers.66.self_attn.embed_q.weight": "model-00068.safetensors",
+    "model.layers.66.self_attn.embed_q.scales": "model-00068.safetensors",
+    "model.layers.66.self_attn.embed_q.biases": "model-00068.safetensors",
+    "model.layers.66.self_attn.unembed_out.weight": "model-00068.safetensors",
+    "model.layers.66.self_attn.unembed_out.scales": "model-00068.safetensors",
+    "model.layers.66.self_attn.unembed_out.biases": "model-00068.safetensors",
+    "model.layers.67.input_layernorm.weight": "model-00069.safetensors",
+    "model.layers.67.mlp.gate.e_score_correction_bias": "model-00069.safetensors",
+    "model.layers.67.mlp.gate.weight": "model-00069.safetensors",
+    "model.layers.67.mlp.shared_experts.down_proj.weight": "model-00069.safetensors",
+    "model.layers.67.mlp.shared_experts.down_proj.scales": "model-00069.safetensors",
+    "model.layers.67.mlp.shared_experts.down_proj.biases": "model-00069.safetensors",
+    "model.layers.67.mlp.shared_experts.gate_proj.weight": "model-00069.safetensors",
+    "model.layers.67.mlp.shared_experts.gate_proj.scales": "model-00069.safetensors",
+    "model.layers.67.mlp.shared_experts.gate_proj.biases": "model-00069.safetensors",
+    "model.layers.67.mlp.shared_experts.up_proj.weight": "model-00069.safetensors",
+    "model.layers.67.mlp.shared_experts.up_proj.scales": "model-00069.safetensors",
+    "model.layers.67.mlp.shared_experts.up_proj.biases": "model-00069.safetensors",
+    "model.layers.67.post_attention_layernorm.weight": "model-00069.safetensors",
+    "model.layers.67.self_attn.kv_a_layernorm.weight": "model-00069.safetensors",
+    "model.layers.67.self_attn.kv_a_proj_with_mqa.weight": "model-00069.safetensors",
+    "model.layers.67.self_attn.kv_a_proj_with_mqa.scales": "model-00069.safetensors",
+    "model.layers.67.self_attn.kv_a_proj_with_mqa.biases": "model-00069.safetensors",
+    "model.layers.67.self_attn.o_proj.weight": "model-00069.safetensors",
+    "model.layers.67.self_attn.o_proj.scales": "model-00069.safetensors",
+    "model.layers.67.self_attn.o_proj.biases": "model-00069.safetensors",
+    "model.layers.67.self_attn.q_a_layernorm.weight": "model-00069.safetensors",
+    "model.layers.67.self_attn.q_a_proj.weight": "model-00069.safetensors",
+    "model.layers.67.self_attn.q_a_proj.scales": "model-00069.safetensors",
+    "model.layers.67.self_attn.q_a_proj.biases": "model-00069.safetensors",
+    "model.layers.67.self_attn.q_b_proj.weight": "model-00069.safetensors",
+    "model.layers.67.self_attn.q_b_proj.scales": "model-00069.safetensors",
+    "model.layers.67.self_attn.q_b_proj.biases": "model-00069.safetensors",
+    "model.layers.67.mlp.switch_mlp.gate_proj.weight": "model-00069.safetensors",
+    "model.layers.67.mlp.switch_mlp.gate_proj.scales": "model-00069.safetensors",
+    "model.layers.67.mlp.switch_mlp.gate_proj.biases": "model-00069.safetensors",
+    "model.layers.67.mlp.switch_mlp.down_proj.weight": "model-00069.safetensors",
+    "model.layers.67.mlp.switch_mlp.down_proj.scales": "model-00069.safetensors",
+    "model.layers.67.mlp.switch_mlp.down_proj.biases": "model-00069.safetensors",
+    "model.layers.67.mlp.switch_mlp.up_proj.weight": "model-00069.safetensors",
+    "model.layers.67.mlp.switch_mlp.up_proj.scales": "model-00069.safetensors",
+    "model.layers.67.mlp.switch_mlp.up_proj.biases": "model-00069.safetensors",
+    "model.layers.67.self_attn.embed_q.weight": "model-00069.safetensors",
+    "model.layers.67.self_attn.embed_q.scales": "model-00069.safetensors",
+    "model.layers.67.self_attn.embed_q.biases": "model-00069.safetensors",
+    "model.layers.67.self_attn.unembed_out.weight": "model-00069.safetensors",
+    "model.layers.67.self_attn.unembed_out.scales": "model-00069.safetensors",
+    "model.layers.67.self_attn.unembed_out.biases": "model-00069.safetensors",
+    "model.layers.68.input_layernorm.weight": "model-00070.safetensors",
+    "model.layers.68.mlp.gate.e_score_correction_bias": "model-00070.safetensors",
+    "model.layers.68.mlp.gate.weight": "model-00070.safetensors",
+    "model.layers.68.mlp.shared_experts.down_proj.weight": "model-00070.safetensors",
+    "model.layers.68.mlp.shared_experts.down_proj.scales": "model-00070.safetensors",
+    "model.layers.68.mlp.shared_experts.down_proj.biases": "model-00070.safetensors",
+    "model.layers.68.mlp.shared_experts.gate_proj.weight": "model-00070.safetensors",
+    "model.layers.68.mlp.shared_experts.gate_proj.scales": "model-00070.safetensors",
+    "model.layers.68.mlp.shared_experts.gate_proj.biases": "model-00070.safetensors",
+    "model.layers.68.mlp.shared_experts.up_proj.weight": "model-00070.safetensors",
+    "model.layers.68.mlp.shared_experts.up_proj.scales": "model-00070.safetensors",
+    "model.layers.68.mlp.shared_experts.up_proj.biases": "model-00070.safetensors",
+    "model.layers.68.post_attention_layernorm.weight": "model-00070.safetensors",
+    "model.layers.68.self_attn.kv_a_layernorm.weight": "model-00070.safetensors",
+    "model.layers.68.self_attn.kv_a_proj_with_mqa.weight": "model-00070.safetensors",
+    "model.layers.68.self_attn.kv_a_proj_with_mqa.scales": "model-00070.safetensors",
+    "model.layers.68.self_attn.kv_a_proj_with_mqa.biases": "model-00070.safetensors",
+    "model.layers.68.self_attn.o_proj.weight": "model-00070.safetensors",
+    "model.layers.68.self_attn.o_proj.scales": "model-00070.safetensors",
+    "model.layers.68.self_attn.o_proj.biases": "model-00070.safetensors",
+    "model.layers.68.self_attn.q_a_layernorm.weight": "model-00070.safetensors",
+    "model.layers.68.self_attn.q_a_proj.weight": "model-00070.safetensors",
+    "model.layers.68.self_attn.q_a_proj.scales": "model-00070.safetensors",
+    "model.layers.68.self_attn.q_a_proj.biases": "model-00070.safetensors",
+    "model.layers.68.self_attn.q_b_proj.weight": "model-00070.safetensors",
+    "model.layers.68.self_attn.q_b_proj.scales": "model-00070.safetensors",
+    "model.layers.68.self_attn.q_b_proj.biases": "model-00070.safetensors",
+    "model.layers.68.mlp.switch_mlp.gate_proj.weight": "model-00070.safetensors",
+    "model.layers.68.mlp.switch_mlp.gate_proj.scales": "model-00070.safetensors",
+    "model.layers.68.mlp.switch_mlp.gate_proj.biases": "model-00070.safetensors",
+    "model.layers.68.mlp.switch_mlp.down_proj.weight": "model-00070.safetensors",
+    "model.layers.68.mlp.switch_mlp.down_proj.scales": "model-00070.safetensors",
+    "model.layers.68.mlp.switch_mlp.down_proj.biases": "model-00070.safetensors",
+    "model.layers.68.mlp.switch_mlp.up_proj.weight": "model-00070.safetensors",
+    "model.layers.68.mlp.switch_mlp.up_proj.scales": "model-00070.safetensors",
+    "model.layers.68.mlp.switch_mlp.up_proj.biases": "model-00070.safetensors",
+    "model.layers.68.self_attn.embed_q.weight": "model-00070.safetensors",
+    "model.layers.68.self_attn.embed_q.scales": "model-00070.safetensors",
+    "model.layers.68.self_attn.embed_q.biases": "model-00070.safetensors",
+    "model.layers.68.self_attn.unembed_out.weight": "model-00070.safetensors",
+    "model.layers.68.self_attn.unembed_out.scales": "model-00070.safetensors",
+    "model.layers.68.self_attn.unembed_out.biases": "model-00070.safetensors",
+    "model.layers.69.input_layernorm.weight": "model-00071.safetensors",
+    "model.layers.69.mlp.gate.e_score_correction_bias": "model-00071.safetensors",
+    "model.layers.69.mlp.gate.weight": "model-00071.safetensors",
+    "model.layers.69.mlp.shared_experts.down_proj.weight": "model-00071.safetensors",
+    "model.layers.69.mlp.shared_experts.down_proj.scales": "model-00071.safetensors",
+    "model.layers.69.mlp.shared_experts.down_proj.biases": "model-00071.safetensors",
+    "model.layers.69.mlp.shared_experts.gate_proj.weight": "model-00071.safetensors",
+    "model.layers.69.mlp.shared_experts.gate_proj.scales": "model-00071.safetensors",
+    "model.layers.69.mlp.shared_experts.gate_proj.biases": "model-00071.safetensors",
+    "model.layers.69.mlp.shared_experts.up_proj.weight": "model-00071.safetensors",
+    "model.layers.69.mlp.shared_experts.up_proj.scales": "model-00071.safetensors",
+    "model.layers.69.mlp.shared_experts.up_proj.biases": "model-00071.safetensors",
+    "model.layers.69.post_attention_layernorm.weight": "model-00071.safetensors",
+    "model.layers.69.self_attn.kv_a_layernorm.weight": "model-00071.safetensors",
+    "model.layers.69.self_attn.kv_a_proj_with_mqa.weight": "model-00071.safetensors",
+    "model.layers.69.self_attn.kv_a_proj_with_mqa.scales": "model-00071.safetensors",
+    "model.layers.69.self_attn.kv_a_proj_with_mqa.biases": "model-00071.safetensors",
+    "model.layers.69.self_attn.o_proj.weight": "model-00071.safetensors",
+    "model.layers.69.self_attn.o_proj.scales": "model-00071.safetensors",
+    "model.layers.69.self_attn.o_proj.biases": "model-00071.safetensors",
+    "model.layers.69.self_attn.q_a_layernorm.weight": "model-00071.safetensors",
+    "model.layers.69.self_attn.q_a_proj.weight": "model-00071.safetensors",
+    "model.layers.69.self_attn.q_a_proj.scales": "model-00071.safetensors",
+    "model.layers.69.self_attn.q_a_proj.biases": "model-00071.safetensors",
+    "model.layers.69.self_attn.q_b_proj.weight": "model-00071.safetensors",
+    "model.layers.69.self_attn.q_b_proj.scales": "model-00071.safetensors",
+    "model.layers.69.self_attn.q_b_proj.biases": "model-00071.safetensors",
+    "model.layers.69.mlp.switch_mlp.gate_proj.weight": "model-00071.safetensors",
+    "model.layers.69.mlp.switch_mlp.gate_proj.scales": "model-00071.safetensors",
+    "model.layers.69.mlp.switch_mlp.gate_proj.biases": "model-00071.safetensors",
+    "model.layers.69.mlp.switch_mlp.down_proj.weight": "model-00071.safetensors",
+    "model.layers.69.mlp.switch_mlp.down_proj.scales": "model-00071.safetensors",
+    "model.layers.69.mlp.switch_mlp.down_proj.biases": "model-00071.safetensors",
+    "model.layers.69.mlp.switch_mlp.up_proj.weight": "model-00071.safetensors",
+    "model.layers.69.mlp.switch_mlp.up_proj.scales": "model-00071.safetensors",
+    "model.layers.69.mlp.switch_mlp.up_proj.biases": "model-00071.safetensors",
+    "model.layers.69.self_attn.embed_q.weight": "model-00071.safetensors",
+    "model.layers.69.self_attn.embed_q.scales": "model-00071.safetensors",
+    "model.layers.69.self_attn.embed_q.biases": "model-00071.safetensors",
+    "model.layers.69.self_attn.unembed_out.weight": "model-00071.safetensors",
+    "model.layers.69.self_attn.unembed_out.scales": "model-00071.safetensors",
+    "model.layers.69.self_attn.unembed_out.biases": "model-00071.safetensors",
+    "model.layers.70.input_layernorm.weight": "model-00072.safetensors",
+    "model.layers.70.mlp.gate.e_score_correction_bias": "model-00072.safetensors",
+    "model.layers.70.mlp.gate.weight": "model-00072.safetensors",
+    "model.layers.70.mlp.shared_experts.down_proj.weight": "model-00072.safetensors",
+    "model.layers.70.mlp.shared_experts.down_proj.scales": "model-00072.safetensors",
+    "model.layers.70.mlp.shared_experts.down_proj.biases": "model-00072.safetensors",
+    "model.layers.70.mlp.shared_experts.gate_proj.weight": "model-00072.safetensors",
+    "model.layers.70.mlp.shared_experts.gate_proj.scales": "model-00072.safetensors",
+    "model.layers.70.mlp.shared_experts.gate_proj.biases": "model-00072.safetensors",
+    "model.layers.70.mlp.shared_experts.up_proj.weight": "model-00072.safetensors",
+    "model.layers.70.mlp.shared_experts.up_proj.scales": "model-00072.safetensors",
+    "model.layers.70.mlp.shared_experts.up_proj.biases": "model-00072.safetensors",
+    "model.layers.70.post_attention_layernorm.weight": "model-00072.safetensors",
+    "model.layers.70.self_attn.indexer.k_norm.bias": "model-00072.safetensors",
+    "model.layers.70.self_attn.indexer.k_norm.weight": "model-00072.safetensors",
+    "model.layers.70.self_attn.indexer.weights_proj.weight": "model-00072.safetensors",
+    "model.layers.70.self_attn.indexer.wk.weight": "model-00072.safetensors",
+    "model.layers.70.self_attn.indexer.wq_b.weight": "model-00072.safetensors",
+    "model.layers.70.self_attn.kv_a_layernorm.weight": "model-00072.safetensors",
+    "model.layers.70.self_attn.kv_a_proj_with_mqa.weight": "model-00072.safetensors",
+    "model.layers.70.self_attn.kv_a_proj_with_mqa.scales": "model-00072.safetensors",
+    "model.layers.70.self_attn.kv_a_proj_with_mqa.biases": "model-00072.safetensors",
+    "model.layers.70.self_attn.o_proj.weight": "model-00072.safetensors",
+    "model.layers.70.self_attn.o_proj.scales": "model-00072.safetensors",
+    "model.layers.70.self_attn.o_proj.biases": "model-00072.safetensors",
+    "model.layers.70.self_attn.q_a_layernorm.weight": "model-00072.safetensors",
+    "model.layers.70.self_attn.q_a_proj.weight": "model-00072.safetensors",
+    "model.layers.70.self_attn.q_a_proj.scales": "model-00072.safetensors",
+    "model.layers.70.self_attn.q_a_proj.biases": "model-00072.safetensors",
+    "model.layers.70.self_attn.q_b_proj.weight": "model-00072.safetensors",
+    "model.layers.70.self_attn.q_b_proj.scales": "model-00072.safetensors",
+    "model.layers.70.self_attn.q_b_proj.biases": "model-00072.safetensors",
+    "model.layers.70.mlp.switch_mlp.gate_proj.weight": "model-00072.safetensors",
+    "model.layers.70.mlp.switch_mlp.gate_proj.scales": "model-00072.safetensors",
+    "model.layers.70.mlp.switch_mlp.gate_proj.biases": "model-00072.safetensors",
+    "model.layers.70.mlp.switch_mlp.down_proj.weight": "model-00072.safetensors",
+    "model.layers.70.mlp.switch_mlp.down_proj.scales": "model-00072.safetensors",
+    "model.layers.70.mlp.switch_mlp.down_proj.biases": "model-00072.safetensors",
+    "model.layers.70.mlp.switch_mlp.up_proj.weight": "model-00072.safetensors",
+    "model.layers.70.mlp.switch_mlp.up_proj.scales": "model-00072.safetensors",
+    "model.layers.70.mlp.switch_mlp.up_proj.biases": "model-00072.safetensors",
+    "model.layers.70.self_attn.embed_q.weight": "model-00072.safetensors",
+    "model.layers.70.self_attn.embed_q.scales": "model-00072.safetensors",
+    "model.layers.70.self_attn.embed_q.biases": "model-00072.safetensors",
+    "model.layers.70.self_attn.unembed_out.weight": "model-00072.safetensors",
+    "model.layers.70.self_attn.unembed_out.scales": "model-00072.safetensors",
+    "model.layers.70.self_attn.unembed_out.biases": "model-00072.safetensors",
+    "model.layers.71.input_layernorm.weight": "model-00073.safetensors",
+    "model.layers.71.mlp.gate.e_score_correction_bias": "model-00073.safetensors",
+    "model.layers.71.mlp.gate.weight": "model-00073.safetensors",
+    "model.layers.71.mlp.shared_experts.down_proj.weight": "model-00073.safetensors",
+    "model.layers.71.mlp.shared_experts.down_proj.scales": "model-00073.safetensors",
+    "model.layers.71.mlp.shared_experts.down_proj.biases": "model-00073.safetensors",
+    "model.layers.71.mlp.shared_experts.gate_proj.weight": "model-00073.safetensors",
+    "model.layers.71.mlp.shared_experts.gate_proj.scales": "model-00073.safetensors",
+    "model.layers.71.mlp.shared_experts.gate_proj.biases": "model-00073.safetensors",
+    "model.layers.71.mlp.shared_experts.up_proj.weight": "model-00073.safetensors",
+    "model.layers.71.mlp.shared_experts.up_proj.scales": "model-00073.safetensors",
+    "model.layers.71.mlp.shared_experts.up_proj.biases": "model-00073.safetensors",
+    "model.layers.71.post_attention_layernorm.weight": "model-00073.safetensors",
+    "model.layers.71.self_attn.kv_a_layernorm.weight": "model-00073.safetensors",
+    "model.layers.71.self_attn.kv_a_proj_with_mqa.weight": "model-00073.safetensors",
+    "model.layers.71.self_attn.kv_a_proj_with_mqa.scales": "model-00073.safetensors",
+    "model.layers.71.self_attn.kv_a_proj_with_mqa.biases": "model-00073.safetensors",
+    "model.layers.71.self_attn.o_proj.weight": "model-00073.safetensors",
+    "model.layers.71.self_attn.o_proj.scales": "model-00073.safetensors",
+    "model.layers.71.self_attn.o_proj.biases": "model-00073.safetensors",
+    "model.layers.71.self_attn.q_a_layernorm.weight": "model-00073.safetensors",
+    "model.layers.71.self_attn.q_a_proj.weight": "model-00073.safetensors",
+    "model.layers.71.self_attn.q_a_proj.scales": "model-00073.safetensors",
+    "model.layers.71.self_attn.q_a_proj.biases": "model-00073.safetensors",
+    "model.layers.71.self_attn.q_b_proj.weight": "model-00073.safetensors",
+    "model.layers.71.self_attn.q_b_proj.scales": "model-00073.safetensors",
+    "model.layers.71.self_attn.q_b_proj.biases": "model-00073.safetensors",
+    "model.layers.71.mlp.switch_mlp.gate_proj.weight": "model-00073.safetensors",
+    "model.layers.71.mlp.switch_mlp.gate_proj.scales": "model-00073.safetensors",
+    "model.layers.71.mlp.switch_mlp.gate_proj.biases": "model-00073.safetensors",
+    "model.layers.71.mlp.switch_mlp.down_proj.weight": "model-00073.safetensors",
+    "model.layers.71.mlp.switch_mlp.down_proj.scales": "model-00073.safetensors",
+    "model.layers.71.mlp.switch_mlp.down_proj.biases": "model-00073.safetensors",
+    "model.layers.71.mlp.switch_mlp.up_proj.weight": "model-00073.safetensors",
+    "model.layers.71.mlp.switch_mlp.up_proj.scales": "model-00073.safetensors",
+    "model.layers.71.mlp.switch_mlp.up_proj.biases": "model-00073.safetensors",
+    "model.layers.71.self_attn.embed_q.weight": "model-00073.safetensors",
+    "model.layers.71.self_attn.embed_q.scales": "model-00073.safetensors",
+    "model.layers.71.self_attn.embed_q.biases": "model-00073.safetensors",
+    "model.layers.71.self_attn.unembed_out.weight": "model-00073.safetensors",
+    "model.layers.71.self_attn.unembed_out.scales": "model-00073.safetensors",
+    "model.layers.71.self_attn.unembed_out.biases": "model-00073.safetensors",
+    "model.layers.72.input_layernorm.weight": "model-00074.safetensors",
+    "model.layers.72.mlp.gate.e_score_correction_bias": "model-00074.safetensors",
+    "model.layers.72.mlp.gate.weight": "model-00074.safetensors",
+    "model.layers.72.mlp.shared_experts.down_proj.weight": "model-00074.safetensors",
+    "model.layers.72.mlp.shared_experts.down_proj.scales": "model-00074.safetensors",
+    "model.layers.72.mlp.shared_experts.down_proj.biases": "model-00074.safetensors",
+    "model.layers.72.mlp.shared_experts.gate_proj.weight": "model-00074.safetensors",
+    "model.layers.72.mlp.shared_experts.gate_proj.scales": "model-00074.safetensors",
+    "model.layers.72.mlp.shared_experts.gate_proj.biases": "model-00074.safetensors",
+    "model.layers.72.mlp.shared_experts.up_proj.weight": "model-00074.safetensors",
+    "model.layers.72.mlp.shared_experts.up_proj.scales": "model-00074.safetensors",
+    "model.layers.72.mlp.shared_experts.up_proj.biases": "model-00074.safetensors",
+    "model.layers.72.post_attention_layernorm.weight": "model-00074.safetensors",
+    "model.layers.72.self_attn.kv_a_layernorm.weight": "model-00074.safetensors",
+    "model.layers.72.self_attn.kv_a_proj_with_mqa.weight": "model-00074.safetensors",
+    "model.layers.72.self_attn.kv_a_proj_with_mqa.scales": "model-00074.safetensors",
+    "model.layers.72.self_attn.kv_a_proj_with_mqa.biases": "model-00074.safetensors",
+    "model.layers.72.self_attn.o_proj.weight": "model-00074.safetensors",
+    "model.layers.72.self_attn.o_proj.scales": "model-00074.safetensors",
+    "model.layers.72.self_attn.o_proj.biases": "model-00074.safetensors",
+    "model.layers.72.self_attn.q_a_layernorm.weight": "model-00074.safetensors",
+    "model.layers.72.self_attn.q_a_proj.weight": "model-00074.safetensors",
+    "model.layers.72.self_attn.q_a_proj.scales": "model-00074.safetensors",
+    "model.layers.72.self_attn.q_a_proj.biases": "model-00074.safetensors",
+    "model.layers.72.self_attn.q_b_proj.weight": "model-00074.safetensors",
+    "model.layers.72.self_attn.q_b_proj.scales": "model-00074.safetensors",
+    "model.layers.72.self_attn.q_b_proj.biases": "model-00074.safetensors",
+    "model.layers.72.mlp.switch_mlp.gate_proj.weight": "model-00074.safetensors",
+    "model.layers.72.mlp.switch_mlp.gate_proj.scales": "model-00074.safetensors",
+    "model.layers.72.mlp.switch_mlp.gate_proj.biases": "model-00074.safetensors",
+    "model.layers.72.mlp.switch_mlp.down_proj.weight": "model-00074.safetensors",
+    "model.layers.72.mlp.switch_mlp.down_proj.scales": "model-00074.safetensors",
+    "model.layers.72.mlp.switch_mlp.down_proj.biases": "model-00074.safetensors",
+    "model.layers.72.mlp.switch_mlp.up_proj.weight": "model-00074.safetensors",
+    "model.layers.72.mlp.switch_mlp.up_proj.scales": "model-00074.safetensors",
+    "model.layers.72.mlp.switch_mlp.up_proj.biases": "model-00074.safetensors",
+    "model.layers.72.self_attn.embed_q.weight": "model-00074.safetensors",
+    "model.layers.72.self_attn.embed_q.scales": "model-00074.safetensors",
+    "model.layers.72.self_attn.embed_q.biases": "model-00074.safetensors",
+    "model.layers.72.self_attn.unembed_out.weight": "model-00074.safetensors",
+    "model.layers.72.self_attn.unembed_out.scales": "model-00074.safetensors",
+    "model.layers.72.self_attn.unembed_out.biases": "model-00074.safetensors",
+    "model.layers.73.input_layernorm.weight": "model-00075.safetensors",
+    "model.layers.73.mlp.gate.e_score_correction_bias": "model-00075.safetensors",
+    "model.layers.73.mlp.gate.weight": "model-00075.safetensors",
+    "model.layers.73.mlp.shared_experts.down_proj.weight": "model-00075.safetensors",
+    "model.layers.73.mlp.shared_experts.down_proj.scales": "model-00075.safetensors",
+    "model.layers.73.mlp.shared_experts.down_proj.biases": "model-00075.safetensors",
+    "model.layers.73.mlp.shared_experts.gate_proj.weight": "model-00075.safetensors",
+    "model.layers.73.mlp.shared_experts.gate_proj.scales": "model-00075.safetensors",
+    "model.layers.73.mlp.shared_experts.gate_proj.biases": "model-00075.safetensors",
+    "model.layers.73.mlp.shared_experts.up_proj.weight": "model-00075.safetensors",
+    "model.layers.73.mlp.shared_experts.up_proj.scales": "model-00075.safetensors",
+    "model.layers.73.mlp.shared_experts.up_proj.biases": "model-00075.safetensors",
+    "model.layers.73.post_attention_layernorm.weight": "model-00075.safetensors",
+    "model.layers.73.self_attn.kv_a_layernorm.weight": "model-00075.safetensors",
+    "model.layers.73.self_attn.kv_a_proj_with_mqa.weight": "model-00075.safetensors",
+    "model.layers.73.self_attn.kv_a_proj_with_mqa.scales": "model-00075.safetensors",
+    "model.layers.73.self_attn.kv_a_proj_with_mqa.biases": "model-00075.safetensors",
+    "model.layers.73.self_attn.o_proj.weight": "model-00075.safetensors",
+    "model.layers.73.self_attn.o_proj.scales": "model-00075.safetensors",
+    "model.layers.73.self_attn.o_proj.biases": "model-00075.safetensors",
+    "model.layers.73.self_attn.q_a_layernorm.weight": "model-00075.safetensors",
+    "model.layers.73.self_attn.q_a_proj.weight": "model-00075.safetensors",
+    "model.layers.73.self_attn.q_a_proj.scales": "model-00075.safetensors",
+    "model.layers.73.self_attn.q_a_proj.biases": "model-00075.safetensors",
+    "model.layers.73.self_attn.q_b_proj.weight": "model-00075.safetensors",
+    "model.layers.73.self_attn.q_b_proj.scales": "model-00075.safetensors",
+    "model.layers.73.self_attn.q_b_proj.biases": "model-00075.safetensors",
+    "model.layers.73.mlp.switch_mlp.gate_proj.weight": "model-00075.safetensors",
+    "model.layers.73.mlp.switch_mlp.gate_proj.scales": "model-00075.safetensors",
+    "model.layers.73.mlp.switch_mlp.gate_proj.biases": "model-00075.safetensors",
+    "model.layers.73.mlp.switch_mlp.down_proj.weight": "model-00075.safetensors",
+    "model.layers.73.mlp.switch_mlp.down_proj.scales": "model-00075.safetensors",
+    "model.layers.73.mlp.switch_mlp.down_proj.biases": "model-00075.safetensors",
+    "model.layers.73.mlp.switch_mlp.up_proj.weight": "model-00075.safetensors",
+    "model.layers.73.mlp.switch_mlp.up_proj.scales": "model-00075.safetensors",
+    "model.layers.73.mlp.switch_mlp.up_proj.biases": "model-00075.safetensors",
+    "model.layers.73.self_attn.embed_q.weight": "model-00075.safetensors",
+    "model.layers.73.self_attn.embed_q.scales": "model-00075.safetensors",
+    "model.layers.73.self_attn.embed_q.biases": "model-00075.safetensors",
+    "model.layers.73.self_attn.unembed_out.weight": "model-00075.safetensors",
+    "model.layers.73.self_attn.unembed_out.scales": "model-00075.safetensors",
+    "model.layers.73.self_attn.unembed_out.biases": "model-00075.safetensors",
+    "model.layers.74.input_layernorm.weight": "model-00076.safetensors",
+    "model.layers.74.mlp.gate.e_score_correction_bias": "model-00076.safetensors",
+    "model.layers.74.mlp.gate.weight": "model-00076.safetensors",
+    "model.layers.74.mlp.shared_experts.down_proj.weight": "model-00076.safetensors",
+    "model.layers.74.mlp.shared_experts.down_proj.scales": "model-00076.safetensors",
+    "model.layers.74.mlp.shared_experts.down_proj.biases": "model-00076.safetensors",
+    "model.layers.74.mlp.shared_experts.gate_proj.weight": "model-00076.safetensors",
+    "model.layers.74.mlp.shared_experts.gate_proj.scales": "model-00076.safetensors",
+    "model.layers.74.mlp.shared_experts.gate_proj.biases": "model-00076.safetensors",
+    "model.layers.74.mlp.shared_experts.up_proj.weight": "model-00076.safetensors",
+    "model.layers.74.mlp.shared_experts.up_proj.scales": "model-00076.safetensors",
+    "model.layers.74.mlp.shared_experts.up_proj.biases": "model-00076.safetensors",
+    "model.layers.74.post_attention_layernorm.weight": "model-00076.safetensors",
+    "model.layers.74.self_attn.indexer.k_norm.bias": "model-00076.safetensors",
+    "model.layers.74.self_attn.indexer.k_norm.weight": "model-00076.safetensors",
+    "model.layers.74.self_attn.indexer.weights_proj.weight": "model-00076.safetensors",
+    "model.layers.74.self_attn.indexer.wk.weight": "model-00076.safetensors",
+    "model.layers.74.self_attn.indexer.wq_b.weight": "model-00076.safetensors",
+    "model.layers.74.self_attn.kv_a_layernorm.weight": "model-00076.safetensors",
+    "model.layers.74.self_attn.kv_a_proj_with_mqa.weight": "model-00076.safetensors",
+    "model.layers.74.self_attn.kv_a_proj_with_mqa.scales": "model-00076.safetensors",
+    "model.layers.74.self_attn.kv_a_proj_with_mqa.biases": "model-00076.safetensors",
+    "model.layers.74.self_attn.o_proj.weight": "model-00076.safetensors",
+    "model.layers.74.self_attn.o_proj.scales": "model-00076.safetensors",
+    "model.layers.74.self_attn.o_proj.biases": "model-00076.safetensors",
+    "model.layers.74.self_attn.q_a_layernorm.weight": "model-00076.safetensors",
+    "model.layers.74.self_attn.q_a_proj.weight": "model-00076.safetensors",
+    "model.layers.74.self_attn.q_a_proj.scales": "model-00076.safetensors",
+    "model.layers.74.self_attn.q_a_proj.biases": "model-00076.safetensors",
+    "model.layers.74.self_attn.q_b_proj.weight": "model-00076.safetensors",
+    "model.layers.74.self_attn.q_b_proj.scales": "model-00076.safetensors",
+    "model.layers.74.self_attn.q_b_proj.biases": "model-00076.safetensors",
+    "model.layers.74.mlp.switch_mlp.gate_proj.weight": "model-00076.safetensors",
+    "model.layers.74.mlp.switch_mlp.gate_proj.scales": "model-00076.safetensors",
+    "model.layers.74.mlp.switch_mlp.gate_proj.biases": "model-00076.safetensors",
+    "model.layers.74.mlp.switch_mlp.down_proj.weight": "model-00076.safetensors",
+    "model.layers.74.mlp.switch_mlp.down_proj.scales": "model-00076.safetensors",
+    "model.layers.74.mlp.switch_mlp.down_proj.biases": "model-00076.safetensors",
+    "model.layers.74.mlp.switch_mlp.up_proj.weight": "model-00076.safetensors",
+    "model.layers.74.mlp.switch_mlp.up_proj.scales": "model-00076.safetensors",
+    "model.layers.74.mlp.switch_mlp.up_proj.biases": "model-00076.safetensors",
+    "model.layers.74.self_attn.embed_q.weight": "model-00076.safetensors",
+    "model.layers.74.self_attn.embed_q.scales": "model-00076.safetensors",
+    "model.layers.74.self_attn.embed_q.biases": "model-00076.safetensors",
+    "model.layers.74.self_attn.unembed_out.weight": "model-00076.safetensors",
+    "model.layers.74.self_attn.unembed_out.scales": "model-00076.safetensors",
+    "model.layers.74.self_attn.unembed_out.biases": "model-00076.safetensors",
+    "model.layers.75.input_layernorm.weight": "model-00077.safetensors",
+    "model.layers.75.mlp.gate.e_score_correction_bias": "model-00077.safetensors",
+    "model.layers.75.mlp.gate.weight": "model-00077.safetensors",
+    "model.layers.75.mlp.shared_experts.down_proj.weight": "model-00077.safetensors",
+    "model.layers.75.mlp.shared_experts.down_proj.scales": "model-00077.safetensors",
+    "model.layers.75.mlp.shared_experts.down_proj.biases": "model-00077.safetensors",
+    "model.layers.75.mlp.shared_experts.gate_proj.weight": "model-00077.safetensors",
+    "model.layers.75.mlp.shared_experts.gate_proj.scales": "model-00077.safetensors",
+    "model.layers.75.mlp.shared_experts.gate_proj.biases": "model-00077.safetensors",
+    "model.layers.75.mlp.shared_experts.up_proj.weight": "model-00077.safetensors",
+    "model.layers.75.mlp.shared_experts.up_proj.scales": "model-00077.safetensors",
+    "model.layers.75.mlp.shared_experts.up_proj.biases": "model-00077.safetensors",
+    "model.layers.75.post_attention_layernorm.weight": "model-00077.safetensors",
+    "model.layers.75.self_attn.kv_a_layernorm.weight": "model-00077.safetensors",
+    "model.layers.75.self_attn.kv_a_proj_with_mqa.weight": "model-00077.safetensors",
+    "model.layers.75.self_attn.kv_a_proj_with_mqa.scales": "model-00077.safetensors",
+    "model.layers.75.self_attn.kv_a_proj_with_mqa.biases": "model-00077.safetensors",
+    "model.layers.75.self_attn.o_proj.weight": "model-00077.safetensors",
+    "model.layers.75.self_attn.o_proj.scales": "model-00077.safetensors",
+    "model.layers.75.self_attn.o_proj.biases": "model-00077.safetensors",
+    "model.layers.75.self_attn.q_a_layernorm.weight": "model-00077.safetensors",
+    "model.layers.75.self_attn.q_a_proj.weight": "model-00077.safetensors",
+    "model.layers.75.self_attn.q_a_proj.scales": "model-00077.safetensors",
+    "model.layers.75.self_attn.q_a_proj.biases": "model-00077.safetensors",
+    "model.layers.75.self_attn.q_b_proj.weight": "model-00077.safetensors",
+    "model.layers.75.self_attn.q_b_proj.scales": "model-00077.safetensors",
+    "model.layers.75.self_attn.q_b_proj.biases": "model-00077.safetensors",
+    "model.layers.75.mlp.switch_mlp.gate_proj.weight": "model-00077.safetensors",
+    "model.layers.75.mlp.switch_mlp.gate_proj.scales": "model-00077.safetensors",
+    "model.layers.75.mlp.switch_mlp.gate_proj.biases": "model-00077.safetensors",
+    "model.layers.75.mlp.switch_mlp.down_proj.weight": "model-00077.safetensors",
+    "model.layers.75.mlp.switch_mlp.down_proj.scales": "model-00077.safetensors",
+    "model.layers.75.mlp.switch_mlp.down_proj.biases": "model-00077.safetensors",
+    "model.layers.75.mlp.switch_mlp.up_proj.weight": "model-00077.safetensors",
+    "model.layers.75.mlp.switch_mlp.up_proj.scales": "model-00077.safetensors",
+    "model.layers.75.mlp.switch_mlp.up_proj.biases": "model-00077.safetensors",
+    "model.layers.75.self_attn.embed_q.weight": "model-00077.safetensors",
+    "model.layers.75.self_attn.embed_q.scales": "model-00077.safetensors",
+    "model.layers.75.self_attn.embed_q.biases": "model-00077.safetensors",
+    "model.layers.75.self_attn.unembed_out.weight": "model-00077.safetensors",
+    "model.layers.75.self_attn.unembed_out.scales": "model-00077.safetensors",
+    "model.layers.75.self_attn.unembed_out.biases": "model-00077.safetensors",
+    "model.layers.76.input_layernorm.weight": "model-00078.safetensors",
+    "model.layers.76.mlp.gate.e_score_correction_bias": "model-00078.safetensors",
+    "model.layers.76.mlp.gate.weight": "model-00078.safetensors",
+    "model.layers.76.mlp.shared_experts.down_proj.weight": "model-00078.safetensors",
+    "model.layers.76.mlp.shared_experts.down_proj.scales": "model-00078.safetensors",
+    "model.layers.76.mlp.shared_experts.down_proj.biases": "model-00078.safetensors",
+    "model.layers.76.mlp.shared_experts.gate_proj.weight": "model-00078.safetensors",
+    "model.layers.76.mlp.shared_experts.gate_proj.scales": "model-00078.safetensors",
+    "model.layers.76.mlp.shared_experts.gate_proj.biases": "model-00078.safetensors",
+    "model.layers.76.mlp.shared_experts.up_proj.weight": "model-00078.safetensors",
+    "model.layers.76.mlp.shared_experts.up_proj.scales": "model-00078.safetensors",
+    "model.layers.76.mlp.shared_experts.up_proj.biases": "model-00078.safetensors",
+    "model.layers.76.post_attention_layernorm.weight": "model-00078.safetensors",
+    "model.layers.76.self_attn.kv_a_layernorm.weight": "model-00078.safetensors",
+    "model.layers.76.self_attn.kv_a_proj_with_mqa.weight": "model-00078.safetensors",
+    "model.layers.76.self_attn.kv_a_proj_with_mqa.scales": "model-00078.safetensors",
+    "model.layers.76.self_attn.kv_a_proj_with_mqa.biases": "model-00078.safetensors",
+    "model.layers.76.self_attn.o_proj.weight": "model-00078.safetensors",
+    "model.layers.76.self_attn.o_proj.scales": "model-00078.safetensors",
+    "model.layers.76.self_attn.o_proj.biases": "model-00078.safetensors",
+    "model.layers.76.self_attn.q_a_layernorm.weight": "model-00078.safetensors",
+    "model.layers.76.self_attn.q_a_proj.weight": "model-00078.safetensors",
+    "model.layers.76.self_attn.q_a_proj.scales": "model-00078.safetensors",
+    "model.layers.76.self_attn.q_a_proj.biases": "model-00078.safetensors",
+    "model.layers.76.self_attn.q_b_proj.weight": "model-00078.safetensors",
+    "model.layers.76.self_attn.q_b_proj.scales": "model-00078.safetensors",
+    "model.layers.76.self_attn.q_b_proj.biases": "model-00078.safetensors",
+    "model.layers.76.mlp.switch_mlp.gate_proj.weight": "model-00078.safetensors",
+    "model.layers.76.mlp.switch_mlp.gate_proj.scales": "model-00078.safetensors",
+    "model.layers.76.mlp.switch_mlp.gate_proj.biases": "model-00078.safetensors",
+    "model.layers.76.mlp.switch_mlp.down_proj.weight": "model-00078.safetensors",
+    "model.layers.76.mlp.switch_mlp.down_proj.scales": "model-00078.safetensors",
+    "model.layers.76.mlp.switch_mlp.down_proj.biases": "model-00078.safetensors",
+    "model.layers.76.mlp.switch_mlp.up_proj.weight": "model-00078.safetensors",
+    "model.layers.76.mlp.switch_mlp.up_proj.scales": "model-00078.safetensors",
+    "model.layers.76.mlp.switch_mlp.up_proj.biases": "model-00078.safetensors",
+    "model.layers.76.self_attn.embed_q.weight": "model-00078.safetensors",
+    "model.layers.76.self_attn.embed_q.scales": "model-00078.safetensors",
+    "model.layers.76.self_attn.embed_q.biases": "model-00078.safetensors",
+    "model.layers.76.self_attn.unembed_out.weight": "model-00078.safetensors",
+    "model.layers.76.self_attn.unembed_out.scales": "model-00078.safetensors",
+    "model.layers.76.self_attn.unembed_out.biases": "model-00078.safetensors",
+    "model.layers.77.input_layernorm.weight": "model-00079.safetensors",
+    "model.layers.77.mlp.gate.e_score_correction_bias": "model-00079.safetensors",
+    "model.layers.77.mlp.gate.weight": "model-00079.safetensors",
+    "model.layers.77.mlp.shared_experts.down_proj.weight": "model-00079.safetensors",
+    "model.layers.77.mlp.shared_experts.down_proj.scales": "model-00079.safetensors",
+    "model.layers.77.mlp.shared_experts.down_proj.biases": "model-00079.safetensors",
+    "model.layers.77.mlp.shared_experts.gate_proj.weight": "model-00079.safetensors",
+    "model.layers.77.mlp.shared_experts.gate_proj.scales": "model-00079.safetensors",
+    "model.layers.77.mlp.shared_experts.gate_proj.biases": "model-00079.safetensors",
+    "model.layers.77.mlp.shared_experts.up_proj.weight": "model-00079.safetensors",
+    "model.layers.77.mlp.shared_experts.up_proj.scales": "model-00079.safetensors",
+    "model.layers.77.mlp.shared_experts.up_proj.biases": "model-00079.safetensors",
+    "model.layers.77.post_attention_layernorm.weight": "model-00079.safetensors",
+    "model.layers.77.self_attn.kv_a_layernorm.weight": "model-00079.safetensors",
+    "model.layers.77.self_attn.kv_a_proj_with_mqa.weight": "model-00079.safetensors",
+    "model.layers.77.self_attn.kv_a_proj_with_mqa.scales": "model-00079.safetensors",
+    "model.layers.77.self_attn.kv_a_proj_with_mqa.biases": "model-00079.safetensors",
+    "model.layers.77.self_attn.o_proj.weight": "model-00079.safetensors",
+    "model.layers.77.self_attn.o_proj.scales": "model-00079.safetensors",
+    "model.layers.77.self_attn.o_proj.biases": "model-00079.safetensors",
+    "model.layers.77.self_attn.q_a_layernorm.weight": "model-00079.safetensors",
+    "model.layers.77.self_attn.q_a_proj.weight": "model-00079.safetensors",
+    "model.layers.77.self_attn.q_a_proj.scales": "model-00079.safetensors",
+    "model.layers.77.self_attn.q_a_proj.biases": "model-00079.safetensors",
+    "model.layers.77.self_attn.q_b_proj.weight": "model-00079.safetensors",
+    "model.layers.77.self_attn.q_b_proj.scales": "model-00079.safetensors",
+    "model.layers.77.self_attn.q_b_proj.biases": "model-00079.safetensors",
+    "model.layers.77.mlp.switch_mlp.gate_proj.weight": "model-00079.safetensors",
+    "model.layers.77.mlp.switch_mlp.gate_proj.scales": "model-00079.safetensors",
+    "model.layers.77.mlp.switch_mlp.gate_proj.biases": "model-00079.safetensors",
+    "model.layers.77.mlp.switch_mlp.down_proj.weight": "model-00079.safetensors",
+    "model.layers.77.mlp.switch_mlp.down_proj.scales": "model-00079.safetensors",
+    "model.layers.77.mlp.switch_mlp.down_proj.biases": "model-00079.safetensors",
+    "model.layers.77.mlp.switch_mlp.up_proj.weight": "model-00079.safetensors",
+    "model.layers.77.mlp.switch_mlp.up_proj.scales": "model-00079.safetensors",
+    "model.layers.77.mlp.switch_mlp.up_proj.biases": "model-00079.safetensors",
+    "model.layers.77.self_attn.embed_q.weight": "model-00079.safetensors",
+    "model.layers.77.self_attn.embed_q.scales": "model-00079.safetensors",
+    "model.layers.77.self_attn.embed_q.biases": "model-00079.safetensors",
+    "model.layers.77.self_attn.unembed_out.weight": "model-00079.safetensors",
+    "model.layers.77.self_attn.unembed_out.scales": "model-00079.safetensors",
+    "model.layers.77.self_attn.unembed_out.biases": "model-00079.safetensors"
+  }
+}
\ No newline at end of file
diff --git a/tokenizer.json b/tokenizer.json
new file mode 100644
index 0000000000000000000000000000000000000000..aba40197a4cdb5607f4ab7a05fb0a4ee8054fd6d
--- /dev/null
+++ b/tokenizer.json
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19e773648cb4e65de8660ea6365e10acca112d42a854923df93db4a6f333a82d
+size 20217442
diff --git a/tokenizer_config.json b/tokenizer_config.json
new file mode 100644
index 0000000000000000000000000000000000000000..e375fa0a4e16660ce0e2026e39374d35dad4c079
--- /dev/null
+++ b/tokenizer_config.json
@@ -0,0 +1,33 @@
+{
+  "backend": "tokenizers",
+  "clean_up_tokenization_spaces": false,
+  "do_lower_case": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": [
+    "<|endoftext|>",
+    "[MASK]",
+    "[gMASK]",
+    "[sMASK]",
+    "<sop>",
+    "<eop>",
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>",
+    "<|observation|>",
+    "<|begin_of_image|>",
+    "<|end_of_image|>",
+    "<|begin_of_video|>",
+    "<|end_of_video|>",
+    "<|begin_of_audio|>",
+    "<|end_of_audio|>",
+    "<|begin_of_transcription|>",
+    "<|end_of_transcription|>"
+  ],
+  "is_local": true,
+  "model_max_length": 1048576,
+  "model_specific_special_tokens": {},
+  "pad_token": "<|endoftext|>",
+  "padding_side": "left",
+  "remove_space": false,
+  "tokenizer_class": "TokenizersBackend"
+}