Instructions to use Intel/Qwen3-Next-80B-A3B-Thinking-int4-mixed-AutoRound with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Intel/Qwen3-Next-80B-A3B-Thinking-int4-mixed-AutoRound with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Intel/Qwen3-Next-80B-A3B-Thinking-int4-mixed-AutoRound")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Intel/Qwen3-Next-80B-A3B-Thinking-int4-mixed-AutoRound")
model = AutoModelForCausalLM.from_pretrained("Intel/Qwen3-Next-80B-A3B-Thinking-int4-mixed-AutoRound", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Intel/Qwen3-Next-80B-A3B-Thinking-int4-mixed-AutoRound with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Intel/Qwen3-Next-80B-A3B-Thinking-int4-mixed-AutoRound"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Intel/Qwen3-Next-80B-A3B-Thinking-int4-mixed-AutoRound",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Intel/Qwen3-Next-80B-A3B-Thinking-int4-mixed-AutoRound

SGLang

How to use Intel/Qwen3-Next-80B-A3B-Thinking-int4-mixed-AutoRound with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Intel/Qwen3-Next-80B-A3B-Thinking-int4-mixed-AutoRound" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Intel/Qwen3-Next-80B-A3B-Thinking-int4-mixed-AutoRound",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Intel/Qwen3-Next-80B-A3B-Thinking-int4-mixed-AutoRound" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Intel/Qwen3-Next-80B-A3B-Thinking-int4-mixed-AutoRound",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Intel/Qwen3-Next-80B-A3B-Thinking-int4-mixed-AutoRound with Docker Model Runner:
```
docker model run hf.co/Intel/Qwen3-Next-80B-A3B-Thinking-int4-mixed-AutoRound
```

INC4AI commited on Apr 16

Commit

95ba469

verified ·

1 Parent(s): 11612bd

Upload folder using huggingface_hub

Browse files

Files changed (17) hide show

config.json +588 -51
generation_config.json +1 -1
model-00001-of-00011.safetensors +3 -0
model-00002-of-00011.safetensors +3 -0
model-00003-of-00011.safetensors +3 -0
model-00004-of-00011.safetensors +3 -0
model-00005-of-00011.safetensors +3 -0
model-00006-of-00011.safetensors +3 -0
model-00007-of-00011.safetensors +3 -0
model-00008-of-00011.safetensors +3 -0
model-00009-of-00011.safetensors +3 -0
model-00010-of-00011.safetensors +3 -0
model-00011-of-00011.safetensors +3 -0
model.safetensors.index.json +2 -2
model_extra_tensors.safetensors +3 -0
quantization_config.json +627 -98
tokenizer.json +2 -2

config.json CHANGED Viewed

@@ -82,7 +82,7 @@
   "output_router_logits": false,
   "partial_rotary_factor": 0.25,
   "quantization_config": {
-    "autoround_version": "0.7.0",
     "bits": 4,
     "data_type": "int",
     "extra_config": {
@@ -96,10 +96,21 @@
         "bits": 8
       },
       "model.layers.0.mlp.gate": {
         "bits": 8
       },
       "model.layers.0.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.1.linear_attn.in_proj_ba": {
         "bits": 8
@@ -111,10 +122,21 @@
         "bits": 8
       },
       "model.layers.1.mlp.gate": {
         "bits": 8
       },
       "model.layers.1.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.10.linear_attn.in_proj_ba": {
         "bits": 8
@@ -126,16 +148,38 @@
         "bits": 8
       },
       "model.layers.10.mlp.gate": {
         "bits": 8
       },
       "model.layers.10.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.11.mlp.gate": {
         "bits": 8
       },
       "model.layers.11.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.11.self_attn.k_proj": {
         "bits": 8
@@ -159,10 +203,21 @@
         "bits": 8
       },
       "model.layers.12.mlp.gate": {
         "bits": 8
       },
       "model.layers.12.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.13.linear_attn.in_proj_ba": {
         "bits": 8
@@ -174,10 +229,21 @@
         "bits": 8
       },
       "model.layers.13.mlp.gate": {
         "bits": 8
       },
       "model.layers.13.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.14.linear_attn.in_proj_ba": {
         "bits": 8
@@ -189,16 +255,38 @@
         "bits": 8
       },
       "model.layers.14.mlp.gate": {
         "bits": 8
       },
       "model.layers.14.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.15.mlp.gate": {
         "bits": 8
       },
       "model.layers.15.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.15.self_attn.k_proj": {
         "bits": 8
@@ -222,10 +310,21 @@
         "bits": 8
       },
       "model.layers.16.mlp.gate": {
         "bits": 8
       },
       "model.layers.16.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.17.linear_attn.in_proj_ba": {
         "bits": 8
@@ -237,10 +336,21 @@
         "bits": 8
       },
       "model.layers.17.mlp.gate": {
         "bits": 8
       },
       "model.layers.17.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.18.linear_attn.in_proj_ba": {
         "bits": 8
@@ -252,16 +362,38 @@
         "bits": 8
       },
       "model.layers.18.mlp.gate": {
         "bits": 8
       },
       "model.layers.18.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.19.mlp.gate": {
         "bits": 8
       },
       "model.layers.19.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.19.self_attn.k_proj": {
         "bits": 8
@@ -285,10 +417,21 @@
         "bits": 8
       },
       "model.layers.2.mlp.gate": {
         "bits": 8
       },
       "model.layers.2.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.20.linear_attn.in_proj_ba": {
         "bits": 8
@@ -300,10 +443,21 @@
         "bits": 8
       },
       "model.layers.20.mlp.gate": {
         "bits": 8
       },
       "model.layers.20.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.21.linear_attn.in_proj_ba": {
         "bits": 8
@@ -315,10 +469,21 @@
         "bits": 8
       },
       "model.layers.21.mlp.gate": {
         "bits": 8
       },
       "model.layers.21.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.22.linear_attn.in_proj_ba": {
         "bits": 8
@@ -330,16 +495,38 @@
         "bits": 8
       },
       "model.layers.22.mlp.gate": {
         "bits": 8
       },
       "model.layers.22.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.23.mlp.gate": {
         "bits": 8
       },
       "model.layers.23.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.23.self_attn.k_proj": {
         "bits": 8
@@ -363,10 +550,21 @@
         "bits": 8
       },
       "model.layers.24.mlp.gate": {
         "bits": 8
       },
       "model.layers.24.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.25.linear_attn.in_proj_ba": {
         "bits": 8
@@ -378,10 +576,21 @@
         "bits": 8
       },
       "model.layers.25.mlp.gate": {
         "bits": 8
       },
       "model.layers.25.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.26.linear_attn.in_proj_ba": {
         "bits": 8
@@ -393,16 +602,38 @@
         "bits": 8
       },
       "model.layers.26.mlp.gate": {
         "bits": 8
       },
       "model.layers.26.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.27.mlp.gate": {
         "bits": 8
       },
       "model.layers.27.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.27.self_attn.k_proj": {
         "bits": 8
@@ -426,10 +657,21 @@
         "bits": 8
       },
       "model.layers.28.mlp.gate": {
         "bits": 8
       },
       "model.layers.28.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.29.linear_attn.in_proj_ba": {
         "bits": 8
@@ -441,16 +683,38 @@
         "bits": 8
       },
       "model.layers.29.mlp.gate": {
         "bits": 8
       },
       "model.layers.29.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.3.mlp.gate": {
         "bits": 8
       },
       "model.layers.3.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.3.self_attn.k_proj": {
         "bits": 8
@@ -474,16 +738,38 @@
         "bits": 8
       },
       "model.layers.30.mlp.gate": {
         "bits": 8
       },
       "model.layers.30.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.31.mlp.gate": {
         "bits": 8
       },
       "model.layers.31.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.31.self_attn.k_proj": {
         "bits": 8
@@ -507,10 +793,21 @@
         "bits": 8
       },
       "model.layers.32.mlp.gate": {
         "bits": 8
       },
       "model.layers.32.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.33.linear_attn.in_proj_ba": {
         "bits": 8
@@ -522,10 +819,21 @@
         "bits": 8
       },
       "model.layers.33.mlp.gate": {
         "bits": 8
       },
       "model.layers.33.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.34.linear_attn.in_proj_ba": {
         "bits": 8
@@ -537,16 +845,38 @@
         "bits": 8
       },
       "model.layers.34.mlp.gate": {
         "bits": 8
       },
       "model.layers.34.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.35.mlp.gate": {
         "bits": 8
       },
       "model.layers.35.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.35.self_attn.k_proj": {
         "bits": 8
@@ -570,10 +900,21 @@
         "bits": 8
       },
       "model.layers.36.mlp.gate": {
         "bits": 8
       },
       "model.layers.36.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.37.linear_attn.in_proj_ba": {
         "bits": 8
@@ -585,10 +926,21 @@
         "bits": 8
       },
       "model.layers.37.mlp.gate": {
         "bits": 8
       },
       "model.layers.37.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.38.linear_attn.in_proj_ba": {
         "bits": 8
@@ -600,16 +952,38 @@
         "bits": 8
       },
       "model.layers.38.mlp.gate": {
         "bits": 8
       },
       "model.layers.38.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.39.mlp.gate": {
         "bits": 8
       },
       "model.layers.39.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.39.self_attn.k_proj": {
         "bits": 8
@@ -633,10 +1007,21 @@
         "bits": 8
       },
       "model.layers.4.mlp.gate": {
         "bits": 8
       },
       "model.layers.4.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.40.linear_attn.in_proj_ba": {
         "bits": 8
@@ -648,10 +1033,21 @@
         "bits": 8
       },
       "model.layers.40.mlp.gate": {
         "bits": 8
       },
       "model.layers.40.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.41.linear_attn.in_proj_ba": {
         "bits": 8
@@ -663,10 +1059,21 @@
         "bits": 8
       },
       "model.layers.41.mlp.gate": {
         "bits": 8
       },
       "model.layers.41.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.42.linear_attn.in_proj_ba": {
         "bits": 8
@@ -678,16 +1085,38 @@
         "bits": 8
       },
       "model.layers.42.mlp.gate": {
         "bits": 8
       },
       "model.layers.42.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.43.mlp.gate": {
         "bits": 8
       },
       "model.layers.43.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.43.self_attn.k_proj": {
         "bits": 8
@@ -711,10 +1140,21 @@
         "bits": 8
       },
       "model.layers.44.mlp.gate": {
         "bits": 8
       },
       "model.layers.44.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.45.linear_attn.in_proj_ba": {
         "bits": 8
@@ -726,10 +1166,21 @@
         "bits": 8
       },
       "model.layers.45.mlp.gate": {
         "bits": 8
       },
       "model.layers.45.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.46.linear_attn.in_proj_ba": {
         "bits": 8
@@ -741,16 +1192,38 @@
         "bits": 8
       },
       "model.layers.46.mlp.gate": {
         "bits": 8
       },
       "model.layers.46.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.47.mlp.gate": {
         "bits": 8
       },
       "model.layers.47.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.47.self_attn.k_proj": {
         "bits": 8
@@ -774,10 +1247,21 @@
         "bits": 8
       },
       "model.layers.5.mlp.gate": {
         "bits": 8
       },
       "model.layers.5.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.6.linear_attn.in_proj_ba": {
         "bits": 8
@@ -789,16 +1273,38 @@
         "bits": 8
       },
       "model.layers.6.mlp.gate": {
         "bits": 8
       },
       "model.layers.6.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.7.mlp.gate": {
         "bits": 8
       },
       "model.layers.7.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.7.self_attn.k_proj": {
         "bits": 8
@@ -822,10 +1328,21 @@
         "bits": 8
       },
       "model.layers.8.mlp.gate": {
         "bits": 8
       },
       "model.layers.8.mlp.shared_expert_gate": {
-        "bits": 16
       },
       "model.layers.9.linear_attn.in_proj_ba": {
         "bits": 8
@@ -837,17 +1354,37 @@
         "bits": 8
       },
       "model.layers.9.mlp.gate": {
         "bits": 8
       },
       "model.layers.9.mlp.shared_expert_gate": {
-        "bits": 16
       }
     },
     "group_size": 128,
     "iters": 0,
     "packing_format": "auto_round:auto_gptq",
     "quant_method": "auto-round",
-    "sym": true
   },
   "rms_norm_eps": 1e-06,
   "rope_scaling": null,
@@ -855,7 +1392,7 @@
   "router_aux_loss_coef": 0.001,
   "shared_expert_intermediate_size": 512,
   "tie_word_embeddings": false,
-  "transformers_version": "4.57.0.dev0",
   "use_cache": true,
   "use_sliding_window": false,
   "vocab_size": 151936

   "output_router_logits": false,
   "partial_rotary_factor": 0.25,
   "quantization_config": {
+    "autoround_version": "0.12.0",
     "bits": 4,
     "data_type": "int",
     "extra_config": {
         "bits": 8
       },
       "model.layers.0.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.0.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.0.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.0.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.0.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.1.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.1.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.1.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.1.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.1.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.1.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.10.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.10.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.10.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.10.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.10.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.10.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.11.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.11.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.11.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.11.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.11.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.11.self_attn.k_proj": {
         "bits": 8
         "bits": 8
       },
       "model.layers.12.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.12.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.12.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.12.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.12.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.13.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.13.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.13.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.13.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.13.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.13.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.14.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.14.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.14.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.14.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.14.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.14.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.15.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.15.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.15.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.15.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.15.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.15.self_attn.k_proj": {
         "bits": 8
         "bits": 8
       },
       "model.layers.16.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.16.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.16.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.16.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.16.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.17.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.17.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.17.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.17.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.17.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.17.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.18.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.18.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.18.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.18.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.18.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.18.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.19.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.19.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.19.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.19.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.19.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.19.self_attn.k_proj": {
         "bits": 8
         "bits": 8
       },
       "model.layers.2.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.2.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.2.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.2.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.2.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.20.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.20.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.20.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.20.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.20.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.20.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.21.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.21.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.21.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.21.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.21.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.21.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.22.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.22.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.22.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.22.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.22.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.22.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.23.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.23.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.23.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.23.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.23.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.23.self_attn.k_proj": {
         "bits": 8
         "bits": 8
       },
       "model.layers.24.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.24.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.24.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.24.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.24.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.25.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.25.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.25.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.25.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.25.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.25.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.26.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.26.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.26.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.26.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.26.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.26.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.27.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.27.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.27.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.27.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.27.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.27.self_attn.k_proj": {
         "bits": 8
         "bits": 8
       },
       "model.layers.28.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.28.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.28.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.28.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.28.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.29.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.29.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.29.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.29.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.29.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.29.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.3.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.3.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.3.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.3.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.3.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.3.self_attn.k_proj": {
         "bits": 8
         "bits": 8
       },
       "model.layers.30.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.30.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.30.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.30.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.30.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.31.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.31.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.31.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.31.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.31.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.31.self_attn.k_proj": {
         "bits": 8
         "bits": 8
       },
       "model.layers.32.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.32.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.32.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.32.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.32.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.33.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.33.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.33.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.33.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.33.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.33.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.34.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.34.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.34.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.34.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.34.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.34.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.35.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.35.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.35.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.35.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.35.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.35.self_attn.k_proj": {
         "bits": 8
         "bits": 8
       },
       "model.layers.36.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.36.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.36.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.36.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.36.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.37.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.37.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.37.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.37.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.37.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.37.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.38.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.38.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.38.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.38.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.38.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.38.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.39.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.39.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.39.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.39.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.39.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.39.self_attn.k_proj": {
         "bits": 8
         "bits": 8
       },
       "model.layers.4.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.4.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.4.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.4.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.4.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.40.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.40.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.40.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.40.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.40.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.40.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.41.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.41.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.41.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.41.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.41.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.41.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.42.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.42.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.42.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.42.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.42.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.42.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.43.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.43.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.43.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.43.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.43.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.43.self_attn.k_proj": {
         "bits": 8
         "bits": 8
       },
       "model.layers.44.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.44.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.44.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.44.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.44.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.45.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.45.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.45.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.45.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.45.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.45.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.46.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.46.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.46.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.46.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.46.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.46.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.47.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.47.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.47.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.47.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.47.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.47.self_attn.k_proj": {
         "bits": 8
         "bits": 8
       },
       "model.layers.5.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.5.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.5.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.5.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.5.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.6.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.6.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.6.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.6.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.6.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.6.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.7.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.7.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.7.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.7.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.7.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.7.self_attn.k_proj": {
         "bits": 8
         "bits": 8
       },
       "model.layers.8.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.8.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.8.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.8.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.8.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       },
       "model.layers.9.linear_attn.in_proj_ba": {
         "bits": 8
         "bits": 8
       },
       "model.layers.9.mlp.gate": {
+        "bits": 16,
+        "data_type": "float"
+      },
+      "model.layers.9.mlp.shared_expert.down_proj": {
+        "bits": 8
+      },
+      "model.layers.9.mlp.shared_expert.gate_proj": {
+        "bits": 8
+      },
+      "model.layers.9.mlp.shared_expert.up_proj": {
         "bits": 8
       },
       "model.layers.9.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "mtp.layers.0.mlp.gate": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "mtp.layers.0.mlp.shared_expert_gate": {
+        "bits": 16,
+        "data_type": "fp"
       }
     },
     "group_size": 128,
     "iters": 0,
     "packing_format": "auto_round:auto_gptq",
     "quant_method": "auto-round",
+    "sym": true,
+    "transform_config": {}
   },
   "rms_norm_eps": 1e-06,
   "rope_scaling": null,
   "router_aux_loss_coef": 0.001,
   "shared_expert_intermediate_size": 512,
   "tie_word_embeddings": false,
+  "transformers_version": "4.57.6",
   "use_cache": true,
   "use_sliding_window": false,
   "vocab_size": 151936

generation_config.json CHANGED Viewed

@@ -9,5 +9,5 @@
   "temperature": 0.6,
   "top_k": 20,
   "top_p": 0.95,
-  "transformers_version": "4.57.0.dev0"
 }

   "temperature": 0.6,
   "top_k": 20,
   "top_p": 0.95,
+  "transformers_version": "4.57.6"
 }

model-00001-of-00011.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:73f3cd317675f5afc1fce57d59340a06e544e1625a23f58f9a64b7e273c1f782
+size 4297702384

model-00002-of-00011.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3e5fba281bdb2366704a405e3ff7bf01a479bc61220b2b935477519bf6595478
+size 4297652104

model-00003-of-00011.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3e556aee2b08059422d3ac68d66e1d047200bf7a1b7514fa347ae6a8a8b03fe3
+size 4297673880

model-00004-of-00011.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:784e3556b63156b1025e5510ca3b38edf874bf16ed7d1085eb27522c7c189263
+size 4297642968

model-00005-of-00011.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:272eb52f2e602e30046a72647d2603dfabbc7799a288a68851b521bcc9d173f1
+size 4297674352

model-00006-of-00011.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:859f5517ac7288614523bcd5b9518a24a94105319b9b1fc9a6691c4bd09e907e
+size 4297674344

model-00007-of-00011.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b0f8d1cf24f4735f9ff1a23ad3ea4ea64032388d3868603205493d7300c67665
+size 4297674344

model-00008-of-00011.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0d78b633c971d61cf438d82b9f06f9dad952e5d459a689de3261751bde6b404e
+size 4297642968

model-00009-of-00011.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d5132fda1d2a06e6f11ef96d4eb89ae3a5991140089d3b9eed03b00d22baec89
+size 4297674344

model-00010-of-00011.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5ba91d9c361a5edfc4df227250eb5e13208c46ee9b0d6819e38cd2fd8d5886ce
+size 3969181936

model-00011-of-00011.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bbf87839cbb12e36eebf8dc2dd41362d4b92510c537ce6e5d7dd70c130e52ccd
+size 622329952

model.safetensors.index.json CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:725d03d8f1570858c9c13b15ec3424f06086f6bf1a49a50d341ee89d80a18d56
-size 20300105

 version https://git-lfs.github.com/spec/v1
+oid sha256:b215a5b30715120e9bc208f338ed7d1bb7fd2b2f14bf36782f7f2853a03d1c6d
+size 20698521

model_extra_tensors.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8e2cc472f2a9e08a8034149266e7c2cf756a2e47fd6cc04a1c2f86e11f1dc952
+size 859586872

quantization_config.json CHANGED Viewed

@@ -1,13 +1,206 @@
 {
   "bits": 4,
   "group_size": 128,
   "sym": true,
-  "data_type": "int",
   "iters": 0,
-  "autoround_version": "0.7.0",
   "quant_method": "auto-round",
   "packing_format": "auto_round:auto_gptq",
   "extra_config": {
     "model.layers.0.linear_attn.in_proj_qkvz": {
       "bits": 8
     },
@@ -18,10 +211,17 @@
       "bits": 8
     },
     "model.layers.0.mlp.gate": {
       "bits": 8
     },
-    "model.layers.0.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.1.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -33,10 +233,17 @@
       "bits": 8
     },
     "model.layers.1.mlp.gate": {
       "bits": 8
     },
-    "model.layers.1.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.2.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -48,10 +255,17 @@
       "bits": 8
     },
     "model.layers.2.mlp.gate": {
       "bits": 8
     },
-    "model.layers.2.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.3.self_attn.q_proj": {
       "bits": 8
@@ -66,10 +280,17 @@
       "bits": 8
     },
     "model.layers.3.mlp.gate": {
       "bits": 8
     },
-    "model.layers.3.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.4.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -81,10 +302,17 @@
       "bits": 8
     },
     "model.layers.4.mlp.gate": {
       "bits": 8
     },
-    "model.layers.4.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.5.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -96,10 +324,17 @@
       "bits": 8
     },
     "model.layers.5.mlp.gate": {
       "bits": 8
     },
-    "model.layers.5.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.6.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -111,10 +346,17 @@
       "bits": 8
     },
     "model.layers.6.mlp.gate": {
       "bits": 8
     },
-    "model.layers.6.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.7.self_attn.q_proj": {
       "bits": 8
@@ -129,10 +371,17 @@
       "bits": 8
     },
     "model.layers.7.mlp.gate": {
       "bits": 8
     },
-    "model.layers.7.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.8.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -144,10 +393,17 @@
       "bits": 8
     },
     "model.layers.8.mlp.gate": {
       "bits": 8
     },
-    "model.layers.8.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.9.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -159,10 +415,17 @@
       "bits": 8
     },
     "model.layers.9.mlp.gate": {
       "bits": 8
     },
-    "model.layers.9.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.10.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -174,10 +437,17 @@
       "bits": 8
     },
     "model.layers.10.mlp.gate": {
       "bits": 8
     },
-    "model.layers.10.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.11.self_attn.q_proj": {
       "bits": 8
@@ -192,10 +462,17 @@
       "bits": 8
     },
     "model.layers.11.mlp.gate": {
       "bits": 8
     },
-    "model.layers.11.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.12.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -207,10 +484,17 @@
       "bits": 8
     },
     "model.layers.12.mlp.gate": {
       "bits": 8
     },
-    "model.layers.12.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.13.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -222,10 +506,17 @@
       "bits": 8
     },
     "model.layers.13.mlp.gate": {
       "bits": 8
     },
-    "model.layers.13.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.14.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -237,10 +528,17 @@
       "bits": 8
     },
     "model.layers.14.mlp.gate": {
       "bits": 8
     },
-    "model.layers.14.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.15.self_attn.q_proj": {
       "bits": 8
@@ -255,10 +553,17 @@
       "bits": 8
     },
     "model.layers.15.mlp.gate": {
       "bits": 8
     },
-    "model.layers.15.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.16.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -270,10 +575,17 @@
       "bits": 8
     },
     "model.layers.16.mlp.gate": {
       "bits": 8
     },
-    "model.layers.16.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.17.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -285,10 +597,17 @@
       "bits": 8
     },
     "model.layers.17.mlp.gate": {
       "bits": 8
     },
-    "model.layers.17.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.18.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -300,10 +619,17 @@
       "bits": 8
     },
     "model.layers.18.mlp.gate": {
       "bits": 8
     },
-    "model.layers.18.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.19.self_attn.q_proj": {
       "bits": 8
@@ -318,10 +644,17 @@
       "bits": 8
     },
     "model.layers.19.mlp.gate": {
       "bits": 8
     },
-    "model.layers.19.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.20.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -333,10 +666,17 @@
       "bits": 8
     },
     "model.layers.20.mlp.gate": {
       "bits": 8
     },
-    "model.layers.20.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.21.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -348,10 +688,17 @@
       "bits": 8
     },
     "model.layers.21.mlp.gate": {
       "bits": 8
     },
-    "model.layers.21.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.22.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -363,10 +710,17 @@
       "bits": 8
     },
     "model.layers.22.mlp.gate": {
       "bits": 8
     },
-    "model.layers.22.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.23.self_attn.q_proj": {
       "bits": 8
@@ -381,10 +735,17 @@
       "bits": 8
     },
     "model.layers.23.mlp.gate": {
       "bits": 8
     },
-    "model.layers.23.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.24.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -396,10 +757,17 @@
       "bits": 8
     },
     "model.layers.24.mlp.gate": {
       "bits": 8
     },
-    "model.layers.24.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.25.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -411,10 +779,17 @@
       "bits": 8
     },
     "model.layers.25.mlp.gate": {
       "bits": 8
     },
-    "model.layers.25.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.26.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -426,10 +801,17 @@
       "bits": 8
     },
     "model.layers.26.mlp.gate": {
       "bits": 8
     },
-    "model.layers.26.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.27.self_attn.q_proj": {
       "bits": 8
@@ -444,10 +826,17 @@
       "bits": 8
     },
     "model.layers.27.mlp.gate": {
       "bits": 8
     },
-    "model.layers.27.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.28.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -459,10 +848,17 @@
       "bits": 8
     },
     "model.layers.28.mlp.gate": {
       "bits": 8
     },
-    "model.layers.28.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.29.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -474,10 +870,17 @@
       "bits": 8
     },
     "model.layers.29.mlp.gate": {
       "bits": 8
     },
-    "model.layers.29.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.30.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -489,10 +892,17 @@
       "bits": 8
     },
     "model.layers.30.mlp.gate": {
       "bits": 8
     },
-    "model.layers.30.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.31.self_attn.q_proj": {
       "bits": 8
@@ -507,10 +917,17 @@
       "bits": 8
     },
     "model.layers.31.mlp.gate": {
       "bits": 8
     },
-    "model.layers.31.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.32.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -522,10 +939,17 @@
       "bits": 8
     },
     "model.layers.32.mlp.gate": {
       "bits": 8
     },
-    "model.layers.32.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.33.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -537,10 +961,17 @@
       "bits": 8
     },
     "model.layers.33.mlp.gate": {
       "bits": 8
     },
-    "model.layers.33.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.34.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -552,10 +983,17 @@
       "bits": 8
     },
     "model.layers.34.mlp.gate": {
       "bits": 8
     },
-    "model.layers.34.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.35.self_attn.q_proj": {
       "bits": 8
@@ -570,10 +1008,17 @@
       "bits": 8
     },
     "model.layers.35.mlp.gate": {
       "bits": 8
     },
-    "model.layers.35.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.36.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -585,10 +1030,17 @@
       "bits": 8
     },
     "model.layers.36.mlp.gate": {
       "bits": 8
     },
-    "model.layers.36.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.37.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -600,10 +1052,17 @@
       "bits": 8
     },
     "model.layers.37.mlp.gate": {
       "bits": 8
     },
-    "model.layers.37.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.38.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -615,10 +1074,17 @@
       "bits": 8
     },
     "model.layers.38.mlp.gate": {
       "bits": 8
     },
-    "model.layers.38.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.39.self_attn.q_proj": {
       "bits": 8
@@ -633,10 +1099,17 @@
       "bits": 8
     },
     "model.layers.39.mlp.gate": {
       "bits": 8
     },
-    "model.layers.39.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.40.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -648,10 +1121,17 @@
       "bits": 8
     },
     "model.layers.40.mlp.gate": {
       "bits": 8
     },
-    "model.layers.40.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.41.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -663,10 +1143,17 @@
       "bits": 8
     },
     "model.layers.41.mlp.gate": {
       "bits": 8
     },
-    "model.layers.41.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.42.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -678,10 +1165,17 @@
       "bits": 8
     },
     "model.layers.42.mlp.gate": {
       "bits": 8
     },
-    "model.layers.42.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.43.self_attn.q_proj": {
       "bits": 8
@@ -696,10 +1190,17 @@
       "bits": 8
     },
     "model.layers.43.mlp.gate": {
       "bits": 8
     },
-    "model.layers.43.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.44.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -711,10 +1212,17 @@
       "bits": 8
     },
     "model.layers.44.mlp.gate": {
       "bits": 8
     },
-    "model.layers.44.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.45.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -726,10 +1234,17 @@
       "bits": 8
     },
     "model.layers.45.mlp.gate": {
       "bits": 8
     },
-    "model.layers.45.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.46.linear_attn.in_proj_qkvz": {
       "bits": 8
@@ -741,10 +1256,17 @@
       "bits": 8
     },
     "model.layers.46.mlp.gate": {
       "bits": 8
     },
-    "model.layers.46.mlp.shared_expert_gate": {
-      "bits": 16
     },
     "model.layers.47.self_attn.q_proj": {
       "bits": 8
@@ -759,10 +1281,17 @@
       "bits": 8
     },
     "model.layers.47.mlp.gate": {
       "bits": 8
     },
-    "model.layers.47.mlp.shared_expert_gate": {
-      "bits": 16
     }
   }
 }

 {
   "bits": 4,
+  "data_type": "int",
   "group_size": 128,
   "sym": true,
   "iters": 0,
+  "transform_config": {},
+  "autoround_version": "0.12.0",
   "quant_method": "auto-round",
   "packing_format": "auto_round:auto_gptq",
   "extra_config": {
+    "model.layers.0.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.1.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.2.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.3.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.4.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.5.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.6.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.7.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.8.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.9.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.10.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.11.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.12.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.13.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.14.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.15.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.16.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.17.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.18.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.19.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.20.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.21.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.22.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.23.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.24.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.25.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.26.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.27.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.28.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.29.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.30.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.31.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.32.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.33.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.34.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.35.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.36.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.37.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.38.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.39.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.40.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.41.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.42.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.43.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.44.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.45.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.46.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.layers.47.mlp.shared_expert_gate": {
+      "bits": 16,
+      "data_type": "fp"
+    },
     "model.layers.0.linear_attn.in_proj_qkvz": {
       "bits": 8
     },
       "bits": 8
     },
     "model.layers.0.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.0.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.0.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.0.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.1.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.1.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.1.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.1.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.1.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.2.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.2.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.2.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.2.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.2.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.3.self_attn.q_proj": {
       "bits": 8
       "bits": 8
     },
     "model.layers.3.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.3.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.3.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.3.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.4.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.4.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.4.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.4.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.4.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.5.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.5.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.5.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.5.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.5.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.6.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.6.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.6.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.6.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.6.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.7.self_attn.q_proj": {
       "bits": 8
       "bits": 8
     },
     "model.layers.7.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.7.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.7.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.7.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.8.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.8.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.8.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.8.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.8.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.9.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.9.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.9.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.9.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.9.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.10.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.10.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.10.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.10.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.10.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.11.self_attn.q_proj": {
       "bits": 8
       "bits": 8
     },
     "model.layers.11.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.11.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.11.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.11.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.12.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.12.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.12.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.12.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.12.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.13.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.13.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.13.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.13.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.13.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.14.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.14.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.14.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.14.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.14.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.15.self_attn.q_proj": {
       "bits": 8
       "bits": 8
     },
     "model.layers.15.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.15.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.15.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.15.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.16.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.16.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.16.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.16.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.16.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.17.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.17.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.17.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.17.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.17.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.18.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.18.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.18.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.18.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.18.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.19.self_attn.q_proj": {
       "bits": 8
       "bits": 8
     },
     "model.layers.19.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.19.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.19.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.19.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.20.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.20.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.20.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.20.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.20.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.21.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.21.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.21.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.21.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.21.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.22.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.22.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.22.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.22.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.22.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.23.self_attn.q_proj": {
       "bits": 8
       "bits": 8
     },
     "model.layers.23.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.23.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.23.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.23.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.24.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.24.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.24.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.24.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.24.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.25.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.25.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.25.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.25.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.25.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.26.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.26.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.26.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.26.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.26.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.27.self_attn.q_proj": {
       "bits": 8
       "bits": 8
     },
     "model.layers.27.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.27.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.27.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.27.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.28.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.28.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.28.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.28.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.28.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.29.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.29.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.29.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.29.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.29.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.30.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.30.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.30.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.30.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.30.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.31.self_attn.q_proj": {
       "bits": 8
       "bits": 8
     },
     "model.layers.31.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.31.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.31.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.31.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.32.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.32.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.32.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.32.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.32.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.33.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.33.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.33.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.33.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.33.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.34.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.34.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.34.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.34.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.34.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.35.self_attn.q_proj": {
       "bits": 8
       "bits": 8
     },
     "model.layers.35.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.35.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.35.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.35.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.36.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.36.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.36.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.36.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.36.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.37.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.37.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.37.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.37.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.37.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.38.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.38.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.38.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.38.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.38.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.39.self_attn.q_proj": {
       "bits": 8
       "bits": 8
     },
     "model.layers.39.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.39.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.39.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.39.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.40.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.40.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.40.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.40.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.40.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.41.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.41.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.41.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.41.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.41.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.42.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.42.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.42.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.42.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.42.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.43.self_attn.q_proj": {
       "bits": 8
       "bits": 8
     },
     "model.layers.43.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.43.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.43.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.43.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.44.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.44.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.44.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.44.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.44.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.45.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.45.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.45.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.45.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.45.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.46.linear_attn.in_proj_qkvz": {
       "bits": 8
       "bits": 8
     },
     "model.layers.46.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.46.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.46.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.46.mlp.shared_expert.down_proj": {
+      "bits": 8
     },
     "model.layers.47.self_attn.q_proj": {
       "bits": 8
       "bits": 8
     },
     "model.layers.47.mlp.gate": {
+      "bits": 16,
+      "data_type": "float"
+    },
+    "model.layers.47.mlp.shared_expert.gate_proj": {
       "bits": 8
     },
+    "model.layers.47.mlp.shared_expert.up_proj": {
+      "bits": 8
+    },
+    "model.layers.47.mlp.shared_expert.down_proj": {
+      "bits": 8
     }
   }
 }

tokenizer.json CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:aeb13307a71acd8fe81861d94ad54ab689df773318809eed3cbe794b4492dae4
-size 11422654

 version https://git-lfs.github.com/spec/v1
+oid sha256:c0acdaba32b920d640afb36af4396c91974e074735636e4016d17a8ed9c03730
+size 11422753