Upload Solar Open2 250B Nota NVFP4

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+model.safetensors.index.json filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,109 @@
+---
+base_model: upstage/Solar-Open2-250B
+base_model_relation: quantized
+library_name: vllm
+pipeline_tag: text-generation
+language:
+  - en
+  - ko
+  - ja
+tags:
+  - quantization
+  - nvfp4
+  - vllm
+  - moe
+  - nota
+---
+
+# Solar Open2 250B — Nota NVFP4
+
+[Nota AI](https://www.nota.ai/) presents a 4-bit quantized release of
+[Upstage](https://www.upstage.ai/)'s
+[Solar Open2 250B](https://huggingface.co/upstage/Solar-Open2-250B), produced with
+Nota AI's proprietary quantization technology specialized for Mixture-of-Experts (MoE)
+large language models.
+
+## Highlights
+
+- **NVFP4 (4-bit float, W4A4)** — `group_size=16`, packed in the `llm-compressor`
+  (compressed-tensors) format for direct serving in
+  [vLLM](https://github.com/vllm-project/vllm). Both weights and activations are
+  quantized to 4-bit floating point.
+  - > **Requires NVIDIA Blackwell.** NVFP4 relies on the FP4 tensor cores introduced in
+    > the Blackwell architecture (e.g. B200 / GB200), so inference must run on a
+    > Blackwell-class GPU. Earlier architectures (Hopper, Ada, Ampere) do not support
+    > NVFP4 execution.
+- **Nota AI's proprietary MoE quantization framework.** This release is built upon a
+  suite of techniques developed by Nota AI to preserve model quality under aggressive
+  low-bit quantization of MoE architectures:
+  - A MoE-specialized calibration-dataset construction method, which achieved
+    **1st place across all tracks** at the NVIDIA Nemotron Hackathon.
+  - [**DREAM-MoE**](https://openreview.net/pdf?id=Wyhqwjl51A) and
+    [**SRA-MoE**](https://openreview.net/pdf?id=H0NoX02erJ), two quantization algorithms
+    proposed by Nota AI (published at the *ICML 2026 Workshop on AdaptFM*), which
+    preserve MoE routing decisions and align expert-routing behavior throughout the
+    quantization process.
+
+
+## Performance
+
+### Weight footprint
+
+| Precision          | Weight footprint |
+| ------------------ | :--------------: |
+| BF16               |     500.6 GB     |
+| W4A4 (NVFP4-Nota)  |     153.3 GB      |
+
+### Benchmarks
+
+| Benchmark               |       BF16       | W4A4 (NVFP4-Nota) |
+| ----------------------- | :--------------: | :---------------: |
+| Tau2-Bench              |      75.20       |       75.08       |
+| HLE                     |      27.88       |       27.66       |
+| GPQA Diamond            |      86.26       |       85.45       |
+| IFBench                 |      80.00       |       81.02       |
+| LiveCodeBench (v5–v6)   |      87.03       |       88.55       |
+| MMLU-Pro                |      86.19       |       86.15       |
+| AIME 2026 (EN)          |      95.67       |       96.67       |
+| IFEval (EN)             |      94.09       |       92.61       |
+| HMMT                    |      92.05       |       90.15       |
+| KMMLU-Pro               |      78.38       |       77.93       |
+| HAE-RAE Bench v1.1      |      73.84       |       72.84       |
+| AIME (KO)               |      97.67       |       97.00       |
+| KBL                     |      75.51       |       75.40       |
+| KBank-MMLU              |      80.80       |       80.68       |
+| KorMedMCQA              |      92.99       |       93.05       |
+| **Avg.**                |    **81.57**     |     **81.35**     |
+
+## Usage
+
+This model is packed in the NVFP4 (compressed-tensors) format and can be served directly
+with vLLM on a **Blackwell-class GPU**:
+
+```bash
+vllm serve nota-ai/Solar-Open2-250B-Nota-NVFP4 \
+  --tensor-parallel-size 8 \
+  --trust-remote-code
+```
+
+See the original model card for prompt format and further details:
+[upstage/Solar-Open2-250B](https://huggingface.co/upstage/Solar-Open2-250B).
+
+## Citation
+```bibtex
+@inproceedings{park2026dreammoe,
+  title     = {{DREAM-MoE}: Downstream Routing Error-Aware Margin-Preserving Quantization for Mixture-of-Experts Large Language Models},
+  author    = {Park, Hancheol and Lee, Geonho and Kim, Tae-Ho},
+  booktitle = {ICML 2026 Workshop on Adaptive Foundation Models (AdaptFM)},
+  year      = {2026},
+  url       = {https://openreview.net/forum?id=Wyhqwjl51A},
+}
+
+@inproceedings{lee2026sramoe,
+  title     = {{SRA-MoE}: Output-Aware Selective Router Alignment for MoE Quantization},
+  author    = {Lee, Geonho and Park, Hancheol and Lee, Seunghyun and Choi, Jungwook and Kim, Tae-Ho},
+  booktitle = {ICML 2026 Workshop on Adaptive Foundation Models (AdaptFM)},
+  year      = {2026},
+  url       = {https://openreview.net/forum?id=H0NoX02erJ},
+}
+```

chat_template.jinja

@@ -0,0 +1,229 @@
+{#- ======== Template Parameters ========  #}
+{%- set add_generation_prompt = add_generation_prompt if add_generation_prompt is defined else true %}
+{%- set provider_system_prompt = provider_system_prompt if provider_system_prompt else false %}
+{%- set reasoning_effort = reasoning_effort if reasoning_effort is defined else "high" %}
+{%- set think_render_option = think_render_option if think_render_option is defined else "interleaved" %}
+
+{#- ======== System Block State ========  #}
+{%- set sys_ns = namespace(is_first_block=true) -%}
+
+{#- ======== Find last user message index ========  #}
+{%- set last_user_idx = namespace(value=-1) -%}
+{%- for message in messages -%}
+    {%- if message.role == 'user' -%}
+        {%- set last_user_idx.value = loop.index0 -%}
+    {%- endif -%}
+{%- endfor -%}
+
+{#- ======== System messages renderers ========  #}
+{%- macro render_system_message(user_system_messages) %}
+    {%- if provider_system_prompt %}
+        {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
+        {%- set sys_ns.is_first_block = false %}
+        {{- "## Provider System Prompt\n\nYou are Solar Open2 250B, a large language model trained by Upstage AI, a Korean startup. Your knowledge cutoff is 2026-02. The current date is " + strftime_now("%Y-%m-%d") + "." }}
+    {%- endif -%}
+    {%- if user_system_messages %}
+        {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
+        {%- set sys_ns.is_first_block = false %}
+        {{- "## System Prompt" }}
+        {%- for system_message in user_system_messages %}
+            {{- "\n\n" }}
+            {{- system_message }}
+        {%- endfor %}
+    {%- endif -%}
+{%- endmacro %}
+
+{%- macro render_tool_instruction(tools, is_last_block) %}
+    {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
+    {%- set sys_ns.is_first_block = false %}
+    {{- "## Tools\n- You may invoke one or more tools to assist with the user's query." }}
+    {{- "\n\n### Available Tools\n" }}
+    {%- for tool in tools %}
+        {{- "<|tool:start|>" }}
+        {{- tool.function | tojson }}
+        {{- "<|tool:end|>\n" }}
+    {%- endfor %}
+    {{- "\n### Tool Call Instruction\n" }}
+    {{- "- If using a tool, any reasoning must strictly precede the call. Do not append any text after the tool call.\n" }}
+    {{- "- If no tool is required, answer directly from your knowledge without ever mentioning the availability or absence of tools.\n" }}
+    {{- "- Each tool call MUST use this following format: <|tool_call:start|>{example-tool-name}\n<|tool_arg:start|>{example-key-name-1}<|tool_arg:value|>{example-value-1}<|tool_arg:end|>\n<|tool_arg:start|>{example-key-name-2}<|tool_arg:value|>{example-value-2}<|tool_arg:end|>\n<|tool_call:end|>\n" }}
+{%- endmacro %}
+
+{#-
+Input convention for `response_format`:
+- Per the OpenAI Chat Completions API, `response_format` is an object of one of:
+    {"type": "json_schema", "json_schema": {"name": ..., "schema": {...}, "strict": ...}}
+    {"type": "json_object"}
+- Only `response_format.json_schema.schema` (the inner JSON Schema) is injected
+  into the prompt; the OpenAI wrapper fields (`name`, `strict`, ...) are dropped.
+- For `json_object`, no schema exists, so we emit a generic JSON-only instruction.
+- vLLM does NOT pass `response_format` to chat templates by default; it routes it
+  to guided decoding instead. This template/whale opts into prompt injection,
+  so callers serving the model directly via vLLM must opt in via
+  `chat_template_kwargs` (or rely on whale's normalization).
+-#}
+{%- macro render_json_response_format_instruction(response_format) %}
+    {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
+    {%- set sys_ns.is_first_block = false %}
+    {{- "## Output Format Constraint" }}
+    {%- if response_format.type == "json_schema" %}
+        {{- "\n\n- Your final response should follow the JSON schema:\n```json\n" }}
+        {{- response_format.json_schema.schema | tojson }}
+        {{- "\n```\n- Please ensure your answers adhere to this format and do not contain any unnecessary text." }}
+    {%- elif response_format.type == "json_object" %}
+        {{- "\n\n- Your final response must be a valid JSON object." }}
+        {{- "\n- Do not include any text outside the JSON object." }}
+    {%- endif %}
+{%- endmacro %}
+
+{#-
+Input convention for `tool_arguments`:
+- Must be a mapping (dict), NOT a JSON-encoded string.
+- The OpenAI API spec defines `tool_calls[].function.arguments` as a JSON string,
+  but vLLM, HuggingFace transformers tool-use templates (Qwen, Hermes, Llama, ...),
+  and this template all expect the value to be parsed into a dict before rendering.
+- vLLM normalizes this in `_postprocess_messages()` (vllm/entrypoints/chat_utils.py).
+- Callers (whale, training pipelines that call `tokenizer.apply_chat_template`
+  directly, ...) are responsible for parsing the JSON string into a dict before
+  invoking the template.
+-#}
+{%- macro render_tool_arguments(tool_arguments) %}
+    {%- for args_name, args_value in tool_arguments|items %}
+        {{- "<|tool_arg:start|>"+ args_name + "<|tool_arg:value|>"}}
+        {%- set args_value = args_value if args_value is string else (args_value | tojson | safe) %}
+        {{- args_value }}
+        {{- "<|tool_arg:end|>\n" }}
+    {%- endfor %}
+{%- endmacro %}
+
+{#- ======== Render system message ========  #}
+{%- set ns = namespace(system_messages=[]) -%}
+{%- for message in messages -%}
+    {%- if message.role == 'system' -%}
+        {%- set ns.system_messages = ns.system_messages + [message.content] -%}
+    {%- endif -%}
+{%- endfor -%}
+
+{%- if ns.system_messages or provider_system_prompt or tools or response_format -%}
+    {{- "<|im:start|>system<|im:content|>" }}
+        {{- render_system_message(ns.system_messages) }}
+        {%- if tools -%}
+            {{- render_tool_instruction(tools, not response_format) }}
+        {%- endif %}
+        {%- if response_format -%}
+            {{- render_json_response_format_instruction(response_format) }}
+        {%- endif %}
+    {{- "<|im:end|>\n" }}
+{%- endif -%}
+
+{#- ======== Render main messages ========  #}
+{%- for message in messages -%}
+    {%- if message.role == 'user' -%}
+         {{- "<|im:start|>user<|im:content|>" + message.content + "<|im:end|>\n" }}
+    {%- elif message.role == 'tool' -%}
+        {%- set prev_is_tool = loop.index0 > 0 and messages[loop.index0 - 1].role == 'tool' -%}
+        {#- Only render at the start of a contiguous tool block; subsequent tool messages -#}
+        {#- are already emitted (reordered) within the first one. -#}
+        {%- if not prev_is_tool -%}
+            {%- set start_idx = loop.index0 -%}
+            {#- Collect contiguous tool messages starting at start_idx -#}
+            {%- set tool_ns = namespace(items=[], scanning=true) -%}
+            {%- for j in range(start_idx, messages | length) -%}
+                {%- if tool_ns.scanning -%}
+                    {%- if messages[j].role == 'tool' -%}
+                        {%- set tool_ns.items = tool_ns.items + [messages[j]] -%}
+                    {%- else -%}
+                        {%- set tool_ns.scanning = false -%}
+                    {%- endif -%}
+                {%- endif -%}
+            {%- endfor -%}
+
+            {{- "<|im:start|>tool<|im:content|>" }}
+
+            {%- set prev_msg = messages[start_idx - 1] if start_idx > 0 else none -%}
+            {%- if prev_msg and prev_msg.role == 'assistant' and prev_msg.tool_calls -%}
+                {#- Render tool responses in the order of the preceding assistant's tool_calls ids. -#}
+                {%- set sep_ns = namespace(first=true, rendered_ids=[]) -%}
+                {%- for tool_call in prev_msg.tool_calls -%}
+                    {%- for tool_msg in tool_ns.items -%}
+                        {%- if tool_msg.tool_call_id == tool_call.id -%}
+                            {%- if not sep_ns.first -%}{{- "\n" }}{%- endif -%}
+                            {%- set sep_ns.first = false -%}
+                            {%- set sep_ns.rendered_ids = sep_ns.rendered_ids + [tool_msg.tool_call_id] -%}
+                            {{- "<|tool_response:start|>" + tool_msg.content + "<|tool_response:end|>" }}
+                        {%- endif -%}
+                    {%- endfor -%}
+                {%- endfor -%}
+                {#- Append any orphan tool messages (no matching id) in their original order. -#}
+                {%- for tool_msg in tool_ns.items -%}
+                    {%- if tool_msg.tool_call_id not in sep_ns.rendered_ids -%}
+                        {%- if not sep_ns.first -%}{{- "\n" }}{%- endif -%}
+                        {%- set sep_ns.first = false -%}
+                        {{- "<|tool_response:start|>" + tool_msg.content + "<|tool_response:end|>" }}
+                    {%- endif -%}
+                {%- endfor -%}
+            {%- else -%}
+                {%- for tool_msg in tool_ns.items -%}
+                    {%- if not loop.first -%}{{- "\n" }}{%- endif -%}
+                    {{- "<|tool_response:start|>" + tool_msg.content + "<|tool_response:end|>" }}
+                {%- endfor -%}
+            {%- endif -%}
+
+            {{- "\n<|im:end|>\n" }}
+        {%- endif -%}
+    {%- elif message.role == 'assistant' -%}
+        {{- "<|im:start|>assistant<|im:content|>" }}
+
+        {#- ======== Resolve reasoning field (reasoning or reasoning_content) ========  #}
+        {%- if message.reasoning is defined -%}
+            {%- set reasoning = message.reasoning -%}
+        {%- elif message.reasoning_content is defined -%}
+            {%- set reasoning = message.reasoning_content -%}
+        {%- else -%}
+            {%- set reasoning = none -%}
+        {%- endif -%}
+
+        {#- ======== Assistant Thinking ========  #}
+        {%- if think_render_option == "preserved" -%}
+            {%- if reasoning -%}
+                {{- "<|think:start|>" + reasoning + "<|think:end|>" }}
+            {%- else -%}
+                {{- "<|think:start|><|think:end|>" }}
+            {%- endif -%}
+        {%- elif think_render_option == "interleaved" -%}
+            {%- if reasoning and loop.index0 > last_user_idx.value -%}
+                {{- "<|think:start|>" + reasoning + "<|think:end|>" }}
+            {%- else -%}
+                {{- "<|think:start|><|think:end|>" }}
+            {%- endif -%}
+        {%- endif -%}
+
+        {#- ======== Assistant Messages ========  #}
+        {%- if message.content -%}
+            {{- message.content }}
+        {%- endif -%}
+
+        {#- ======== Assistant Tool calls ========  #}
+        {%- if message.tool_calls -%}
+            {%- for tool_call in message.tool_calls -%}
+                {%- if not loop.first -%}{{- "\n" }}{%- endif -%}
+                {{- "<|tool_call:start|>" + tool_call.function.name + "\n" }}
+                {#- ======== FIXME: QWEN3.5 Method ========  #}
+                {%- if tool_call.function.arguments is defined %}
+                    {{- render_tool_arguments(tool_call.function.arguments) }}
+                {%- endif %}
+                {{- "<|tool_call:end|>" }}
+            {%- endfor -%}
+            {{- "\n" }}
+        {%- endif -%}
+        {{- "<|im:end|>\n" }}
+    {%- endif -%}
+{%- endfor -%}
+
+{%- if add_generation_prompt -%}
+    {%- if reasoning_effort in ["medium", "high"] -%}
+        {{- "<|im:start|>assistant<|im:content|><|think:start|>" }}
+    {%- else -%}
+        {{- "<|im:start|>assistant<|im:content|><|think:start|><|think:end|>" }}
+    {%- endif -%}
+{%- endif -%}

config.json

@@ -0,0 +1,652 @@
+{
+  "architectures": [
+    "SolarOpen2ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_solar_open2.SolarOpen2Config",
+    "AutoModel": "modeling_solar_open2.SolarOpen2Model",
+    "AutoModelForCausalLM": "modeling_solar_open2.SolarOpen2ForCausalLM"
+  },
+  "bos_token_id": 1,
+  "dtype": "bfloat16",
+  "eos_token_id": 2,
+  "first_k_dense_replace": 0,
+  "gqa_interval": 3,
+  "gqa_layers": [
+    0,
+    4,
+    8,
+    12,
+    16,
+    20,
+    24,
+    28,
+    32,
+    36,
+    40,
+    44
+  ],
+  "hc_dynamic": true,
+  "hc_rate": 0,
+  "hc_use_sigmoid": false,
+  "hc_use_tanh": true,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 10240,
+  "kda_allow_neg_eigval": true,
+  "kda_gate_lower_bound": -5.0,
+  "kda_use_full_proj": false,
+  "linear_attn_config": {
+    "head_dim": 128,
+    "num_heads": 64,
+    "num_kv_heads": null,
+    "short_conv_kernel_size": 4
+  },
+  "max_position_embeddings": 1048576,
+  "model_type": "solar_open2",
+  "moe_intermediate_size": 1280,
+  "n_group": 1,
+  "n_routed_experts": 320,
+  "n_shared_experts": 1,
+  "norm_topk_prob": true,
+  "num_attention_heads": 64,
+  "num_experts_per_tok": 8,
+  "num_hidden_layers": 48,
+  "num_key_value_heads": 8,
+  "pad_token_id": 2,
+  "partial_rotary_factor": 1.0,
+  "quantization_config": {
+    "config_groups": {
+      "group_0": {
+        "format": null,
+        "input_activations": {
+          "actorder": null,
+          "block_structure": null,
+          "dynamic": "local",
+          "group_size": 16,
+          "num_bits": 4,
+          "observer": "static_minmax",
+          "observer_kwargs": {},
+          "scale_dtype": "torch.float8_e4m3fn",
+          "strategy": "tensor_group",
+          "symmetric": true,
+          "type": "float",
+          "zp_dtype": null
+        },
+        "output_activations": null,
+        "targets": [
+          "Linear"
+        ],
+        "weights": {
+          "actorder": null,
+          "block_structure": null,
+          "dynamic": false,
+          "group_size": 16,
+          "num_bits": 4,
+          "observer": "static_minmax",
+          "observer_kwargs": {},
+          "scale_dtype": "torch.float8_e4m3fn",
+          "strategy": "tensor_group",
+          "symmetric": true,
+          "type": "float",
+          "zp_dtype": null
+        }
+      }
+    },
+    "format": "nvfp4-pack-quantized",
+    "global_compression_ratio": null,
+    "ignore": [
+      "re:.*model\\.layers\\.[0-47]\\.mlp\\.gate.*",
+      "model.layers.0.self_attn.q_proj",
+      "model.layers.0.self_attn.k_proj",
+      "model.layers.0.self_attn.v_proj",
+      "model.layers.0.self_attn.o_proj",
+      "model.layers.0.self_attn.g_proj",
+      "model.layers.1.self_attn.q_proj",
+      "model.layers.1.self_attn.k_proj",
+      "model.layers.1.self_attn.v_proj",
+      "model.layers.1.self_attn.f_a_proj",
+      "model.layers.1.self_attn.f_b_proj",
+      "model.layers.1.self_attn.b_proj",
+      "model.layers.1.self_attn.g_a_proj",
+      "model.layers.1.self_attn.g_b_proj",
+      "model.layers.1.self_attn.o_proj",
+      "model.layers.2.self_attn.q_proj",
+      "model.layers.2.self_attn.k_proj",
+      "model.layers.2.self_attn.v_proj",
+      "model.layers.2.self_attn.f_a_proj",
+      "model.layers.2.self_attn.f_b_proj",
+      "model.layers.2.self_attn.b_proj",
+      "model.layers.2.self_attn.g_a_proj",
+      "model.layers.2.self_attn.g_b_proj",
+      "model.layers.2.self_attn.o_proj",
+      "model.layers.3.self_attn.q_proj",
+      "model.layers.3.self_attn.k_proj",
+      "model.layers.3.self_attn.v_proj",
+      "model.layers.3.self_attn.f_a_proj",
+      "model.layers.3.self_attn.f_b_proj",
+      "model.layers.3.self_attn.b_proj",
+      "model.layers.3.self_attn.g_a_proj",
+      "model.layers.3.self_attn.g_b_proj",
+      "model.layers.3.self_attn.o_proj",
+      "model.layers.4.self_attn.q_proj",
+      "model.layers.4.self_attn.k_proj",
+      "model.layers.4.self_attn.v_proj",
+      "model.layers.4.self_attn.o_proj",
+      "model.layers.4.self_attn.g_proj",
+      "model.layers.5.self_attn.q_proj",
+      "model.layers.5.self_attn.k_proj",
+      "model.layers.5.self_attn.v_proj",
+      "model.layers.5.self_attn.f_a_proj",
+      "model.layers.5.self_attn.f_b_proj",
+      "model.layers.5.self_attn.b_proj",
+      "model.layers.5.self_attn.g_a_proj",
+      "model.layers.5.self_attn.g_b_proj",
+      "model.layers.5.self_attn.o_proj",
+      "model.layers.6.self_attn.q_proj",
+      "model.layers.6.self_attn.k_proj",
+      "model.layers.6.self_attn.v_proj",
+      "model.layers.6.self_attn.f_a_proj",
+      "model.layers.6.self_attn.f_b_proj",
+      "model.layers.6.self_attn.b_proj",
+      "model.layers.6.self_attn.g_a_proj",
+      "model.layers.6.self_attn.g_b_proj",
+      "model.layers.6.self_attn.o_proj",
+      "model.layers.7.self_attn.q_proj",
+      "model.layers.7.self_attn.k_proj",
+      "model.layers.7.self_attn.v_proj",
+      "model.layers.7.self_attn.f_a_proj",
+      "model.layers.7.self_attn.f_b_proj",
+      "model.layers.7.self_attn.b_proj",
+      "model.layers.7.self_attn.g_a_proj",
+      "model.layers.7.self_attn.g_b_proj",
+      "model.layers.7.self_attn.o_proj",
+      "model.layers.8.self_attn.q_proj",
+      "model.layers.8.self_attn.k_proj",
+      "model.layers.8.self_attn.v_proj",
+      "model.layers.8.self_attn.o_proj",
+      "model.layers.8.self_attn.g_proj",
+      "model.layers.9.self_attn.q_proj",
+      "model.layers.9.self_attn.k_proj",
+      "model.layers.9.self_attn.v_proj",
+      "model.layers.9.self_attn.f_a_proj",
+      "model.layers.9.self_attn.f_b_proj",
+      "model.layers.9.self_attn.b_proj",
+      "model.layers.9.self_attn.g_a_proj",
+      "model.layers.9.self_attn.g_b_proj",
+      "model.layers.9.self_attn.o_proj",
+      "model.layers.10.self_attn.q_proj",
+      "model.layers.10.self_attn.k_proj",
+      "model.layers.10.self_attn.v_proj",
+      "model.layers.10.self_attn.f_a_proj",
+      "model.layers.10.self_attn.f_b_proj",
+      "model.layers.10.self_attn.b_proj",
+      "model.layers.10.self_attn.g_a_proj",
+      "model.layers.10.self_attn.g_b_proj",
+      "model.layers.10.self_attn.o_proj",
+      "model.layers.11.self_attn.q_proj",
+      "model.layers.11.self_attn.k_proj",
+      "model.layers.11.self_attn.v_proj",
+      "model.layers.11.self_attn.f_a_proj",
+      "model.layers.11.self_attn.f_b_proj",
+      "model.layers.11.self_attn.b_proj",
+      "model.layers.11.self_attn.g_a_proj",
+      "model.layers.11.self_attn.g_b_proj",
+      "model.layers.11.self_attn.o_proj",
+      "model.layers.12.self_attn.q_proj",
+      "model.layers.12.self_attn.k_proj",
+      "model.layers.12.self_attn.v_proj",
+      "model.layers.12.self_attn.o_proj",
+      "model.layers.12.self_attn.g_proj",
+      "model.layers.13.self_attn.q_proj",
+      "model.layers.13.self_attn.k_proj",
+      "model.layers.13.self_attn.v_proj",
+      "model.layers.13.self_attn.f_a_proj",
+      "model.layers.13.self_attn.f_b_proj",
+      "model.layers.13.self_attn.b_proj",
+      "model.layers.13.self_attn.g_a_proj",
+      "model.layers.13.self_attn.g_b_proj",
+      "model.layers.13.self_attn.o_proj",
+      "model.layers.14.self_attn.q_proj",
+      "model.layers.14.self_attn.k_proj",
+      "model.layers.14.self_attn.v_proj",
+      "model.layers.14.self_attn.f_a_proj",
+      "model.layers.14.self_attn.f_b_proj",
+      "model.layers.14.self_attn.b_proj",
+      "model.layers.14.self_attn.g_a_proj",
+      "model.layers.14.self_attn.g_b_proj",
+      "model.layers.14.self_attn.o_proj",
+      "model.layers.15.self_attn.q_proj",
+      "model.layers.15.self_attn.k_proj",
+      "model.layers.15.self_attn.v_proj",
+      "model.layers.15.self_attn.f_a_proj",
+      "model.layers.15.self_attn.f_b_proj",
+      "model.layers.15.self_attn.b_proj",
+      "model.layers.15.self_attn.g_a_proj",
+      "model.layers.15.self_attn.g_b_proj",
+      "model.layers.15.self_attn.o_proj",
+      "model.layers.16.self_attn.q_proj",
+      "model.layers.16.self_attn.k_proj",
+      "model.layers.16.self_attn.v_proj",
+      "model.layers.16.self_attn.o_proj",
+      "model.layers.16.self_attn.g_proj",
+      "model.layers.17.self_attn.q_proj",
+      "model.layers.17.self_attn.k_proj",
+      "model.layers.17.self_attn.v_proj",
+      "model.layers.17.self_attn.f_a_proj",
+      "model.layers.17.self_attn.f_b_proj",
+      "model.layers.17.self_attn.b_proj",
+      "model.layers.17.self_attn.g_a_proj",
+      "model.layers.17.self_attn.g_b_proj",
+      "model.layers.17.self_attn.o_proj",
+      "model.layers.18.self_attn.q_proj",
+      "model.layers.18.self_attn.k_proj",
+      "model.layers.18.self_attn.v_proj",
+      "model.layers.18.self_attn.f_a_proj",
+      "model.layers.18.self_attn.f_b_proj",
+      "model.layers.18.self_attn.b_proj",
+      "model.layers.18.self_attn.g_a_proj",
+      "model.layers.18.self_attn.g_b_proj",
+      "model.layers.18.self_attn.o_proj",
+      "model.layers.19.self_attn.q_proj",
+      "model.layers.19.self_attn.k_proj",
+      "model.layers.19.self_attn.v_proj",
+      "model.layers.19.self_attn.f_a_proj",
+      "model.layers.19.self_attn.f_b_proj",
+      "model.layers.19.self_attn.b_proj",
+      "model.layers.19.self_attn.g_a_proj",
+      "model.layers.19.self_attn.g_b_proj",
+      "model.layers.19.self_attn.o_proj",
+      "model.layers.20.self_attn.q_proj",
+      "model.layers.20.self_attn.k_proj",
+      "model.layers.20.self_attn.v_proj",
+      "model.layers.20.self_attn.o_proj",
+      "model.layers.20.self_attn.g_proj",
+      "model.layers.21.self_attn.q_proj",
+      "model.layers.21.self_attn.k_proj",
+      "model.layers.21.self_attn.v_proj",
+      "model.layers.21.self_attn.f_a_proj",
+      "model.layers.21.self_attn.f_b_proj",
+      "model.layers.21.self_attn.b_proj",
+      "model.layers.21.self_attn.g_a_proj",
+      "model.layers.21.self_attn.g_b_proj",
+      "model.layers.21.self_attn.o_proj",
+      "model.layers.22.self_attn.q_proj",
+      "model.layers.22.self_attn.k_proj",
+      "model.layers.22.self_attn.v_proj",
+      "model.layers.22.self_attn.f_a_proj",
+      "model.layers.22.self_attn.f_b_proj",
+      "model.layers.22.self_attn.b_proj",
+      "model.layers.22.self_attn.g_a_proj",
+      "model.layers.22.self_attn.g_b_proj",
+      "model.layers.22.self_attn.o_proj",
+      "model.layers.23.self_attn.q_proj",
+      "model.layers.23.self_attn.k_proj",
+      "model.layers.23.self_attn.v_proj",
+      "model.layers.23.self_attn.f_a_proj",
+      "model.layers.23.self_attn.f_b_proj",
+      "model.layers.23.self_attn.b_proj",
+      "model.layers.23.self_attn.g_a_proj",
+      "model.layers.23.self_attn.g_b_proj",
+      "model.layers.23.self_attn.o_proj",
+      "model.layers.24.self_attn.q_proj",
+      "model.layers.24.self_attn.k_proj",
+      "model.layers.24.self_attn.v_proj",
+      "model.layers.24.self_attn.o_proj",
+      "model.layers.24.self_attn.g_proj",
+      "model.layers.25.self_attn.q_proj",
+      "model.layers.25.self_attn.k_proj",
+      "model.layers.25.self_attn.v_proj",
+      "model.layers.25.self_attn.f_a_proj",
+      "model.layers.25.self_attn.f_b_proj",
+      "model.layers.25.self_attn.b_proj",
+      "model.layers.25.self_attn.g_a_proj",
+      "model.layers.25.self_attn.g_b_proj",
+      "model.layers.25.self_attn.o_proj",
+      "model.layers.26.self_attn.q_proj",
+      "model.layers.26.self_attn.k_proj",
+      "model.layers.26.self_attn.v_proj",
+      "model.layers.26.self_attn.f_a_proj",
+      "model.layers.26.self_attn.f_b_proj",
+      "model.layers.26.self_attn.b_proj",
+      "model.layers.26.self_attn.g_a_proj",
+      "model.layers.26.self_attn.g_b_proj",
+      "model.layers.26.self_attn.o_proj",
+      "model.layers.27.self_attn.q_proj",
+      "model.layers.27.self_attn.k_proj",
+      "model.layers.27.self_attn.v_proj",
+      "model.layers.27.self_attn.f_a_proj",
+      "model.layers.27.self_attn.f_b_proj",
+      "model.layers.27.self_attn.b_proj",
+      "model.layers.27.self_attn.g_a_proj",
+      "model.layers.27.self_attn.g_b_proj",
+      "model.layers.27.self_attn.o_proj",
+      "model.layers.28.self_attn.q_proj",
+      "model.layers.28.self_attn.k_proj",
+      "model.layers.28.self_attn.v_proj",
+      "model.layers.28.self_attn.o_proj",
+      "model.layers.28.self_attn.g_proj",
+      "model.layers.29.self_attn.q_proj",
+      "model.layers.29.self_attn.k_proj",
+      "model.layers.29.self_attn.v_proj",
+      "model.layers.29.self_attn.f_a_proj",
+      "model.layers.29.self_attn.f_b_proj",
+      "model.layers.29.self_attn.b_proj",
+      "model.layers.29.self_attn.g_a_proj",
+      "model.layers.29.self_attn.g_b_proj",
+      "model.layers.29.self_attn.o_proj",
+      "model.layers.30.self_attn.q_proj",
+      "model.layers.30.self_attn.k_proj",
+      "model.layers.30.self_attn.v_proj",
+      "model.layers.30.self_attn.f_a_proj",
+      "model.layers.30.self_attn.f_b_proj",
+      "model.layers.30.self_attn.b_proj",
+      "model.layers.30.self_attn.g_a_proj",
+      "model.layers.30.self_attn.g_b_proj",
+      "model.layers.30.self_attn.o_proj",
+      "model.layers.31.self_attn.q_proj",
+      "model.layers.31.self_attn.k_proj",
+      "model.layers.31.self_attn.v_proj",
+      "model.layers.31.self_attn.f_a_proj",
+      "model.layers.31.self_attn.f_b_proj",
+      "model.layers.31.self_attn.b_proj",
+      "model.layers.31.self_attn.g_a_proj",
+      "model.layers.31.self_attn.g_b_proj",
+      "model.layers.31.self_attn.o_proj",
+      "model.layers.32.self_attn.q_proj",
+      "model.layers.32.self_attn.k_proj",
+      "model.layers.32.self_attn.v_proj",
+      "model.layers.32.self_attn.o_proj",
+      "model.layers.32.self_attn.g_proj",
+      "model.layers.33.self_attn.q_proj",
+      "model.layers.33.self_attn.k_proj",
+      "model.layers.33.self_attn.v_proj",
+      "model.layers.33.self_attn.f_a_proj",
+      "model.layers.33.self_attn.f_b_proj",
+      "model.layers.33.self_attn.b_proj",
+      "model.layers.33.self_attn.g_a_proj",
+      "model.layers.33.self_attn.g_b_proj",
+      "model.layers.33.self_attn.o_proj",
+      "model.layers.34.self_attn.q_proj",
+      "model.layers.34.self_attn.k_proj",
+      "model.layers.34.self_attn.v_proj",
+      "model.layers.34.self_attn.f_a_proj",
+      "model.layers.34.self_attn.f_b_proj",
+      "model.layers.34.self_attn.b_proj",
+      "model.layers.34.self_attn.g_a_proj",
+      "model.layers.34.self_attn.g_b_proj",
+      "model.layers.34.self_attn.o_proj",
+      "model.layers.35.self_attn.q_proj",
+      "model.layers.35.self_attn.k_proj",
+      "model.layers.35.self_attn.v_proj",
+      "model.layers.35.self_attn.f_a_proj",
+      "model.layers.35.self_attn.f_b_proj",
+      "model.layers.35.self_attn.b_proj",
+      "model.layers.35.self_attn.g_a_proj",
+      "model.layers.35.self_attn.g_b_proj",
+      "model.layers.35.self_attn.o_proj",
+      "model.layers.36.self_attn.q_proj",
+      "model.layers.36.self_attn.k_proj",
+      "model.layers.36.self_attn.v_proj",
+      "model.layers.36.self_attn.o_proj",
+      "model.layers.36.self_attn.g_proj",
+      "model.layers.37.self_attn.q_proj",
+      "model.layers.37.self_attn.k_proj",
+      "model.layers.37.self_attn.v_proj",
+      "model.layers.37.self_attn.f_a_proj",
+      "model.layers.37.self_attn.f_b_proj",
+      "model.layers.37.self_attn.b_proj",
+      "model.layers.37.self_attn.g_a_proj",
+      "model.layers.37.self_attn.g_b_proj",
+      "model.layers.37.self_attn.o_proj",
+      "model.layers.38.self_attn.q_proj",
+      "model.layers.38.self_attn.k_proj",
+      "model.layers.38.self_attn.v_proj",
+      "model.layers.38.self_attn.f_a_proj",
+      "model.layers.38.self_attn.f_b_proj",
+      "model.layers.38.self_attn.b_proj",
+      "model.layers.38.self_attn.g_a_proj",
+      "model.layers.38.self_attn.g_b_proj",
+      "model.layers.38.self_attn.o_proj",
+      "model.layers.39.self_attn.q_proj",
+      "model.layers.39.self_attn.k_proj",
+      "model.layers.39.self_attn.v_proj",
+      "model.layers.39.self_attn.f_a_proj",
+      "model.layers.39.self_attn.f_b_proj",
+      "model.layers.39.self_attn.b_proj",
+      "model.layers.39.self_attn.g_a_proj",
+      "model.layers.39.self_attn.g_b_proj",
+      "model.layers.39.self_attn.o_proj",
+      "model.layers.40.self_attn.q_proj",
+      "model.layers.40.self_attn.k_proj",
+      "model.layers.40.self_attn.v_proj",
+      "model.layers.40.self_attn.o_proj",
+      "model.layers.40.self_attn.g_proj",
+      "model.layers.41.self_attn.q_proj",
+      "model.layers.41.self_attn.k_proj",
+      "model.layers.41.self_attn.v_proj",
+      "model.layers.41.self_attn.f_a_proj",
+      "model.layers.41.self_attn.f_b_proj",
+      "model.layers.41.self_attn.b_proj",
+      "model.layers.41.self_attn.g_a_proj",
+      "model.layers.41.self_attn.g_b_proj",
+      "model.layers.41.self_attn.o_proj",
+      "model.layers.42.self_attn.q_proj",
+      "model.layers.42.self_attn.k_proj",
+      "model.layers.42.self_attn.v_proj",
+      "model.layers.42.self_attn.f_a_proj",
+      "model.layers.42.self_attn.f_b_proj",
+      "model.layers.42.self_attn.b_proj",
+      "model.layers.42.self_attn.g_a_proj",
+      "model.layers.42.self_attn.g_b_proj",
+      "model.layers.42.self_attn.o_proj",
+      "model.layers.43.self_attn.q_proj",
+      "model.layers.43.self_attn.k_proj",
+      "model.layers.43.self_attn.v_proj",
+      "model.layers.43.self_attn.f_a_proj",
+      "model.layers.43.self_attn.f_b_proj",
+      "model.layers.43.self_attn.b_proj",
+      "model.layers.43.self_attn.g_a_proj",
+      "model.layers.43.self_attn.g_b_proj",
+      "model.layers.43.self_attn.o_proj",
+      "model.layers.44.self_attn.q_proj",
+      "model.layers.44.self_attn.k_proj",
+      "model.layers.44.self_attn.v_proj",
+      "model.layers.44.self_attn.o_proj",
+      "model.layers.44.self_attn.g_proj",
+      "model.layers.45.self_attn.q_proj",
+      "model.layers.45.self_attn.k_proj",
+      "model.layers.45.self_attn.v_proj",
+      "model.layers.45.self_attn.f_a_proj",
+      "model.layers.45.self_attn.f_b_proj",
+      "model.layers.45.self_attn.b_proj",
+      "model.layers.45.self_attn.g_a_proj",
+      "model.layers.45.self_attn.g_b_proj",
+      "model.layers.45.self_attn.o_proj",
+      "model.layers.46.self_attn.q_proj",
+      "model.layers.46.self_attn.k_proj",
+      "model.layers.46.self_attn.v_proj",
+      "model.layers.46.self_attn.f_a_proj",
+      "model.layers.46.self_attn.f_b_proj",
+      "model.layers.46.self_attn.b_proj",
+      "model.layers.46.self_attn.g_a_proj",
+      "model.layers.46.self_attn.g_b_proj",
+      "model.layers.46.self_attn.o_proj",
+      "model.layers.47.self_attn.q_proj",
+      "model.layers.47.self_attn.k_proj",
+      "model.layers.47.self_attn.v_proj",
+      "model.layers.47.self_attn.f_a_proj",
+      "model.layers.47.self_attn.f_b_proj",
+      "model.layers.47.self_attn.b_proj",
+      "model.layers.47.self_attn.g_a_proj",
+      "model.layers.47.self_attn.g_b_proj",
+      "model.layers.47.self_attn.o_proj",
+      "model.layers.0.mlp.shared_experts.gate_proj",
+      "model.layers.0.mlp.shared_experts.up_proj",
+      "model.layers.0.mlp.shared_experts.down_proj",
+      "model.layers.1.mlp.shared_experts.gate_proj",
+      "model.layers.1.mlp.shared_experts.up_proj",
+      "model.layers.1.mlp.shared_experts.down_proj",
+      "model.layers.2.mlp.shared_experts.gate_proj",
+      "model.layers.2.mlp.shared_experts.up_proj",
+      "model.layers.2.mlp.shared_experts.down_proj",
+      "model.layers.3.mlp.shared_experts.gate_proj",
+      "model.layers.3.mlp.shared_experts.up_proj",
+      "model.layers.3.mlp.shared_experts.down_proj",
+      "model.layers.4.mlp.shared_experts.gate_proj",
+      "model.layers.4.mlp.shared_experts.up_proj",
+      "model.layers.4.mlp.shared_experts.down_proj",
+      "model.layers.5.mlp.shared_experts.gate_proj",
+      "model.layers.5.mlp.shared_experts.up_proj",
+      "model.layers.5.mlp.shared_experts.down_proj",
+      "model.layers.6.mlp.shared_experts.gate_proj",
+      "model.layers.6.mlp.shared_experts.up_proj",
+      "model.layers.6.mlp.shared_experts.down_proj",
+      "model.layers.7.mlp.shared_experts.gate_proj",
+      "model.layers.7.mlp.shared_experts.up_proj",
+      "model.layers.7.mlp.shared_experts.down_proj",
+      "model.layers.8.mlp.shared_experts.gate_proj",
+      "model.layers.8.mlp.shared_experts.up_proj",
+      "model.layers.8.mlp.shared_experts.down_proj",
+      "model.layers.9.mlp.shared_experts.gate_proj",
+      "model.layers.9.mlp.shared_experts.up_proj",
+      "model.layers.9.mlp.shared_experts.down_proj",
+      "model.layers.10.mlp.shared_experts.gate_proj",
+      "model.layers.10.mlp.shared_experts.up_proj",
+      "model.layers.10.mlp.shared_experts.down_proj",
+      "model.layers.11.mlp.shared_experts.gate_proj",
+      "model.layers.11.mlp.shared_experts.up_proj",
+      "model.layers.11.mlp.shared_experts.down_proj",
+      "model.layers.12.mlp.shared_experts.gate_proj",
+      "model.layers.12.mlp.shared_experts.up_proj",
+      "model.layers.12.mlp.shared_experts.down_proj",
+      "model.layers.13.mlp.shared_experts.gate_proj",
+      "model.layers.13.mlp.shared_experts.up_proj",
+      "model.layers.13.mlp.shared_experts.down_proj",
+      "model.layers.14.mlp.shared_experts.gate_proj",
+      "model.layers.14.mlp.shared_experts.up_proj",
+      "model.layers.14.mlp.shared_experts.down_proj",
+      "model.layers.15.mlp.shared_experts.gate_proj",
+      "model.layers.15.mlp.shared_experts.up_proj",
+      "model.layers.15.mlp.shared_experts.down_proj",
+      "model.layers.16.mlp.shared_experts.gate_proj",
+      "model.layers.16.mlp.shared_experts.up_proj",
+      "model.layers.16.mlp.shared_experts.down_proj",
+      "model.layers.17.mlp.shared_experts.gate_proj",
+      "model.layers.17.mlp.shared_experts.up_proj",
+      "model.layers.17.mlp.shared_experts.down_proj",
+      "model.layers.18.mlp.shared_experts.gate_proj",
+      "model.layers.18.mlp.shared_experts.up_proj",
+      "model.layers.18.mlp.shared_experts.down_proj",
+      "model.layers.19.mlp.shared_experts.gate_proj",
+      "model.layers.19.mlp.shared_experts.up_proj",
+      "model.layers.19.mlp.shared_experts.down_proj",
+      "model.layers.20.mlp.shared_experts.gate_proj",
+      "model.layers.20.mlp.shared_experts.up_proj",
+      "model.layers.20.mlp.shared_experts.down_proj",
+      "model.layers.21.mlp.shared_experts.gate_proj",
+      "model.layers.21.mlp.shared_experts.up_proj",
+      "model.layers.21.mlp.shared_experts.down_proj",
+      "model.layers.22.mlp.shared_experts.gate_proj",
+      "model.layers.22.mlp.shared_experts.up_proj",
+      "model.layers.22.mlp.shared_experts.down_proj",
+      "model.layers.23.mlp.shared_experts.gate_proj",
+      "model.layers.23.mlp.shared_experts.up_proj",
+      "model.layers.23.mlp.shared_experts.down_proj",
+      "model.layers.24.mlp.shared_experts.gate_proj",
+      "model.layers.24.mlp.shared_experts.up_proj",
+      "model.layers.24.mlp.shared_experts.down_proj",
+      "model.layers.25.mlp.shared_experts.gate_proj",
+      "model.layers.25.mlp.shared_experts.up_proj",
+      "model.layers.25.mlp.shared_experts.down_proj",
+      "model.layers.26.mlp.shared_experts.gate_proj",
+      "model.layers.26.mlp.shared_experts.up_proj",
+      "model.layers.26.mlp.shared_experts.down_proj",
+      "model.layers.27.mlp.shared_experts.gate_proj",
+      "model.layers.27.mlp.shared_experts.up_proj",
+      "model.layers.27.mlp.shared_experts.down_proj",
+      "model.layers.28.mlp.shared_experts.gate_proj",
+      "model.layers.28.mlp.shared_experts.up_proj",
+      "model.layers.28.mlp.shared_experts.down_proj",
+      "model.layers.29.mlp.shared_experts.gate_proj",
+      "model.layers.29.mlp.shared_experts.up_proj",
+      "model.layers.29.mlp.shared_experts.down_proj",
+      "model.layers.30.mlp.shared_experts.gate_proj",
+      "model.layers.30.mlp.shared_experts.up_proj",
+      "model.layers.30.mlp.shared_experts.down_proj",
+      "model.layers.31.mlp.shared_experts.gate_proj",
+      "model.layers.31.mlp.shared_experts.up_proj",
+      "model.layers.31.mlp.shared_experts.down_proj",
+      "model.layers.32.mlp.shared_experts.gate_proj",
+      "model.layers.32.mlp.shared_experts.up_proj",
+      "model.layers.32.mlp.shared_experts.down_proj",
+      "model.layers.33.mlp.shared_experts.gate_proj",
+      "model.layers.33.mlp.shared_experts.up_proj",
+      "model.layers.33.mlp.shared_experts.down_proj",
+      "model.layers.34.mlp.shared_experts.gate_proj",
+      "model.layers.34.mlp.shared_experts.up_proj",
+      "model.layers.34.mlp.shared_experts.down_proj",
+      "model.layers.35.mlp.shared_experts.gate_proj",
+      "model.layers.35.mlp.shared_experts.up_proj",
+      "model.layers.35.mlp.shared_experts.down_proj",
+      "model.layers.36.mlp.shared_experts.gate_proj",
+      "model.layers.36.mlp.shared_experts.up_proj",
+      "model.layers.36.mlp.shared_experts.down_proj",
+      "model.layers.37.mlp.shared_experts.gate_proj",
+      "model.layers.37.mlp.shared_experts.up_proj",
+      "model.layers.37.mlp.shared_experts.down_proj",
+      "model.layers.38.mlp.shared_experts.gate_proj",
+      "model.layers.38.mlp.shared_experts.up_proj",
+      "model.layers.38.mlp.shared_experts.down_proj",
+      "model.layers.39.mlp.shared_experts.gate_proj",
+      "model.layers.39.mlp.shared_experts.up_proj",
+      "model.layers.39.mlp.shared_experts.down_proj",
+      "model.layers.40.mlp.shared_experts.gate_proj",
+      "model.layers.40.mlp.shared_experts.up_proj",
+      "model.layers.40.mlp.shared_experts.down_proj",
+      "model.layers.41.mlp.shared_experts.gate_proj",
+      "model.layers.41.mlp.shared_experts.up_proj",
+      "model.layers.41.mlp.shared_experts.down_proj",
+      "model.layers.42.mlp.shared_experts.gate_proj",
+      "model.layers.42.mlp.shared_experts.up_proj",
+      "model.layers.42.mlp.shared_experts.down_proj",
+      "model.layers.43.mlp.shared_experts.gate_proj",
+      "model.layers.43.mlp.shared_experts.up_proj",
+      "model.layers.43.mlp.shared_experts.down_proj",
+      "model.layers.44.mlp.shared_experts.gate_proj",
+      "model.layers.44.mlp.shared_experts.up_proj",
+      "model.layers.44.mlp.shared_experts.down_proj",
+      "model.layers.45.mlp.shared_experts.gate_proj",
+      "model.layers.45.mlp.shared_experts.up_proj",
+      "model.layers.45.mlp.shared_experts.down_proj",
+      "model.layers.46.mlp.shared_experts.gate_proj",
+      "model.layers.46.mlp.shared_experts.up_proj",
+      "model.layers.46.mlp.shared_experts.down_proj",
+      "model.layers.47.mlp.shared_experts.gate_proj",
+      "model.layers.47.mlp.shared_experts.up_proj",
+      "model.layers.47.mlp.shared_experts.down_proj",
+      "lm_head"
+    ],
+    "kv_cache_scheme": null,
+    "provider": "auto-round",
+    "quant_method": "compressed-tensors",
+    "quantization_status": "compressed"
+  },
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000,
+  "routed_scaling_factor": 1.0,
+  "tie_word_embeddings": false,
+  "topk_group": 1,
+  "transformers_version": "4.57.6",
+  "use_cache": true,
+  "use_gqa_gate": true,
+  "use_gqa_gate_bias": false,
+  "use_qk_norm": false,
+  "use_rope": false,
+  "vocab_size": 196608
+}

configuration_solar_open2.py

@@ -0,0 +1,277 @@
+# coding=utf-8
+# Copyright 2025 Upstage AI.
+# Copyright 2025 The ZhipuAI Inc. team and HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# This file has been modified by Upstage AI including
+# -  Hyperparameter Adjustments: Modified the model architecture by increasing vocab_size and num_hidden_layers, while decreasing num_attention_heads, intermediate_size, and moe_intermediate_size.
+# RoPE Configuration: Replaced the generic rope_parameters argument with explicit rope_theta and rope_scaling parameters to define Rotary Positional Embeddings settings.
+#
+# Based on code from: https://github.com/huggingface/transformers/blob/main/src/transformers/models/glm4_moe/configuration_glm4_moe.py
+
+from transformers.configuration_utils import PretrainedConfig
+from transformers.modeling_rope_utils import rope_config_validation
+
+
+class SolarOpen2Config(PretrainedConfig):
+    r"""
+    This is the configuration class to store the configuration of a [`SolarOpen2Model`]. It is used to instantiate a
+    SolarOpen2 model according to the specified arguments, defining the model architecture.
+
+    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
+    documentation from [`PretrainedConfig`] for more information.
+
+
+    Args:
+        vocab_size (`int`, *optional*, defaults to 196608):
+            Vocabulary size of the SolarOpen2 model. Defines the number of different tokens that can be represented by the
+            `inputs_ids` passed when calling [`SolarOpen2Model`]
+        hidden_size (`int`, *optional*, defaults to 4096):
+            Dimension of the hidden representations.
+        intermediate_size (`int`, *optional*, defaults to 10240):
+            Dimension of the MLP representations.
+        num_hidden_layers (`int`, *optional*, defaults to 48):
+            Number of hidden layers in the Transformer encoder.
+        num_attention_heads (`int`, *optional*, defaults to 64):
+            Number of attention heads for each attention layer in the Transformer encoder.
+        partial_rotary_factor (`float`, *optional*, defaults to 1.0):
+            The factor of the partial rotary position.
+        num_key_value_heads (`int`, *optional*, defaults to 8):
+            This is the number of key_value heads that should be used to implement Grouped Query Attention. If
+            `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
+            `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When
+            converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
+            by meanpooling all the original heads within that group. For more details, check out [this
+            paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `32`.
+
+        hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
+            The non-linear activation function (function or string) in the decoder.
+        max_position_embeddings (`int`, *optional*, defaults to 131072):
+            The maximum sequence length that this model might ever be used with.
+        initializer_range (`float`, *optional*, defaults to 0.02):
+            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
+        rms_norm_eps (`float`, *optional*, defaults to 1e-05):
+            The epsilon used by the rms normalization layers.
+        use_cache (`bool`, *optional*, defaults to `True`):
+            Whether or not the model should return the last key/values attentions (not used by all models). Only
+            relevant if `config.is_decoder=True`.
+        tie_word_embeddings (`bool`, *optional*, defaults to `False`):
+            Whether the model's input and output word embeddings should be tied.
+        rope_theta (`float`, *optional*, defaults to 1000000.0):
+            The base period of the RoPE embeddings.
+        rope_scaling (`Dict`, *optional*):
+            Dictionary containing the scaling configuration for the RoPE embeddings. NOTE: if you apply new rope type
+            and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value
+            accordingly.
+            Expected contents:
+                `rope_type` (`str`):
+                    The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope',
+                    'llama3'], with 'default' being the original RoPE implementation.
+                `factor` (`float`, *optional*):
+                    Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In
+                    most scaling types, a `factor` of x will enable the model to handle sequences of length x *
+                    original maximum pre-trained length.
+                `original_max_position_embeddings` (`int`, *optional*):
+                    Used with 'dynamic', 'longrope' and 'llama3'. The original max position embeddings used during
+                    pretraining.
+                `attention_factor` (`float`, *optional*):
+                    Used with 'yarn' and 'longrope'. The scaling factor to be applied on the attention
+                    computation. If unspecified, it defaults to value recommended by the implementation, using the
+                    `factor` field to infer the suggested value.
+                `beta_fast` (`float`, *optional*):
+                    Only used with 'yarn'. Parameter to set the boundary for extrapolation (only) in the linear
+                    ramp function. If unspecified, it defaults to 32.
+                `beta_slow` (`float`, *optional*):
+                    Only used with 'yarn'. Parameter to set the boundary for interpolation (only) in the linear
+                    ramp function. If unspecified, it defaults to 1.
+                `short_factor` (`list[float]`, *optional*):
+                    Only used with 'longrope'. The scaling factor to be applied to short contexts (<
+                    `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden
+                    size divided by the number of attention heads divided by 2
+                `long_factor` (`list[float]`, *optional*):
+                    Only used with 'longrope'. The scaling factor to be applied to long contexts (<
+                    `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden
+                    size divided by the number of attention heads divided by 2
+                `low_freq_factor` (`float`, *optional*):
+                    Only used with 'llama3'. Scaling factor applied to low frequency components of the RoPE
+                `high_freq_factor` (`float`, *optional*):
+                    Only used with 'llama3'. Scaling factor applied to high frequency components of the RoPE
+        attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`):
+            Whether to use a bias in the query, key, value and output projection layers during self-attention.
+        attention_dropout (`float`, *optional*, defaults to 0.0):
+            The dropout ratio for the attention probabilities.
+        moe_intermediate_size (`int`, *optional*, defaults to 1280):
+            Intermediate size of the routed expert.
+        num_experts_per_tok (`int`, *optional*, defaults to 8):
+            number of experts per token.
+        n_shared_experts (`int`, *optional*, defaults to 1):
+            Number of shared experts.
+        n_routed_experts (`int`, *optional*, defaults to 128):
+            Number of routed experts.
+        routed_scaling_factor (`float`, *optional*, defaults to 1.0):
+            Scaling factor or routed experts.
+        n_group (`int`, *optional*, defaults to 1):
+            Number of groups for routed experts.
+        topk_group (`int`, *optional*, defaults to 1):
+            Number of selected groups for each token(for each token, ensuring the selected experts is only within `topk_group` groups).
+        first_k_dense_replace (`int`, *optional*, defaults to 0):
+            Number of dense layers in shallow layers(embed->dense->dense->...->dense->moe->moe...->lm_head).
+                                                            \--k dense layers--/
+        norm_topk_prob (`bool`, *optional*, defaults to `True`):
+            Whether to normalize the topk probabilities.
+        use_qk_norm (`bool`, *optional*, defaults to `False`):
+            Whether to use query-key normalization in the attention
+    ```python
+    >>> from transformers import SolarOpen2Model, SolarOpen2Config
+
+    >>> # Initializing a SolarOpen2 style configuration
+    >>> configuration = SolarOpen2Config()
+
+    >>> # Initializing a model from the SolarOpen2 style configuration
+    >>> model = SolarOpen2Model(configuration)
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```"""
+
+    model_type = "solar_open2"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    # Default tensor parallel plan for base model `SolarOpen2`
+    base_model_tp_plan = {
+        "layers.*.self_attn.q_proj": "colwise",
+        "layers.*.self_attn.k_proj": "colwise",
+        "layers.*.self_attn.v_proj": "colwise",
+        "layers.*.self_attn.o_proj": "rowwise",
+        "layers.*.mlp.experts.*.gate_proj": "colwise",
+        "layers.*.mlp.experts.*.up_proj": "colwise",
+        "layers.*.mlp.experts.*.down_proj": "rowwise",
+        "layers.*.mlp.gate_proj": "colwise",
+        "layers.*.mlp.up_proj": "colwise",
+        "layers.*.mlp.down_proj": "rowwise",
+    }
+    base_model_pp_plan = {
+        "embed_tokens": (["input_ids"], ["inputs_embeds"]),
+        "layers": (["hidden_states", "attention_mask"], ["hidden_states"]),
+        "norm": (["hidden_states"], ["hidden_states"]),
+    }
+
+    def __init__(
+        self,
+        vocab_size=196608,
+        hidden_size=4096,
+        intermediate_size=10240,
+        num_hidden_layers=48,
+        num_attention_heads=64,
+        partial_rotary_factor=1.0,
+        num_key_value_heads=8,
+        hidden_act="silu",
+        max_position_embeddings=131072,
+        initializer_range=0.02,
+        rms_norm_eps=1e-5,
+        use_cache=True,
+        tie_word_embeddings=False,
+        rope_theta=1000000.0,
+        rope_scaling=None,
+        attention_bias=False,
+        attention_dropout=0.0,
+        moe_intermediate_size=1280,
+        num_experts_per_tok=8,
+        n_shared_experts=1,
+        n_routed_experts=128,
+        routed_scaling_factor=1.0,
+        n_group=1,
+        topk_group=1,
+        first_k_dense_replace=0,
+        norm_topk_prob=True,
+        use_qk_norm=False,
+        use_rope=False,
+        gqa_interval=4,
+        gqa_layers=None,
+        use_gqa_gate=True,
+        use_gqa_gate_bias=False,
+        linear_attn_config={
+            "short_conv_kernel_size": 4,
+            "head_dim": 128,
+            "num_heads": 16,
+            "num_kv_heads": None,
+        },
+        kda_use_full_proj=False,
+        kda_gate_lower_bound=-5.0,
+        kda_allow_neg_eigval=False,
+        # Hyper-Connections
+        hc_rate=0,
+        hc_dynamic=True,
+        hc_use_tanh=True,
+        hc_use_sigmoid=False,
+        **kwargs,
+    ):
+        self.vocab_size = vocab_size
+        self.max_position_embeddings = max_position_embeddings
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.partial_rotary_factor = partial_rotary_factor
+
+        self.num_key_value_heads = num_key_value_heads
+        self.hidden_act = hidden_act
+        self.initializer_range = initializer_range
+        self.rms_norm_eps = rms_norm_eps
+        self.use_cache = use_cache
+        self.rope_theta = rope_theta
+        self.rope_scaling = rope_scaling
+        self.attention_bias = attention_bias
+        self.attention_dropout = attention_dropout
+        # Validate the correctness of rotary position embeddings parameters
+        # BC: if there is a 'type' field, move it to 'rope_type'.
+        if self.rope_scaling is not None and "type" in self.rope_scaling:
+            self.rope_scaling["rope_type"] = self.rope_scaling["type"]
+        rope_config_validation(self)
+
+        # MoE arguments
+        self.moe_intermediate_size = moe_intermediate_size
+        self.num_experts_per_tok = num_experts_per_tok
+        self.n_group = n_group
+        self.topk_group = topk_group
+        self.n_shared_experts = n_shared_experts
+        self.n_routed_experts = n_routed_experts
+        self.routed_scaling_factor = routed_scaling_factor
+        self.first_k_dense_replace = first_k_dense_replace
+        self.norm_topk_prob = norm_topk_prob
+        self.use_qk_norm = use_qk_norm
+
+        self.use_rope = use_rope
+        self.gqa_interval = gqa_interval
+        self.gqa_layers = gqa_layers
+        self.use_gqa_gate = use_gqa_gate
+        self.use_gqa_gate_bias = use_gqa_gate_bias
+        self.linear_attn_config = linear_attn_config
+        self.kda_use_full_proj = kda_use_full_proj
+        self.kda_gate_lower_bound = kda_gate_lower_bound
+        self.kda_allow_neg_eigval = kda_allow_neg_eigval
+
+        # Hyper-Connections
+        self.hc_rate = hc_rate
+        self.hc_dynamic = hc_dynamic
+        self.hc_use_tanh = hc_use_tanh
+        self.hc_use_sigmoid = hc_use_sigmoid
+
+        super().__init__(
+            tie_word_embeddings=tie_word_embeddings,
+            **kwargs,
+        )
+
+
+__all__ = ["SolarOpen2Config"]

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "bos_token_id": 1,
+  "do_sample": true,
+  "eos_token_id": [
+    2,
+    129
+  ],
+  "transformers_version": "4.57.6"
+}

model-00001-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:07451dd18310558e454fd8a2a95f372988949a1f772999ae6c84d45fca40f36a
+size 5368311368

model-00002-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3e322b9b0aa7062ac8a0adcb22cc6f594a067de4983bb4f09868136b5879c3f5
+size 5368311368

model-00003-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:46a1f906378db65a9ac118fbc563ef7f7141e2c98e868ad177ac3aa76b198c38
+size 5368311360

model-00004-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c552e6a90fd3cbb6fba0f30a5d5a408759cbd48bcd3ad340f0191edc945b1fba
+size 5368311368

model-00005-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a0f6b9bf2fb07d40fb78ad548ccf1334c86b5ba9ece9b6dd0593fd757644bd77
+size 5368311368

model-00006-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9f59879e36f39c223e4e302b022871e69ecaca4000577e168cce2af056810761
+size 5368316640

model-00007-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9a46ea6deffa229406b849e6cb8e1643085ae2549c89db7a8fffd0a922f58338
+size 5368318808

model-00008-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:932abd6d70d4a472607d2e09f4ff01c7ee2e433768003d92ce6ea49e3794fa40
+size 5368319048

model-00009-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b6f5a208ec2c8a1ff1b09a34df87d5d9ab87dcf0f83f709328beb2ab762740bd
+size 5368319040

model-00010-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7e78368b30ae9af72626c77c58c2df004ca3ea8c9fbb2a7897d51ec245d51c19
+size 5368319008

model-00011-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:99092cfe8b0ddc87a4ca50b1e500c0f1309da36e14b706e28d078582aafe9616
+size 5368318648

model-00012-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e4d9bcc69e3cc34d8727797e8e0abb008f9310a9c3c0fd20c36e2250377b35fc
+size 5368318640

model-00013-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d0226edf43d86d469bd689db619b6f8cff262ae3e77f581f7aba3edf3c95edc4
+size 5368318648

model-00014-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ef25a02afeb25a3aa6413939319c51989408f1cb477b106e57cb4b827f665380
+size 5368318648

model-00015-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b0eb5ecc64a3d497560c9784d7b4667f9ac089c58ca46497ec158bb066ccaafc
+size 5368318640

model-00016-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e1a4cfbfcd30d0f8a5dc7cfd29351558ad188b0024c035772e794817c85df964
+size 5368318648

model-00017-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9f578f125e93429df68acc53fc931af91100d49d0163c7bd8566c999023de4f
+size 5368318968

model-00018-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc2ba42a7a0fa27fffddfc08a7b07ca787949edfcc1ca2124412cedfe0c6c1b1
+size 5368319040

model-00019-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a3fd2d3e6df49c89699fe9a7c9146f2e9c0a8e3a0b22803d8c75e12ae16b0a5c
+size 5368319088

model-00020-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:517121b64ad5bf843e4b95b295e3c8a26d8d35a1202dcda6807a2eb862edfe85
+size 5368318808

model-00021-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5fd6d253c10c1a26c39023846895444c9748f560e4218d40065ed5f9bcde0ddc
+size 5368318640

model-00022-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:92f8c0558fca0b159b262f6874322d64ecd8c215d825b426005227c6df2e561c
+size 5368318648

model-00023-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2e07d47d153cd055e31bef47e9a916821f8e154698b1a712151a6101e2f58534
+size 5368318648

model-00024-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:09c2e20dfdc66bb32a6b648e252abcf554d1cf9f0f4b2c5c40c02fb6c8f3895a
+size 5368318640

model-00025-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:570e620c72e0d8c1dc856e50c22212041fd67c28346e02942e9901a44d9cf2d2
+size 5368318648

model-00026-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b2e5efa2f4b70a4d8cbc86c5e5498d56dcb5ea2c725e287cf2e9c551c108bf1
+size 5363202928

model-00027-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:39ec1dcb38e75fb7fffd077e2c9dc936dae58e14342bfd01ac992974c6827702
+size 5363193184

model-00028-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3cf52b9234d0597376a9f964f119c682690a4ea763f415ecadb5c383141e2847
+size 5342926944

model-00029-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:868c4b3343be02265644f226d7065a9176253cac0cf1b71d51c642895d2f45bd
+size 3034000688

model.safetensors.index.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ada21368497b36a82031d0674f99475f73cb025c7d2f3e5b7a31e5c08755581f
+size 18563709

modeling_solar_open2.py

@@ -0,0 +1,1083 @@
+# coding=utf-8
+# Copyright 2025 Upstage AI.
+# Copyright 2025 The GLM4 & ZhipuAI team and HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# This file has been modified by Upstage AI including:
+# - Hybrid MoE Architecture: Replaced the standard dense structure with a depth-dependent Hybrid MoE, adding `SolarOpen2MoE` and `SolarOpen2TopkRouter` classes.
+# - RoPE Strategy: Changed the rotary position embedding strategy from GLM4's interleaved rotation to Llama-style block rotation (via modified `rotate_half`).
+# - Normalization Logic: Simplified the layer normalization structure by removing GLM4's extra post-operation norms and adding optional Query-Key Normalization (`use_qk_norm`).
+#
+# Based on code from: https://github.com/huggingface/transformers/blob/main/src/transformers/models/glm4/modeling_glm4.py
+
+from typing import Callable, Optional, Union, Any
+
+import torch
+import torch.nn.functional as F
+from torch import nn
+from einops import rearrange
+
+from transformers.activations import ACT2FN
+from transformers.cache_utils import Cache, DynamicCache
+from transformers.generation import GenerationMixin
+from transformers.integrations import use_kernel_forward_from_hub
+from transformers.masking_utils import create_causal_mask
+from transformers.modeling_flash_attention_utils import FlashAttentionKwargs
+from transformers.modeling_layers import GradientCheckpointingLayer
+from transformers.modeling_outputs import BaseModelOutputWithPast, CausalLMOutputWithPast
+from transformers.modeling_rope_utils import ROPE_INIT_FUNCTIONS, dynamic_rope_update
+from transformers.modeling_utils import ALL_ATTENTION_FUNCTIONS, PreTrainedModel
+from transformers.processing_utils import Unpack
+from transformers.utils import TransformersKwargs, auto_docstring, can_return_tuple
+from transformers.utils.deprecation import deprecate_kwarg
+from transformers.utils.generic import check_model_inputs
+from .configuration_solar_open2 import SolarOpen2Config
+
+try:
+    from fla.modules import FusedRMSNormGated, ShortConvolution
+    from fla.ops.kda import chunk_kda, fused_recurrent_kda
+    from fla.ops.kda.gate import fused_kda_gate
+    from fla.layers.utils import get_unpad_data, index_first_axis, pad_input
+except ImportError:
+    raise ImportError("Plese run `pip install -U fla-core`")
+
+def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
+    """
+    This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
+    num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
+    """
+    batch, num_key_value_heads, slen, head_dim = hidden_states.shape
+    if n_rep == 1:
+        return hidden_states
+    hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)
+    return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
+
+
+def repeat_kv_linear(x: torch.Tensor, n_rep: int) -> torch.Tensor:
+    """torch.repeat_interleave(x, dim=-2, repeats=n_rep) for (..., n_kv_heads, head_dim) layout."""
+    if n_rep == 1:
+        return x
+    shape = x.shape
+    return (
+        x.unsqueeze(-2)
+        .expand(*shape[:-1], n_rep, shape[-1])
+        .reshape(*shape[:-2], shape[-2] * n_rep, shape[-1])
+    )
+
+
+def eager_attention_forward(
+    module: nn.Module,
+    query: torch.Tensor,
+    key: torch.Tensor,
+    value: torch.Tensor,
+    attention_mask: Optional[torch.Tensor],
+    scaling: float,
+    dropout: float = 0.0,
+    **kwargs: Unpack[TransformersKwargs],
+):
+    key_states = repeat_kv(key, module.num_key_value_groups)
+    value_states = repeat_kv(value, module.num_key_value_groups)
+
+    attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling
+    if attention_mask is not None:
+        causal_mask = attention_mask[:, :, :, : key_states.shape[-2]]
+        attn_weights = attn_weights + causal_mask
+
+    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
+    attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)
+    attn_output = torch.matmul(attn_weights, value_states)
+    attn_output = attn_output.transpose(1, 2).contiguous()
+
+    return attn_output, attn_weights
+
+
+def rotate_half(x):
+    """Rotates half the hidden dims of the input."""
+    x1 = x[..., : x.shape[-1] // 2]
+    x2 = x[..., x.shape[-1] // 2 :]
+    return torch.cat((-x2, x1), dim=-1)
+
+
+def apply_rotary_pos_emb(q, k, cos, sin, position_ids=None, unsqueeze_dim=1):
+    """Applies Rotary Position Embedding to the query and key tensors.
+
+    Args:
+        q (`torch.Tensor`): The query tensor.
+        k (`torch.Tensor`): The key tensor.
+        cos (`torch.Tensor`): The cosine part of the rotary embedding.
+        sin (`torch.Tensor`): The sine part of the rotary embedding.
+        position_ids (`torch.Tensor`, *optional*):
+            Deprecated and unused.
+        unsqueeze_dim (`int`, *optional*, defaults to 1):
+            The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and
+            sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note
+            that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and
+            k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes
+            cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have
+            the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2.
+    Returns:
+        `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding.
+    """
+    cos = cos.unsqueeze(unsqueeze_dim)
+    sin = sin.unsqueeze(unsqueeze_dim)
+
+    # Keep half or full tensor for later concatenation
+    rotary_dim = cos.shape[-1]
+    q_rot, q_pass = q[..., :rotary_dim], q[..., rotary_dim:]
+    k_rot, k_pass = k[..., :rotary_dim], k[..., rotary_dim:]
+
+    # Apply rotary embeddings on the first half or full tensor
+    q_embed = (q_rot * cos) + (rotate_half(q_rot) * sin)
+    k_embed = (k_rot * cos) + (rotate_half(k_rot) * sin)
+
+    # Concatenate back to full shape
+    q_embed = torch.cat([q_embed, q_pass], dim=-1)
+    k_embed = torch.cat([k_embed, k_pass], dim=-1)
+    return q_embed, k_embed
+
+
+class SolarOpen2DynamicCache:
+    """
+    Dynamic cache for SolarOpen2 model.
+    Inspired by Kimi-Linear
+    """
+    is_compileable = False
+
+    def __init__(self, config: SolarOpen2Config):
+        super().__init__()
+        self.config = config
+
+        if config.linear_attn_config is not None:
+            self.layer_types = []
+            for i in range(config.num_hidden_layers):
+                # Determine if this layer uses GQA: gqa_layers takes priority over gqa_interval
+                if config.gqa_layers is not None:
+                    use_gqa = i in config.gqa_layers
+                else:
+                    use_gqa = (i + 1) % config.gqa_interval == 0
+                if use_gqa:
+                    self.layer_types.append("full_attention")
+                else:
+                    self.layer_types.append("linear_attention")
+        else:
+            self.layer_types = ["full_attention"] * config.num_hidden_layers
+
+        self.transformer_layers = [
+            i for i in range(config.num_hidden_layers) if self.layer_types[i] == "full_attention"
+        ]
+
+        linear_layers = [i for i in range(
+            config.num_hidden_layers) if self.layer_types[i] == "linear_attention"]
+        self.last_linear_layer = linear_layers[-1] if linear_layers else -1
+
+        self.conv_states = [None for _ in range(config.num_hidden_layers)]
+        self.recurrent_states = [None for _ in range(config.num_hidden_layers)]
+        self.key_cache = [None for _ in range(config.num_hidden_layers)]
+        self.value_cache = [None for _ in range(config.num_hidden_layers)]
+
+    def __len__(self):
+        return len(self.layer_types)
+
+    def update(
+        self,
+        key_states: torch.Tensor,
+        value_states: torch.Tensor,
+        layer_idx: int,
+        cache_kwargs: dict[str, Any] | None = None,
+    ) -> tuple[torch.Tensor, torch.Tensor]:
+        if self.key_cache[layer_idx] is None:
+            self.key_cache[layer_idx] = key_states
+            self.value_cache[layer_idx] = value_states
+        else:
+            self.key_cache[layer_idx] = torch.cat(
+                [self.key_cache[layer_idx], key_states], dim=2)
+            self.value_cache[layer_idx] = torch.cat(
+                [self.value_cache[layer_idx], value_states], dim=2)
+
+        return self.key_cache[layer_idx], self.value_cache[layer_idx]
+
+    def reorder_cache(self, beam_idx: torch.LongTensor):
+        """Reorders the cache for beam search, given the selected beam indices."""
+        for layer_idx in range(len(self.key_cache)):
+            if self.key_cache[layer_idx] is not None:
+                device = self.key_cache[layer_idx].device
+                beam_idx = beam_idx.to(device)
+                self.key_cache[layer_idx] = self.key_cache[layer_idx].index_select(
+                    0, beam_idx)
+                self.value_cache[layer_idx] = self.value_cache[layer_idx].index_select(
+                    0, beam_idx)
+
+            if self.conv_states[layer_idx] is not None:
+                device = self.conv_states[layer_idx][0].device
+                beam_idx = beam_idx.to(device)
+                q_conv, k_conv, v_conv = self.conv_states[layer_idx]
+                self.conv_states[layer_idx] = (
+                    q_conv.index_select(0, beam_idx),
+                    k_conv.index_select(0, beam_idx),
+                    v_conv.index_select(0, beam_idx),
+                )
+                self.recurrent_states[layer_idx] = self.recurrent_states[layer_idx].index_select(
+                    0, beam_idx)
+
+    def get_seq_length(self, layer_idx: int | None = 0) -> int:
+        """Returns the sequence length of the cached states. A layer index can be optionally passed."""
+        # take any layer that contains cache and not empty tensor
+        layer_idx = self.transformer_layers[0] if layer_idx not in self.transformer_layers else layer_idx
+        if len(self.key_cache) <= layer_idx or self.key_cache[layer_idx] is None:
+            return 0
+        return self.key_cache[layer_idx].shape[-2]
+
+    def get_mask_sizes(self, cache_position: torch.Tensor, layer_idx: int) -> tuple[int, int]:
+        """
+        Return a tuple (kv_length, kv_offset) corresponding to the length and offset that will be returned for
+        the given layer at `layer_idx`.
+        The masks are then prepared according to the given lengths (kv_length, kv_offset) and patterns for each layer.
+        """
+        kv_offset = 0
+        query_length = cache_position.shape[0]
+        past_seen_tokens = self.get_seq_length(layer_idx)
+        kv_length = query_length + past_seen_tokens
+        return kv_length, kv_offset
+
+    @property
+    def has_previous_state(self):
+        """We have a previous state if the last linear (conv) layer was already updated."""
+        if self.last_linear_layer == -1:
+            return False
+        return self.conv_states[self.last_linear_layer] is not None
+
+
+class SolarOpen2LinearAttention(nn.Module):
+    def __init__(self, config: SolarOpen2Config, layer_idx: int):
+        super().__init__()
+        self.config = config
+        self.mode = "chunk"
+
+        self.hidden_size = config.hidden_size
+        self.conv_size = config.linear_attn_config["short_conv_kernel_size"]
+        self.head_dim = config.linear_attn_config["head_dim"]
+        self.num_heads = config.linear_attn_config["num_heads"]
+        self.num_kv_heads = config.linear_attn_config.get("num_kv_heads", None) or self.num_heads
+        self.n_rep = self.num_heads // self.num_kv_heads
+        self.head_k_dim = self.head_dim
+        self.num_k_heads = self.num_heads
+
+        self.layer_idx = layer_idx
+        self.use_full_proj = getattr(config, 'kda_use_full_proj', False)
+        self.gate_lower_bound = getattr(config, 'kda_gate_lower_bound', -5.0)
+        self.allow_neg_eigval = getattr(config, 'kda_allow_neg_eigval', False)
+
+        assert self.mode in [
+            'chunk', 'fused_recurrent'], f"Not suppoerted mode `{self.mode}`."
+
+        projection_k_size = self.head_k_dim * self.num_k_heads
+        projection_size = self.head_dim * self.num_heads
+        kv_projection_size = self.head_dim * self.num_kv_heads
+
+        self.q_proj = nn.Linear(
+            self.hidden_size, projection_k_size, bias=False)
+        self.k_proj = nn.Linear(
+            self.hidden_size, kv_projection_size, bias=False)
+        self.v_proj = nn.Linear(self.hidden_size, kv_projection_size, bias=False)
+
+        self.q_conv1d = ShortConvolution(
+            hidden_size=projection_k_size,
+            kernel_size=self.conv_size,
+            activation='silu',
+        )
+        self.k_conv1d = ShortConvolution(
+            hidden_size=kv_projection_size,
+            kernel_size=self.conv_size,
+            activation='silu',
+        )
+        self.v_conv1d = ShortConvolution(
+            hidden_size=kv_projection_size,
+            kernel_size=self.conv_size,
+            activation='silu',
+        )
+
+        self.A_log = torch.nn.Parameter(torch.log(torch.empty(
+            self.num_heads, dtype=torch.float32).uniform_(1, 16)).view(1, 1, -1, 1))
+
+        if self.use_full_proj:
+            self.f_proj = nn.Linear(self.hidden_size, projection_size, bias=False)
+        else:
+            self.f_a_proj = nn.Linear(self.hidden_size, self.head_dim, bias=False)
+            self.f_b_proj = nn.Linear(self.head_dim, projection_size, bias=False)
+
+        self.dt_bias = nn.Parameter(
+            torch.empty(projection_size, dtype=torch.float32))
+
+        self.b_proj = nn.Linear(self.hidden_size, self.num_heads, bias=False)
+
+        if self.use_full_proj:
+            self.g_proj = nn.Linear(self.hidden_size, projection_size, bias=False)
+        else:
+            self.g_a_proj = nn.Linear(self.hidden_size, self.head_dim, bias=False)
+            self.g_b_proj = nn.Linear(self.head_dim, projection_size, bias=False)
+
+        self.o_norm = FusedRMSNormGated(
+            self.head_dim, eps=config.rms_norm_eps, activation='sigmoid')
+        self.o_proj = nn.Linear(projection_size, self.hidden_size, bias=False)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: torch.Tensor | None = None,
+        past_key_values: SolarOpen2DynamicCache | None = None,
+        **kwargs: Unpack[dict],
+    ) -> tuple[torch.Tensor, torch.Tensor | None, Cache | None]:
+        if attention_mask is not None:
+            if attention_mask.dim() != 2:
+                attention_mask = kwargs.get("padding_mask")
+
+            if attention_mask is not None and attention_mask.dim() != 2:
+                raise ValueError(
+                    "attention_mask must be a 0-1 matrix of shape [batch_size, seq_len] "
+                    "(0 = padding). 3D masks are not supported here.",
+                )
+        use_cache = past_key_values is not None
+        batch_size, q_len, _ = hidden_states.shape
+        mode = 'fused_recurrent' if q_len <= 64 else self.mode
+        if self.training:
+            assert mode == 'chunk', "Only chunk mode is supported in training."
+
+        cu_seqlens = kwargs.get('cu_seqlens')
+        indices = None
+        if attention_mask is not None:
+            indices, cu_seqlens, _ = get_unpad_data(attention_mask[:, -q_len:])
+            hidden_states = index_first_axis(
+                rearrange(hidden_states, "b s ... -> (b s) ..."), indices).unsqueeze(0)
+
+        conv_state_q, conv_state_k, conv_state_v = None, None, None
+        recurrent_state = None
+        if past_key_values is not None:
+            if past_key_values.conv_states[self.layer_idx] is not None:
+                conv_state_q, conv_state_k, conv_state_v = past_key_values.conv_states[
+                    self.layer_idx]
+            recurrent_state = past_key_values.recurrent_states[self.layer_idx]
+        q, conv_state_q = self.q_conv1d(
+            x=self.q_proj(hidden_states),
+            cache=conv_state_q,
+            output_final_state=use_cache,
+            cu_seqlens=cu_seqlens,
+        )
+        k, conv_state_k = self.k_conv1d(
+            x=self.k_proj(hidden_states),
+            cache=conv_state_k,
+            output_final_state=use_cache,
+            cu_seqlens=cu_seqlens,
+        )
+        v, conv_state_v = self.v_conv1d(
+            x=self.v_proj(hidden_states),
+            cache=conv_state_v,
+            output_final_state=use_cache,
+            cu_seqlens=cu_seqlens,
+        )
+        if self.use_full_proj:
+            g = self.f_proj(hidden_states)
+        else:
+            g = self.f_b_proj(self.f_a_proj(hidden_states))
+        q, k, g = (rearrange(x, "... (h d) -> ... h d", d=self.head_k_dim) for x in (q, k, g))
+        v = rearrange(v, '... (h d) -> ... h d', d=self.head_dim)
+
+        # repeat k/v heads if num_kv_heads < num_heads (GQA)
+        k = repeat_kv_linear(k, self.n_rep)
+        v = repeat_kv_linear(v, self.n_rep)
+
+        g = fused_kda_gate(g, self.A_log, dt_bias=self.dt_bias, lower_bound=self.gate_lower_bound)
+        beta = self.b_proj(hidden_states).sigmoid()
+        if self.allow_neg_eigval:
+            beta = beta * 2.0
+
+        if mode == 'chunk':
+            o, recurrent_state = chunk_kda(
+                q=q,
+                k=k,
+                v=v,
+                g=g,
+                beta=beta,
+                initial_state=recurrent_state,
+                output_final_state=True,
+                use_qk_l2norm_in_kernel=True,
+                cu_seqlens=cu_seqlens,
+            )
+        else:
+            o, recurrent_state = fused_recurrent_kda(
+                q=q,
+                k=k,
+                v=v,
+                g=g,
+                beta=beta,
+                initial_state=recurrent_state,
+                output_final_state=True,
+                use_qk_l2norm_in_kernel=True,
+                cu_seqlens=cu_seqlens,
+            )
+        if past_key_values is not None:
+            past_key_values.recurrent_states[self.layer_idx] = recurrent_state
+            past_key_values.conv_states[self.layer_idx] = (
+                conv_state_q, conv_state_k, conv_state_v)
+
+        if self.use_full_proj:
+            g = self.g_proj(hidden_states)
+        else:
+            g = self.g_b_proj(self.g_a_proj(hidden_states))
+        g = rearrange(g, '... (h d) -> ... h d', d=self.head_dim)
+        o = self.o_norm(o, g)
+
+        o = rearrange(o, 'b t h d -> b t (h d)')
+        o = self.o_proj(o)
+        if attention_mask is not None:
+            o = pad_input(o.squeeze(0), indices, batch_size, q_len)
+
+        return o, _
+
+
+class SolarOpen2Attention(nn.Module):
+    """Multi-headed attention from 'Attention Is All You Need' paper"""
+
+    def __init__(self, config: SolarOpen2Config, layer_idx: Optional[int] = None):
+        super().__init__()
+        self.config = config
+        self.layer_idx = layer_idx
+        self.head_dim = getattr(config, "head_dim", config.hidden_size // config.num_attention_heads)
+        self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
+        self.scaling = self.head_dim**-0.5
+        self.rope_scaling = config.rope_scaling
+        self.attention_dropout = config.attention_dropout
+        self.is_causal = True
+
+        self.q_proj = nn.Linear(
+            config.hidden_size, config.num_attention_heads * self.head_dim, bias=config.attention_bias
+        )
+        self.k_proj = nn.Linear(
+            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
+        )
+        self.v_proj = nn.Linear(
+            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
+        )
+        self.o_proj = nn.Linear(config.num_attention_heads * self.head_dim, config.hidden_size, bias=False)
+        self.use_qk_norm = config.use_qk_norm
+        if self.use_qk_norm:
+            self.q_norm = SolarOpen2RMSNorm(self.head_dim, eps=config.rms_norm_eps)
+            self.k_norm = SolarOpen2RMSNorm(self.head_dim, eps=config.rms_norm_eps)
+
+        self.use_gqa_gate = config.use_gqa_gate
+        if self.use_gqa_gate:
+            self.g_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=config.use_gqa_gate_bias)
+
+    @deprecate_kwarg("past_key_value", new_name="past_key_values", version="4.58")
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        position_embeddings: tuple[torch.Tensor, torch.Tensor],
+        attention_mask: Optional[torch.Tensor],
+        past_key_values: Optional[SolarOpen2DynamicCache] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        **kwargs: Unpack[FlashAttentionKwargs],
+    ) -> tuple[torch.Tensor, Optional[torch.Tensor]]:
+        input_shape = hidden_states.shape[:-1]
+        hidden_shape = (*input_shape, -1, self.head_dim)
+
+        query_states = self.q_proj(hidden_states).view(hidden_shape)
+        key_states = self.k_proj(hidden_states).view(hidden_shape)
+        value_states = self.v_proj(hidden_states).view(hidden_shape)
+
+        if self.use_qk_norm:  # main diff from Llama
+            query_states = self.q_norm(query_states)
+            key_states = self.k_norm(key_states)
+
+        query_states = query_states.transpose(1, 2)
+        key_states = key_states.transpose(1, 2)
+        value_states = value_states.transpose(1, 2)
+
+        cos, sin = position_embeddings
+        if self.config.use_rope:
+            query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
+
+        if past_key_values is not None:
+            # sin and cos are specific to RoPE models; position_ids needed for the static cache
+            cache_kwargs = {"sin": sin, "cos": cos, "cache_position": cache_position}
+            key_states, value_states = past_key_values.update(key_states, value_states, self.layer_idx, cache_kwargs)
+
+        attention_interface: Callable = eager_attention_forward
+        if self.config._attn_implementation != "eager":
+            attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]
+
+        attn_output, attn_weights = attention_interface(
+            self,
+            query_states,
+            key_states,
+            value_states,
+            attention_mask,
+            dropout=0.0 if not self.training else self.attention_dropout,
+            scaling=self.scaling,
+            **kwargs,
+        )
+
+        attn_output = attn_output.reshape(*input_shape, -1).contiguous()
+        if self.use_gqa_gate:
+            gate = self.g_proj(hidden_states)
+            attn_output = attn_output * torch.sigmoid(gate)
+
+        attn_output = self.o_proj(attn_output)
+        return attn_output, attn_weights
+
+
+class SolarOpen2MLP(nn.Module):
+    def __init__(self, config, hidden_size=None, intermediate_size=None):
+        super().__init__()
+        self.config = config
+        self.hidden_size = config.hidden_size if hidden_size is None else hidden_size
+        self.intermediate_size = config.intermediate_size if intermediate_size is None else intermediate_size
+
+        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
+        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
+        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
+        self.act_fn = ACT2FN[config.hidden_act]
+
+    def forward(self, x):
+        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
+        return down_proj
+
+
+class SolarOpen2TopkRouter(nn.Module):
+    def __init__(self, config: SolarOpen2Config):
+        super().__init__()
+        self.config = config
+        self.top_k = config.num_experts_per_tok
+        self.n_routed_experts = config.n_routed_experts
+        self.routed_scaling_factor = config.routed_scaling_factor
+        self.n_group = config.n_group
+        self.topk_group = config.topk_group
+        self.norm_topk_prob = config.norm_topk_prob
+
+        self.weight = nn.Parameter(torch.empty((self.n_routed_experts, config.hidden_size)))
+        self.e_score_correction_bias = nn.Parameter(
+            torch.zeros((self.n_routed_experts), dtype=torch.float32))
+
+    @torch.no_grad()
+    def get_topk_indices(self, scores):
+        scores_for_choice = scores.view(-1, self.n_routed_experts) + self.e_score_correction_bias.unsqueeze(0)
+        group_scores = (
+            scores_for_choice.view(-1, self.n_group, self.n_routed_experts // self.n_group)
+            .topk(2, dim=-1)[0]
+            .sum(dim=-1)
+        )
+        group_idx = torch.topk(group_scores, k=self.topk_group, dim=-1, sorted=False)[1]
+        group_mask = torch.zeros_like(group_scores)
+        group_mask.scatter_(1, group_idx, 1)
+        score_mask = (
+            group_mask.unsqueeze(-1)
+            .expand(-1, self.n_group, self.n_routed_experts // self.n_group)
+            .reshape(-1, self.n_routed_experts)
+        )
+        scores_for_choice = scores_for_choice.masked_fill(~score_mask.bool(), 0.0)
+        topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
+        return topk_indices
+
+    def forward(self, hidden_states):
+        hidden_states = hidden_states.view(-1, self.config.hidden_size)
+        router_logits = F.linear(hidden_states.type(torch.float32), self.weight.type(torch.float32))
+        scores = router_logits.sigmoid()
+        topk_indices = self.get_topk_indices(scores)
+        topk_weights = scores.gather(1, topk_indices)
+        if self.norm_topk_prob:
+            denominator = topk_weights.sum(dim=-1, keepdim=True) + 1e-20
+            topk_weights /= denominator
+        topk_weights = topk_weights * self.routed_scaling_factor
+        return topk_indices, topk_weights
+
+
+@use_kernel_forward_from_hub("RMSNorm")
+class SolarOpen2RMSNorm(nn.Module):
+    def __init__(self, hidden_size, eps=1e-6):
+        """
+        SolarOpen2RMSNorm is equivalent to T5LayerNorm
+        """
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.variance_epsilon = eps
+
+    def forward(self, hidden_states):
+        input_dtype = hidden_states.dtype
+        hidden_states = hidden_states.to(torch.float32)
+        variance = hidden_states.pow(2).mean(-1, keepdim=True)
+        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
+        return self.weight * hidden_states.to(input_dtype)
+
+    def extra_repr(self):
+        return f"{tuple(self.weight.shape)}, eps={self.variance_epsilon}"
+
+
+class SolarOpen2MoE(nn.Module):
+    """
+    A mixed expert module containing shared experts.
+    """
+
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.experts = nn.ModuleList(
+            [
+                SolarOpen2MLP(config, intermediate_size=config.moe_intermediate_size)
+                for _ in range(config.n_routed_experts)
+            ]
+        )
+        self.gate = SolarOpen2TopkRouter(config)
+        self.shared_experts = SolarOpen2MLP(
+            config=config, intermediate_size=config.moe_intermediate_size * config.n_shared_experts
+        )
+
+    @torch.compiler.disable()
+    def moe(self, hidden_states: torch.Tensor, topk_indices: torch.Tensor, topk_weights: torch.Tensor):
+        r"""
+        MoE forward pass that only executes selected experts.
+        Uses @torch.compiler.disable() to allow dynamic shape operations.
+        Requires --enforce-eager flag when serving with vLLM.
+        """
+        final_hidden_states = torch.zeros_like(hidden_states)
+
+        for expert_idx in range(len(self.experts)):
+            expert = self.experts[expert_idx]
+
+            # Find positions where this expert was selected
+            batch_idx, topk_pos = torch.where(topk_indices == expert_idx)
+
+            if batch_idx.numel() == 0:
+                continue
+
+            # Extract only the tokens routed to this expert
+            expert_input = hidden_states[batch_idx]
+            expert_output = expert(expert_input)
+
+            # Apply weights and accumulate results
+            weights = topk_weights[batch_idx, topk_pos].unsqueeze(-1)
+            final_hidden_states.index_add_(0, batch_idx, (expert_output * weights).to(hidden_states.dtype))
+
+        return final_hidden_states
+
+    def forward(self, hidden_states):
+        residuals = hidden_states
+        orig_shape = hidden_states.shape
+        topk_indices, topk_weights = self.gate(hidden_states)
+        hidden_states = hidden_states.view(-1, hidden_states.shape[-1])
+        hidden_states = self.moe(hidden_states, topk_indices, topk_weights).view(*orig_shape)
+        hidden_states = hidden_states + self.shared_experts(residuals)
+        return hidden_states
+
+
+class SolarOpen2HyperConnection(nn.Module):
+    """Hyper-Connections module for SolarOpen2 (pure PyTorch, no Triton).
+
+    Learnable replacement for residual connections that expands the hidden
+    state to N copies and uses learned mixing (width) and combining (depth)
+    connections.
+
+    Reference: https://arxiv.org/abs/2409.19606
+    """
+
+    def __init__(self, config: SolarOpen2Config, layer_idx: int):
+        super().__init__()
+        dim = config.hidden_size
+        rate = config.hc_rate
+        self.rate = rate
+        self.dynamic = config.hc_dynamic
+        self.use_tanh = config.hc_use_tanh
+        self.use_sigmoid = getattr(config, "hc_use_sigmoid", False)
+
+        self.static_alpha = nn.Parameter(torch.zeros(rate, rate + 1))
+        self.static_beta = nn.Parameter(torch.ones(rate))
+
+        if self.dynamic:
+            self.dynamic_alpha_fn = nn.Parameter(torch.zeros(dim, rate + 1))
+            self.dynamic_alpha_scale = nn.Parameter(torch.ones(1) * 0.01)
+            self.dynamic_beta_fn = nn.Parameter(torch.zeros(dim))
+            self.dynamic_beta_scale = nn.Parameter(torch.ones(1) * 0.01)
+
+            self.norm = SolarOpen2RMSNorm(dim, eps=config.rms_norm_eps)
+
+    def width_connection(self, h: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+        """Split h into (h_in, h_res, beta) via learned width mixing.
+
+        Args:
+            h: (..., N, D) hidden states with N copies.
+
+        Returns:
+            h_in: (..., D) input for the sublayer.
+            h_res: (..., N, D) residual for depth connection.
+            beta: (N,) or (..., N) depth connection weights.
+        """
+        if self.dynamic:
+            norm_h = self.norm(h)
+            # norm_h: (..., N, D), dynamic_alpha_fn: (D, rate+1)
+            wc_weight = norm_h @ self.dynamic_alpha_fn  # (..., N, rate+1)
+            if self.use_tanh:
+                wc_weight = torch.tanh(wc_weight)
+            alpha = wc_weight * self.dynamic_alpha_scale + self.static_alpha
+            # dc_weight: (..., N)
+            dc_weight = (norm_h * self.dynamic_beta_fn).sum(-1)
+            if self.use_tanh:
+                dc_weight = torch.tanh(dc_weight)
+            beta = dc_weight * self.dynamic_beta_scale + self.static_beta
+        else:
+            alpha = self.static_alpha
+            beta = self.static_beta
+
+        if self.use_sigmoid:
+            alpha = torch.sigmoid(alpha)
+            beta = 2.0 * torch.sigmoid(beta)
+
+        # Mix: alpha^T @ h → (..., N+1, D)  — float32 accumulators to match
+        # the Triton kernels in TorchTitan which compute in fp32 internally.
+        orig_dtype = h.dtype
+        alpha_t = alpha.float().transpose(-2, -1)  # (..., N+1, N) or (N+1, N)
+        mix_h = torch.matmul(alpha_t, h.float())  # (..., N+1, D)
+        h_in = mix_h[..., 0, :].to(orig_dtype)  # (..., D)
+        h_res = mix_h[..., 1:, :].to(orig_dtype)  # (..., N, D)
+        return h_in, h_res, beta
+
+    def depth_connection(self, h_res: torch.Tensor, h_out: torch.Tensor, beta: torch.Tensor) -> torch.Tensor:
+        """Combine sublayer output with residual via learned depth connection.
+
+        Args:
+            h_res: (..., N, D) residual from width connection.
+            h_out: (..., D) sublayer output.
+            beta: (N,) or (..., N) depth connection weights.
+
+        Returns:
+            h: (..., N, D) combined hidden states.
+        """
+        # float32 accumulators to match Triton kernels in TorchTitan.
+        return (h_out.float().unsqueeze(-2) * beta.float().unsqueeze(-1)
+                + h_res.float()).to(h_out.dtype)
+
+
+class SolarOpen2DecoderLayer(GradientCheckpointingLayer):
+    def __init__(self, config: SolarOpen2Config, layer_idx: int):
+        super().__init__()
+        self.hidden_size = config.hidden_size
+
+        # Hyper-Connections
+        self.use_hc = getattr(config, "hc_rate", 0) > 0
+        if self.use_hc:
+            self.attn_hc = SolarOpen2HyperConnection(config, layer_idx=2 * layer_idx)
+            self.ffn_hc = SolarOpen2HyperConnection(config, layer_idx=2 * layer_idx + 1)
+
+        # Determine if this layer uses GQA: gqa_layers takes priority over gqa_interval
+        if config.gqa_layers is not None:
+            use_gqa = layer_idx in config.gqa_layers
+        else:
+            use_gqa = (layer_idx + 1) % config.gqa_interval == 0
+
+        if use_gqa:
+            self.is_linear_attn = False
+            self.self_attn = SolarOpen2Attention(config=config, layer_idx=layer_idx)
+        else:
+            self.is_linear_attn = True
+            self.self_attn = SolarOpen2LinearAttention(config=config, layer_idx=layer_idx)
+
+        if layer_idx >= config.first_k_dense_replace:
+            self.mlp = SolarOpen2MoE(config)
+        else:
+            self.mlp = SolarOpen2MLP(config)
+
+        self.input_layernorm = SolarOpen2RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.post_attention_layernorm = SolarOpen2RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+
+    @deprecate_kwarg("past_key_value", new_name="past_key_values", version="4.58")
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[SolarOpen2DynamicCache] = None,
+        use_cache: Optional[bool] = False,
+        cache_position: Optional[torch.LongTensor] = None,
+        position_embeddings: Optional[tuple[torch.Tensor, torch.Tensor]] = None,  # necessary, but kept here for BC
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> torch.Tensor:
+        if self.use_hc:
+            # ═══ Attention with Hyper-Connections ═══
+            h_in, h_res, attn_beta = self.attn_hc.width_connection(hidden_states)
+            h_in = self.input_layernorm(h_in)
+            h_in, _ = self.self_attn(
+                hidden_states=h_in,
+                attention_mask=attention_mask,
+                position_ids=position_ids,
+                past_key_values=past_key_values,
+                use_cache=use_cache,
+                cache_position=cache_position,
+                position_embeddings=position_embeddings,
+                **kwargs,
+            )
+            hidden_states = self.attn_hc.depth_connection(h_res, h_in, attn_beta)
+
+            # ═══ FFN/MoE with Hyper-Connections ═══
+            ffn_h_in, ffn_h_res, ffn_beta = self.ffn_hc.width_connection(hidden_states)
+            ffn_h_in = self.post_attention_layernorm(ffn_h_in)
+            ffn_h_in = self.mlp(ffn_h_in)
+            hidden_states = self.ffn_hc.depth_connection(ffn_h_res, ffn_h_in, ffn_beta)
+            return hidden_states
+        else:
+            residual = hidden_states
+            hidden_states = self.input_layernorm(hidden_states)
+            # Self Attention
+            hidden_states, _ = self.self_attn(
+                hidden_states=hidden_states,
+                attention_mask=attention_mask,
+                position_ids=position_ids,
+                past_key_values=past_key_values,
+                use_cache=use_cache,
+                cache_position=cache_position,
+                position_embeddings=position_embeddings,
+                **kwargs,
+            )
+            hidden_states = residual + hidden_states
+
+            # Fully Connected
+            residual = hidden_states
+            hidden_states = self.post_attention_layernorm(hidden_states)
+            hidden_states = self.mlp(hidden_states)
+            hidden_states = residual + hidden_states
+            return hidden_states
+
+
+@auto_docstring
+class SolarOpen2PreTrainedModel(PreTrainedModel):
+    config: SolarOpen2Config
+    base_model_prefix = "model"
+    supports_gradient_checkpointing = True
+    _no_split_modules = ["SolarOpen2DecoderLayer"]
+    _skip_keys_device_placement = ["past_key_values"]
+    _supports_flash_attn = True
+    _supports_sdpa = True
+    _supports_flex_attn = True
+    _can_compile_fullgraph = False
+    _supports_attention_backend = True
+    _can_record_outputs = {
+        "hidden_states": SolarOpen2DecoderLayer,
+        "attentions": SolarOpen2Attention,
+    }
+
+    def _init_weights(self, module):
+        super()._init_weights(module)
+        if isinstance(module, SolarOpen2TopkRouter):
+            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+
+
+class SolarOpen2RotaryEmbedding(nn.Module):
+    inv_freq: torch.Tensor  # fix linting for `register_buffer`
+
+    def __init__(self, config: SolarOpen2Config, device=None):
+        super().__init__()
+        # BC: "rope_type" was originally "type"
+        if hasattr(config, "rope_scaling") and isinstance(config.rope_scaling, dict):
+            self.rope_type = config.rope_scaling.get("rope_type", config.rope_scaling.get("type"))
+        else:
+            self.rope_type = "default"
+        self.max_seq_len_cached = config.max_position_embeddings
+        self.original_max_seq_len = config.max_position_embeddings
+
+        self.config = config
+        self.rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]
+
+        inv_freq, self.attention_scaling = self.rope_init_fn(self.config, device)
+        self.register_buffer("inv_freq", inv_freq, persistent=False)
+        self.original_inv_freq = self.inv_freq
+
+    @torch.no_grad()
+    @dynamic_rope_update  # power user: used with advanced RoPE types (e.g. dynamic rope)
+    def forward(self, x, position_ids):
+        inv_freq_expanded = self.inv_freq[None, :, None].float().expand(position_ids.shape[0], -1, 1).to(x.device)
+        position_ids_expanded = position_ids[:, None, :].float()
+
+        device_type = x.device.type if isinstance(x.device.type, str) and x.device.type != "mps" else "cpu"
+        with torch.autocast(device_type=device_type, enabled=False):  # Force float32
+            freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)
+            emb = torch.cat((freqs, freqs), dim=-1)
+            cos = emb.cos() * self.attention_scaling
+            sin = emb.sin() * self.attention_scaling
+
+        return cos.to(dtype=x.dtype), sin.to(dtype=x.dtype)
+
+
+@auto_docstring
+class SolarOpen2Model(SolarOpen2PreTrainedModel):
+    _keys_to_ignore_on_load_unexpected = [r"model\.layers\.92.*", r"model\.layers\.46.*"]
+
+    def __init__(self, config: SolarOpen2Config):
+        super().__init__(config)
+        self.padding_idx = config.pad_token_id
+        self.vocab_size = config.vocab_size
+        self.hc_rate = getattr(config, "hc_rate", 0)
+
+        self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
+        self.layers = nn.ModuleList(
+            [SolarOpen2DecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
+        )
+        self.norm = SolarOpen2RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.rotary_emb = SolarOpen2RotaryEmbedding(config=config)
+        self.gradient_checkpointing = False
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def _update_linear_attn_mask(self, attention_mask, cache_position):
+        """
+        NOTE: Left-padding is used for linear attention mask.
+        No need for zeroing states when
+            1. Cached forward
+            2. Attending to all inputs
+        """
+        linear_attn_mask = attention_mask
+        if cache_position[0] > 0 or (attention_mask is not None and torch.all(attention_mask == 1)):
+            linear_attn_mask = None
+        return linear_attn_mask
+
+    @check_model_inputs
+    @auto_docstring
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[SolarOpen2DynamicCache] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> BaseModelOutputWithPast:
+        if (input_ids is None) ^ (inputs_embeds is not None):
+            raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
+
+        if inputs_embeds is None:
+            inputs_embeds: torch.Tensor = self.embed_tokens(input_ids)
+
+        if use_cache and not isinstance(past_key_values, SolarOpen2DynamicCache):
+            past_key_values = SolarOpen2DynamicCache(config=self.config)
+
+        if cache_position is None:
+            past_seen_tokens = past_key_values.get_seq_length() if past_key_values is not None else 0
+            cache_position: torch.Tensor = torch.arange(
+                past_seen_tokens, past_seen_tokens + inputs_embeds.shape[1], device=inputs_embeds.device
+            )
+
+        if position_ids is None:
+            position_ids = cache_position.unsqueeze(0)
+
+        causal_mask = create_causal_mask(
+            config=self.config,
+            input_embeds=inputs_embeds,
+            attention_mask=attention_mask,
+            cache_position=cache_position,
+            past_key_values=past_key_values,
+            position_ids=position_ids,
+        )
+        linear_attn_mask = self._update_linear_attn_mask(attention_mask, cache_position)
+
+        hidden_states = inputs_embeds
+        position_embeddings = self.rotary_emb(hidden_states, position_ids)
+
+        # Hyper-Connections: expand (B, L, D) → (B, L, N, D) by replicating N times
+        if self.hc_rate > 0:
+            hidden_states = hidden_states.unsqueeze(-2).expand(
+                *hidden_states.shape[:-1], self.hc_rate, hidden_states.shape[-1]
+            ).contiguous()
+
+        for decoder_layer in self.layers[: self.config.num_hidden_layers]:
+            layer_mask = linear_attn_mask if decoder_layer.is_linear_attn else causal_mask
+            hidden_states = decoder_layer(
+                hidden_states,
+                attention_mask=layer_mask,
+                position_ids=position_ids,
+                past_key_values=past_key_values,
+                cache_position=cache_position,
+                position_embeddings=position_embeddings,
+                **kwargs,
+            )
+
+        # Hyper-Connections: contract (B, L, N, D) → (B, L, D) by summing N copies
+        if self.hc_rate > 0:
+            hidden_states = hidden_states.sum(dim=-2)
+
+        hidden_states = self.norm(hidden_states)
+        return BaseModelOutputWithPast(
+            last_hidden_state=hidden_states,
+            past_key_values=past_key_values,
+        )
+
+
+@auto_docstring
+class SolarOpen2ForCausalLM(SolarOpen2PreTrainedModel, GenerationMixin):
+    _tied_weights_keys = ["lm_head.weight"]
+    _tp_plan = {"lm_head": "colwise_rep"}
+    _pp_plan = {"lm_head": (["hidden_states"], ["logits"])}
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.model = SolarOpen2Model(config)
+        self.vocab_size = config.vocab_size
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    @can_return_tuple
+    @auto_docstring
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[SolarOpen2DynamicCache] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        logits_to_keep: Union[int, torch.Tensor] = 0,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> CausalLMOutputWithPast:
+
+        outputs: BaseModelOutputWithPast = self.model(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            use_cache=use_cache,
+            cache_position=cache_position,
+            **kwargs,
+        )
+
+        hidden_states = outputs.last_hidden_state
+        # Only compute necessary logits, and do not upcast them to float if we are not computing the loss
+        slice_indices = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) else logits_to_keep
+        logits = self.lm_head(hidden_states[:, slice_indices, :])
+
+        loss = None
+        if labels is not None:
+            loss = self.loss_function(logits=logits, labels=labels, vocab_size=self.config.vocab_size, **kwargs)
+
+        return CausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+
+__all__ = ["SolarOpen2PreTrainedModel", "SolarOpen2Model", "SolarOpen2ForCausalLM"]

quantization_config.json

@@ -0,0 +1,577 @@
+{
+  "config_groups": {
+    "group_0": {
+      "targets": [
+        "Linear"
+      ],
+      "weights": {
+        "num_bits": 4,
+        "type": "float",
+        "symmetric": true,
+        "group_size": 16,
+        "strategy": "tensor_group",
+        "block_structure": null,
+        "dynamic": false,
+        "actorder": null,
+        "scale_dtype": "torch.float8_e4m3fn",
+        "zp_dtype": null,
+        "observer": "static_minmax",
+        "observer_kwargs": {}
+      },
+      "input_activations": {
+        "num_bits": 4,
+        "type": "float",
+        "symmetric": true,
+        "group_size": 16,
+        "strategy": "tensor_group",
+        "block_structure": null,
+        "dynamic": "local",
+        "actorder": null,
+        "scale_dtype": "torch.float8_e4m3fn",
+        "zp_dtype": null,
+        "observer": "static_minmax",
+        "observer_kwargs": {}
+      },
+      "output_activations": null,
+      "format": null
+    }
+  },
+  "quant_method": "compressed-tensors",
+  "kv_cache_scheme": null,
+  "format": "nvfp4-pack-quantized",
+  "quantization_status": "compressed",
+  "global_compression_ratio": null,
+  "ignore": [
+    "re:.*model\\.layers\\.[0-47]\\.mlp\\.gate.*",
+    "model.layers.0.self_attn.q_proj",
+    "model.layers.0.self_attn.k_proj",
+    "model.layers.0.self_attn.v_proj",
+    "model.layers.0.self_attn.o_proj",
+    "model.layers.0.self_attn.g_proj",
+    "model.layers.1.self_attn.q_proj",
+    "model.layers.1.self_attn.k_proj",
+    "model.layers.1.self_attn.v_proj",
+    "model.layers.1.self_attn.f_a_proj",
+    "model.layers.1.self_attn.f_b_proj",
+    "model.layers.1.self_attn.b_proj",
+    "model.layers.1.self_attn.g_a_proj",
+    "model.layers.1.self_attn.g_b_proj",
+    "model.layers.1.self_attn.o_proj",
+    "model.layers.2.self_attn.q_proj",
+    "model.layers.2.self_attn.k_proj",
+    "model.layers.2.self_attn.v_proj",
+    "model.layers.2.self_attn.f_a_proj",
+    "model.layers.2.self_attn.f_b_proj",
+    "model.layers.2.self_attn.b_proj",
+    "model.layers.2.self_attn.g_a_proj",
+    "model.layers.2.self_attn.g_b_proj",
+    "model.layers.2.self_attn.o_proj",
+    "model.layers.3.self_attn.q_proj",
+    "model.layers.3.self_attn.k_proj",
+    "model.layers.3.self_attn.v_proj",
+    "model.layers.3.self_attn.f_a_proj",
+    "model.layers.3.self_attn.f_b_proj",
+    "model.layers.3.self_attn.b_proj",
+    "model.layers.3.self_attn.g_a_proj",
+    "model.layers.3.self_attn.g_b_proj",
+    "model.layers.3.self_attn.o_proj",
+    "model.layers.4.self_attn.q_proj",
+    "model.layers.4.self_attn.k_proj",
+    "model.layers.4.self_attn.v_proj",
+    "model.layers.4.self_attn.o_proj",
+    "model.layers.4.self_attn.g_proj",
+    "model.layers.5.self_attn.q_proj",
+    "model.layers.5.self_attn.k_proj",
+    "model.layers.5.self_attn.v_proj",
+    "model.layers.5.self_attn.f_a_proj",
+    "model.layers.5.self_attn.f_b_proj",
+    "model.layers.5.self_attn.b_proj",
+    "model.layers.5.self_attn.g_a_proj",
+    "model.layers.5.self_attn.g_b_proj",
+    "model.layers.5.self_attn.o_proj",
+    "model.layers.6.self_attn.q_proj",
+    "model.layers.6.self_attn.k_proj",
+    "model.layers.6.self_attn.v_proj",
+    "model.layers.6.self_attn.f_a_proj",
+    "model.layers.6.self_attn.f_b_proj",
+    "model.layers.6.self_attn.b_proj",
+    "model.layers.6.self_attn.g_a_proj",
+    "model.layers.6.self_attn.g_b_proj",
+    "model.layers.6.self_attn.o_proj",
+    "model.layers.7.self_attn.q_proj",
+    "model.layers.7.self_attn.k_proj",
+    "model.layers.7.self_attn.v_proj",
+    "model.layers.7.self_attn.f_a_proj",
+    "model.layers.7.self_attn.f_b_proj",
+    "model.layers.7.self_attn.b_proj",
+    "model.layers.7.self_attn.g_a_proj",
+    "model.layers.7.self_attn.g_b_proj",
+    "model.layers.7.self_attn.o_proj",
+    "model.layers.8.self_attn.q_proj",
+    "model.layers.8.self_attn.k_proj",
+    "model.layers.8.self_attn.v_proj",
+    "model.layers.8.self_attn.o_proj",
+    "model.layers.8.self_attn.g_proj",
+    "model.layers.9.self_attn.q_proj",
+    "model.layers.9.self_attn.k_proj",
+    "model.layers.9.self_attn.v_proj",
+    "model.layers.9.self_attn.f_a_proj",
+    "model.layers.9.self_attn.f_b_proj",
+    "model.layers.9.self_attn.b_proj",
+    "model.layers.9.self_attn.g_a_proj",
+    "model.layers.9.self_attn.g_b_proj",
+    "model.layers.9.self_attn.o_proj",
+    "model.layers.10.self_attn.q_proj",
+    "model.layers.10.self_attn.k_proj",
+    "model.layers.10.self_attn.v_proj",
+    "model.layers.10.self_attn.f_a_proj",
+    "model.layers.10.self_attn.f_b_proj",
+    "model.layers.10.self_attn.b_proj",
+    "model.layers.10.self_attn.g_a_proj",
+    "model.layers.10.self_attn.g_b_proj",
+    "model.layers.10.self_attn.o_proj",
+    "model.layers.11.self_attn.q_proj",
+    "model.layers.11.self_attn.k_proj",
+    "model.layers.11.self_attn.v_proj",
+    "model.layers.11.self_attn.f_a_proj",
+    "model.layers.11.self_attn.f_b_proj",
+    "model.layers.11.self_attn.b_proj",
+    "model.layers.11.self_attn.g_a_proj",
+    "model.layers.11.self_attn.g_b_proj",
+    "model.layers.11.self_attn.o_proj",
+    "model.layers.12.self_attn.q_proj",
+    "model.layers.12.self_attn.k_proj",
+    "model.layers.12.self_attn.v_proj",
+    "model.layers.12.self_attn.o_proj",
+    "model.layers.12.self_attn.g_proj",
+    "model.layers.13.self_attn.q_proj",
+    "model.layers.13.self_attn.k_proj",
+    "model.layers.13.self_attn.v_proj",
+    "model.layers.13.self_attn.f_a_proj",
+    "model.layers.13.self_attn.f_b_proj",
+    "model.layers.13.self_attn.b_proj",
+    "model.layers.13.self_attn.g_a_proj",
+    "model.layers.13.self_attn.g_b_proj",
+    "model.layers.13.self_attn.o_proj",
+    "model.layers.14.self_attn.q_proj",
+    "model.layers.14.self_attn.k_proj",
+    "model.layers.14.self_attn.v_proj",
+    "model.layers.14.self_attn.f_a_proj",
+    "model.layers.14.self_attn.f_b_proj",
+    "model.layers.14.self_attn.b_proj",
+    "model.layers.14.self_attn.g_a_proj",
+    "model.layers.14.self_attn.g_b_proj",
+    "model.layers.14.self_attn.o_proj",
+    "model.layers.15.self_attn.q_proj",
+    "model.layers.15.self_attn.k_proj",
+    "model.layers.15.self_attn.v_proj",
+    "model.layers.15.self_attn.f_a_proj",
+    "model.layers.15.self_attn.f_b_proj",
+    "model.layers.15.self_attn.b_proj",
+    "model.layers.15.self_attn.g_a_proj",
+    "model.layers.15.self_attn.g_b_proj",
+    "model.layers.15.self_attn.o_proj",
+    "model.layers.16.self_attn.q_proj",
+    "model.layers.16.self_attn.k_proj",
+    "model.layers.16.self_attn.v_proj",
+    "model.layers.16.self_attn.o_proj",
+    "model.layers.16.self_attn.g_proj",
+    "model.layers.17.self_attn.q_proj",
+    "model.layers.17.self_attn.k_proj",
+    "model.layers.17.self_attn.v_proj",
+    "model.layers.17.self_attn.f_a_proj",
+    "model.layers.17.self_attn.f_b_proj",
+    "model.layers.17.self_attn.b_proj",
+    "model.layers.17.self_attn.g_a_proj",
+    "model.layers.17.self_attn.g_b_proj",
+    "model.layers.17.self_attn.o_proj",
+    "model.layers.18.self_attn.q_proj",
+    "model.layers.18.self_attn.k_proj",
+    "model.layers.18.self_attn.v_proj",
+    "model.layers.18.self_attn.f_a_proj",
+    "model.layers.18.self_attn.f_b_proj",
+    "model.layers.18.self_attn.b_proj",
+    "model.layers.18.self_attn.g_a_proj",
+    "model.layers.18.self_attn.g_b_proj",
+    "model.layers.18.self_attn.o_proj",
+    "model.layers.19.self_attn.q_proj",
+    "model.layers.19.self_attn.k_proj",
+    "model.layers.19.self_attn.v_proj",
+    "model.layers.19.self_attn.f_a_proj",
+    "model.layers.19.self_attn.f_b_proj",
+    "model.layers.19.self_attn.b_proj",
+    "model.layers.19.self_attn.g_a_proj",
+    "model.layers.19.self_attn.g_b_proj",
+    "model.layers.19.self_attn.o_proj",
+    "model.layers.20.self_attn.q_proj",
+    "model.layers.20.self_attn.k_proj",
+    "model.layers.20.self_attn.v_proj",
+    "model.layers.20.self_attn.o_proj",
+    "model.layers.20.self_attn.g_proj",
+    "model.layers.21.self_attn.q_proj",
+    "model.layers.21.self_attn.k_proj",
+    "model.layers.21.self_attn.v_proj",
+    "model.layers.21.self_attn.f_a_proj",
+    "model.layers.21.self_attn.f_b_proj",
+    "model.layers.21.self_attn.b_proj",
+    "model.layers.21.self_attn.g_a_proj",
+    "model.layers.21.self_attn.g_b_proj",
+    "model.layers.21.self_attn.o_proj",
+    "model.layers.22.self_attn.q_proj",
+    "model.layers.22.self_attn.k_proj",
+    "model.layers.22.self_attn.v_proj",
+    "model.layers.22.self_attn.f_a_proj",
+    "model.layers.22.self_attn.f_b_proj",
+    "model.layers.22.self_attn.b_proj",
+    "model.layers.22.self_attn.g_a_proj",
+    "model.layers.22.self_attn.g_b_proj",
+    "model.layers.22.self_attn.o_proj",
+    "model.layers.23.self_attn.q_proj",
+    "model.layers.23.self_attn.k_proj",
+    "model.layers.23.self_attn.v_proj",
+    "model.layers.23.self_attn.f_a_proj",
+    "model.layers.23.self_attn.f_b_proj",
+    "model.layers.23.self_attn.b_proj",
+    "model.layers.23.self_attn.g_a_proj",
+    "model.layers.23.self_attn.g_b_proj",
+    "model.layers.23.self_attn.o_proj",
+    "model.layers.24.self_attn.q_proj",
+    "model.layers.24.self_attn.k_proj",
+    "model.layers.24.self_attn.v_proj",
+    "model.layers.24.self_attn.o_proj",
+    "model.layers.24.self_attn.g_proj",
+    "model.layers.25.self_attn.q_proj",
+    "model.layers.25.self_attn.k_proj",
+    "model.layers.25.self_attn.v_proj",
+    "model.layers.25.self_attn.f_a_proj",
+    "model.layers.25.self_attn.f_b_proj",
+    "model.layers.25.self_attn.b_proj",
+    "model.layers.25.self_attn.g_a_proj",
+    "model.layers.25.self_attn.g_b_proj",
+    "model.layers.25.self_attn.o_proj",
+    "model.layers.26.self_attn.q_proj",
+    "model.layers.26.self_attn.k_proj",
+    "model.layers.26.self_attn.v_proj",
+    "model.layers.26.self_attn.f_a_proj",
+    "model.layers.26.self_attn.f_b_proj",
+    "model.layers.26.self_attn.b_proj",
+    "model.layers.26.self_attn.g_a_proj",
+    "model.layers.26.self_attn.g_b_proj",
+    "model.layers.26.self_attn.o_proj",
+    "model.layers.27.self_attn.q_proj",
+    "model.layers.27.self_attn.k_proj",
+    "model.layers.27.self_attn.v_proj",
+    "model.layers.27.self_attn.f_a_proj",
+    "model.layers.27.self_attn.f_b_proj",
+    "model.layers.27.self_attn.b_proj",
+    "model.layers.27.self_attn.g_a_proj",
+    "model.layers.27.self_attn.g_b_proj",
+    "model.layers.27.self_attn.o_proj",
+    "model.layers.28.self_attn.q_proj",
+    "model.layers.28.self_attn.k_proj",
+    "model.layers.28.self_attn.v_proj",
+    "model.layers.28.self_attn.o_proj",
+    "model.layers.28.self_attn.g_proj",
+    "model.layers.29.self_attn.q_proj",
+    "model.layers.29.self_attn.k_proj",
+    "model.layers.29.self_attn.v_proj",
+    "model.layers.29.self_attn.f_a_proj",
+    "model.layers.29.self_attn.f_b_proj",
+    "model.layers.29.self_attn.b_proj",
+    "model.layers.29.self_attn.g_a_proj",
+    "model.layers.29.self_attn.g_b_proj",
+    "model.layers.29.self_attn.o_proj",
+    "model.layers.30.self_attn.q_proj",
+    "model.layers.30.self_attn.k_proj",
+    "model.layers.30.self_attn.v_proj",
+    "model.layers.30.self_attn.f_a_proj",
+    "model.layers.30.self_attn.f_b_proj",
+    "model.layers.30.self_attn.b_proj",
+    "model.layers.30.self_attn.g_a_proj",
+    "model.layers.30.self_attn.g_b_proj",
+    "model.layers.30.self_attn.o_proj",
+    "model.layers.31.self_attn.q_proj",
+    "model.layers.31.self_attn.k_proj",
+    "model.layers.31.self_attn.v_proj",
+    "model.layers.31.self_attn.f_a_proj",
+    "model.layers.31.self_attn.f_b_proj",
+    "model.layers.31.self_attn.b_proj",
+    "model.layers.31.self_attn.g_a_proj",
+    "model.layers.31.self_attn.g_b_proj",
+    "model.layers.31.self_attn.o_proj",
+    "model.layers.32.self_attn.q_proj",
+    "model.layers.32.self_attn.k_proj",
+    "model.layers.32.self_attn.v_proj",
+    "model.layers.32.self_attn.o_proj",
+    "model.layers.32.self_attn.g_proj",
+    "model.layers.33.self_attn.q_proj",
+    "model.layers.33.self_attn.k_proj",
+    "model.layers.33.self_attn.v_proj",
+    "model.layers.33.self_attn.f_a_proj",
+    "model.layers.33.self_attn.f_b_proj",
+    "model.layers.33.self_attn.b_proj",
+    "model.layers.33.self_attn.g_a_proj",
+    "model.layers.33.self_attn.g_b_proj",
+    "model.layers.33.self_attn.o_proj",
+    "model.layers.34.self_attn.q_proj",
+    "model.layers.34.self_attn.k_proj",
+    "model.layers.34.self_attn.v_proj",
+    "model.layers.34.self_attn.f_a_proj",
+    "model.layers.34.self_attn.f_b_proj",
+    "model.layers.34.self_attn.b_proj",
+    "model.layers.34.self_attn.g_a_proj",
+    "model.layers.34.self_attn.g_b_proj",
+    "model.layers.34.self_attn.o_proj",
+    "model.layers.35.self_attn.q_proj",
+    "model.layers.35.self_attn.k_proj",
+    "model.layers.35.self_attn.v_proj",
+    "model.layers.35.self_attn.f_a_proj",
+    "model.layers.35.self_attn.f_b_proj",
+    "model.layers.35.self_attn.b_proj",
+    "model.layers.35.self_attn.g_a_proj",
+    "model.layers.35.self_attn.g_b_proj",
+    "model.layers.35.self_attn.o_proj",
+    "model.layers.36.self_attn.q_proj",
+    "model.layers.36.self_attn.k_proj",
+    "model.layers.36.self_attn.v_proj",
+    "model.layers.36.self_attn.o_proj",
+    "model.layers.36.self_attn.g_proj",
+    "model.layers.37.self_attn.q_proj",
+    "model.layers.37.self_attn.k_proj",
+    "model.layers.37.self_attn.v_proj",
+    "model.layers.37.self_attn.f_a_proj",
+    "model.layers.37.self_attn.f_b_proj",
+    "model.layers.37.self_attn.b_proj",
+    "model.layers.37.self_attn.g_a_proj",
+    "model.layers.37.self_attn.g_b_proj",
+    "model.layers.37.self_attn.o_proj",
+    "model.layers.38.self_attn.q_proj",
+    "model.layers.38.self_attn.k_proj",
+    "model.layers.38.self_attn.v_proj",
+    "model.layers.38.self_attn.f_a_proj",
+    "model.layers.38.self_attn.f_b_proj",
+    "model.layers.38.self_attn.b_proj",
+    "model.layers.38.self_attn.g_a_proj",
+    "model.layers.38.self_attn.g_b_proj",
+    "model.layers.38.self_attn.o_proj",
+    "model.layers.39.self_attn.q_proj",
+    "model.layers.39.self_attn.k_proj",
+    "model.layers.39.self_attn.v_proj",
+    "model.layers.39.self_attn.f_a_proj",
+    "model.layers.39.self_attn.f_b_proj",
+    "model.layers.39.self_attn.b_proj",
+    "model.layers.39.self_attn.g_a_proj",
+    "model.layers.39.self_attn.g_b_proj",
+    "model.layers.39.self_attn.o_proj",
+    "model.layers.40.self_attn.q_proj",
+    "model.layers.40.self_attn.k_proj",
+    "model.layers.40.self_attn.v_proj",
+    "model.layers.40.self_attn.o_proj",
+    "model.layers.40.self_attn.g_proj",
+    "model.layers.41.self_attn.q_proj",
+    "model.layers.41.self_attn.k_proj",
+    "model.layers.41.self_attn.v_proj",
+    "model.layers.41.self_attn.f_a_proj",
+    "model.layers.41.self_attn.f_b_proj",
+    "model.layers.41.self_attn.b_proj",
+    "model.layers.41.self_attn.g_a_proj",
+    "model.layers.41.self_attn.g_b_proj",
+    "model.layers.41.self_attn.o_proj",
+    "model.layers.42.self_attn.q_proj",
+    "model.layers.42.self_attn.k_proj",
+    "model.layers.42.self_attn.v_proj",
+    "model.layers.42.self_attn.f_a_proj",
+    "model.layers.42.self_attn.f_b_proj",
+    "model.layers.42.self_attn.b_proj",
+    "model.layers.42.self_attn.g_a_proj",
+    "model.layers.42.self_attn.g_b_proj",
+    "model.layers.42.self_attn.o_proj",
+    "model.layers.43.self_attn.q_proj",
+    "model.layers.43.self_attn.k_proj",
+    "model.layers.43.self_attn.v_proj",
+    "model.layers.43.self_attn.f_a_proj",
+    "model.layers.43.self_attn.f_b_proj",
+    "model.layers.43.self_attn.b_proj",
+    "model.layers.43.self_attn.g_a_proj",
+    "model.layers.43.self_attn.g_b_proj",
+    "model.layers.43.self_attn.o_proj",
+    "model.layers.44.self_attn.q_proj",
+    "model.layers.44.self_attn.k_proj",
+    "model.layers.44.self_attn.v_proj",
+    "model.layers.44.self_attn.o_proj",
+    "model.layers.44.self_attn.g_proj",
+    "model.layers.45.self_attn.q_proj",
+    "model.layers.45.self_attn.k_proj",
+    "model.layers.45.self_attn.v_proj",
+    "model.layers.45.self_attn.f_a_proj",
+    "model.layers.45.self_attn.f_b_proj",
+    "model.layers.45.self_attn.b_proj",
+    "model.layers.45.self_attn.g_a_proj",
+    "model.layers.45.self_attn.g_b_proj",
+    "model.layers.45.self_attn.o_proj",
+    "model.layers.46.self_attn.q_proj",
+    "model.layers.46.self_attn.k_proj",
+    "model.layers.46.self_attn.v_proj",
+    "model.layers.46.self_attn.f_a_proj",
+    "model.layers.46.self_attn.f_b_proj",
+    "model.layers.46.self_attn.b_proj",
+    "model.layers.46.self_attn.g_a_proj",
+    "model.layers.46.self_attn.g_b_proj",
+    "model.layers.46.self_attn.o_proj",
+    "model.layers.47.self_attn.q_proj",
+    "model.layers.47.self_attn.k_proj",
+    "model.layers.47.self_attn.v_proj",
+    "model.layers.47.self_attn.f_a_proj",
+    "model.layers.47.self_attn.f_b_proj",
+    "model.layers.47.self_attn.b_proj",
+    "model.layers.47.self_attn.g_a_proj",
+    "model.layers.47.self_attn.g_b_proj",
+    "model.layers.47.self_attn.o_proj",
+    "model.layers.0.mlp.shared_experts.gate_proj",
+    "model.layers.0.mlp.shared_experts.up_proj",
+    "model.layers.0.mlp.shared_experts.down_proj",
+    "model.layers.1.mlp.shared_experts.gate_proj",
+    "model.layers.1.mlp.shared_experts.up_proj",
+    "model.layers.1.mlp.shared_experts.down_proj",
+    "model.layers.2.mlp.shared_experts.gate_proj",
+    "model.layers.2.mlp.shared_experts.up_proj",
+    "model.layers.2.mlp.shared_experts.down_proj",
+    "model.layers.3.mlp.shared_experts.gate_proj",
+    "model.layers.3.mlp.shared_experts.up_proj",
+    "model.layers.3.mlp.shared_experts.down_proj",
+    "model.layers.4.mlp.shared_experts.gate_proj",
+    "model.layers.4.mlp.shared_experts.up_proj",
+    "model.layers.4.mlp.shared_experts.down_proj",
+    "model.layers.5.mlp.shared_experts.gate_proj",
+    "model.layers.5.mlp.shared_experts.up_proj",
+    "model.layers.5.mlp.shared_experts.down_proj",
+    "model.layers.6.mlp.shared_experts.gate_proj",
+    "model.layers.6.mlp.shared_experts.up_proj",
+    "model.layers.6.mlp.shared_experts.down_proj",
+    "model.layers.7.mlp.shared_experts.gate_proj",
+    "model.layers.7.mlp.shared_experts.up_proj",
+    "model.layers.7.mlp.shared_experts.down_proj",
+    "model.layers.8.mlp.shared_experts.gate_proj",
+    "model.layers.8.mlp.shared_experts.up_proj",
+    "model.layers.8.mlp.shared_experts.down_proj",
+    "model.layers.9.mlp.shared_experts.gate_proj",
+    "model.layers.9.mlp.shared_experts.up_proj",
+    "model.layers.9.mlp.shared_experts.down_proj",
+    "model.layers.10.mlp.shared_experts.gate_proj",
+    "model.layers.10.mlp.shared_experts.up_proj",
+    "model.layers.10.mlp.shared_experts.down_proj",
+    "model.layers.11.mlp.shared_experts.gate_proj",
+    "model.layers.11.mlp.shared_experts.up_proj",
+    "model.layers.11.mlp.shared_experts.down_proj",
+    "model.layers.12.mlp.shared_experts.gate_proj",
+    "model.layers.12.mlp.shared_experts.up_proj",
+    "model.layers.12.mlp.shared_experts.down_proj",
+    "model.layers.13.mlp.shared_experts.gate_proj",
+    "model.layers.13.mlp.shared_experts.up_proj",
+    "model.layers.13.mlp.shared_experts.down_proj",
+    "model.layers.14.mlp.shared_experts.gate_proj",
+    "model.layers.14.mlp.shared_experts.up_proj",
+    "model.layers.14.mlp.shared_experts.down_proj",
+    "model.layers.15.mlp.shared_experts.gate_proj",
+    "model.layers.15.mlp.shared_experts.up_proj",
+    "model.layers.15.mlp.shared_experts.down_proj",
+    "model.layers.16.mlp.shared_experts.gate_proj",
+    "model.layers.16.mlp.shared_experts.up_proj",
+    "model.layers.16.mlp.shared_experts.down_proj",
+    "model.layers.17.mlp.shared_experts.gate_proj",
+    "model.layers.17.mlp.shared_experts.up_proj",
+    "model.layers.17.mlp.shared_experts.down_proj",
+    "model.layers.18.mlp.shared_experts.gate_proj",
+    "model.layers.18.mlp.shared_experts.up_proj",
+    "model.layers.18.mlp.shared_experts.down_proj",
+    "model.layers.19.mlp.shared_experts.gate_proj",
+    "model.layers.19.mlp.shared_experts.up_proj",
+    "model.layers.19.mlp.shared_experts.down_proj",
+    "model.layers.20.mlp.shared_experts.gate_proj",
+    "model.layers.20.mlp.shared_experts.up_proj",
+    "model.layers.20.mlp.shared_experts.down_proj",
+    "model.layers.21.mlp.shared_experts.gate_proj",
+    "model.layers.21.mlp.shared_experts.up_proj",
+    "model.layers.21.mlp.shared_experts.down_proj",
+    "model.layers.22.mlp.shared_experts.gate_proj",
+    "model.layers.22.mlp.shared_experts.up_proj",
+    "model.layers.22.mlp.shared_experts.down_proj",
+    "model.layers.23.mlp.shared_experts.gate_proj",
+    "model.layers.23.mlp.shared_experts.up_proj",
+    "model.layers.23.mlp.shared_experts.down_proj",
+    "model.layers.24.mlp.shared_experts.gate_proj",
+    "model.layers.24.mlp.shared_experts.up_proj",
+    "model.layers.24.mlp.shared_experts.down_proj",
+    "model.layers.25.mlp.shared_experts.gate_proj",
+    "model.layers.25.mlp.shared_experts.up_proj",
+    "model.layers.25.mlp.shared_experts.down_proj",
+    "model.layers.26.mlp.shared_experts.gate_proj",
+    "model.layers.26.mlp.shared_experts.up_proj",
+    "model.layers.26.mlp.shared_experts.down_proj",
+    "model.layers.27.mlp.shared_experts.gate_proj",
+    "model.layers.27.mlp.shared_experts.up_proj",
+    "model.layers.27.mlp.shared_experts.down_proj",
+    "model.layers.28.mlp.shared_experts.gate_proj",
+    "model.layers.28.mlp.shared_experts.up_proj",
+    "model.layers.28.mlp.shared_experts.down_proj",
+    "model.layers.29.mlp.shared_experts.gate_proj",
+    "model.layers.29.mlp.shared_experts.up_proj",
+    "model.layers.29.mlp.shared_experts.down_proj",
+    "model.layers.30.mlp.shared_experts.gate_proj",
+    "model.layers.30.mlp.shared_experts.up_proj",
+    "model.layers.30.mlp.shared_experts.down_proj",
+    "model.layers.31.mlp.shared_experts.gate_proj",
+    "model.layers.31.mlp.shared_experts.up_proj",
+    "model.layers.31.mlp.shared_experts.down_proj",
+    "model.layers.32.mlp.shared_experts.gate_proj",
+    "model.layers.32.mlp.shared_experts.up_proj",
+    "model.layers.32.mlp.shared_experts.down_proj",
+    "model.layers.33.mlp.shared_experts.gate_proj",
+    "model.layers.33.mlp.shared_experts.up_proj",
+    "model.layers.33.mlp.shared_experts.down_proj",
+    "model.layers.34.mlp.shared_experts.gate_proj",
+    "model.layers.34.mlp.shared_experts.up_proj",
+    "model.layers.34.mlp.shared_experts.down_proj",
+    "model.layers.35.mlp.shared_experts.gate_proj",
+    "model.layers.35.mlp.shared_experts.up_proj",
+    "model.layers.35.mlp.shared_experts.down_proj",
+    "model.layers.36.mlp.shared_experts.gate_proj",
+    "model.layers.36.mlp.shared_experts.up_proj",
+    "model.layers.36.mlp.shared_experts.down_proj",
+    "model.layers.37.mlp.shared_experts.gate_proj",
+    "model.layers.37.mlp.shared_experts.up_proj",
+    "model.layers.37.mlp.shared_experts.down_proj",
+    "model.layers.38.mlp.shared_experts.gate_proj",
+    "model.layers.38.mlp.shared_experts.up_proj",
+    "model.layers.38.mlp.shared_experts.down_proj",
+    "model.layers.39.mlp.shared_experts.gate_proj",
+    "model.layers.39.mlp.shared_experts.up_proj",
+    "model.layers.39.mlp.shared_experts.down_proj",
+    "model.layers.40.mlp.shared_experts.gate_proj",
+    "model.layers.40.mlp.shared_experts.up_proj",
+    "model.layers.40.mlp.shared_experts.down_proj",
+    "model.layers.41.mlp.shared_experts.gate_proj",
+    "model.layers.41.mlp.shared_experts.up_proj",
+    "model.layers.41.mlp.shared_experts.down_proj",
+    "model.layers.42.mlp.shared_experts.gate_proj",
+    "model.layers.42.mlp.shared_experts.up_proj",
+    "model.layers.42.mlp.shared_experts.down_proj",
+    "model.layers.43.mlp.shared_experts.gate_proj",
+    "model.layers.43.mlp.shared_experts.up_proj",
+    "model.layers.43.mlp.shared_experts.down_proj",
+    "model.layers.44.mlp.shared_experts.gate_proj",
+    "model.layers.44.mlp.shared_experts.up_proj",
+    "model.layers.44.mlp.shared_experts.down_proj",
+    "model.layers.45.mlp.shared_experts.gate_proj",
+    "model.layers.45.mlp.shared_experts.up_proj",
+    "model.layers.45.mlp.shared_experts.down_proj",
+    "model.layers.46.mlp.shared_experts.gate_proj",
+    "model.layers.46.mlp.shared_experts.up_proj",
+    "model.layers.46.mlp.shared_experts.down_proj",
+    "model.layers.47.mlp.shared_experts.gate_proj",
+    "model.layers.47.mlp.shared_experts.up_proj",
+    "model.layers.47.mlp.shared_experts.down_proj",
+    "lm_head"
+  ],
+  "provider": "auto-round"
+}

special_tokens_map.json

@@ -0,0 +1,4008 @@
+{
+  "additional_special_tokens": [
+    "<unk>",
+    "<|startoftext|>",
+    "<|endoftext|>",
+    "<|fim_prefix|>",
+    "<|fim_middle|>",
+    "<|fim_suffix|>",
+    "<|repo_name|>",
+    "<|file_sep|>",
+    "<|pr|>",
+    "<|pr_status|>",
+    "<|pr_base|>",
+    "<|pr_file|>",
+    "<|pr_base_code|>",
+    "<|pr_diff|>",
+    "<|pr_diff_hunk|>",
+    "<|pr_comment|>",
+    "<|pr_event_id|>",
+    "<|pr_review|>",
+    "<|pr_review_state|>",
+    "<|pr_review_comment|>",
+    "<|pr_in_reply_to_review_id|>",
+    "<|pr_in_reply_to_comment_id|>",
+    "<|pr_diff_hunk_comment_line|>",
+    "<|pr_is_merged|>",
+    "<|pr_linked_issue|>",
+    "<|pr_linked_issue_body|>",
+    "<|special_26|>",
+    "<|special_27|>",
+    "<|special_28|>",
+    "<|special_29|>",
+    "<|special_30|>",
+    "<|special_31|>",
+    "<|special_32|>",
+    "<|special_33|>",
+    "<|special_34|>",
+    "<|special_35|>",
+    "<|special_36|>",
+    "<|special_37|>",
+    "<|special_38|>",
+    "<|special_39|>",
+    "<|special_40|>",
+    "<|special_41|>",
+    "<|special_42|>",
+    "<|special_43|>",
+    "<|special_44|>",
+    "<|special_45|>",
+    "<|special_46|>",
+    "<|special_47|>",
+    "<|special_48|>",
+    "<|special_49|>",
+    "<|special_50|>",
+    "<|special_51|>",
+    "<|special_52|>",
+    "<|special_53|>",
+    "<|special_54|>",
+    "<|special_55|>",
+    "<|special_56|>",
+    "<|special_57|>",
+    "<|special_58|>",
+    "<|special_59|>",
+    "<|special_60|>",
+    "<|special_61|>",
+    "<|special_62|>",
+    "<|special_63|>",
+    "<|special_64|>",
+    "<|special_65|>",
+    "<|special_66|>",
+    "<|special_67|>",
+    "<|special_68|>",
+    "<|special_69|>",
+    "<|special_70|>",
+    "<|special_71|>",
+    "<|special_72|>",
+    "<|special_73|>",
+    "<|special_74|>",
+    "<|special_75|>",
+    "<|special_76|>",
+    "<|special_77|>",
+    "<|special_78|>",
+    "<|special_79|>",
+    "<|special_80|>",
+    "<|special_81|>",
+    "<|special_82|>",
+    "<|special_83|>",
+    "<|special_84|>",
+    "<|special_85|>",
+    "<|special_86|>",
+    "<|special_87|>",
+    "<|special_88|>",
+    "<|special_89|>",
+    "<|special_90|>",
+    "<|special_91|>",
+    "<|special_92|>",
+    "<|special_93|>",
+    "<|special_94|>",
+    "<|special_95|>",
+    "<|special_96|>",
+    "<|special_97|>",
+    "<|special_98|>",
+    "<|special_99|>",
+    "<|special_100|>",
+    "<|special_101|>",
+    "<|special_102|>",
+    "<|special_103|>",
+    "<|special_104|>",
+    "<|special_105|>",
+    "<|special_106|>",
+    "<|special_107|>",
+    "<|special_108|>",
+    "<|special_109|>",
+    "<|special_110|>",
+    "<|special_111|>",
+    "<|special_112|>",
+    "<|special_113|>",
+    "<|special_114|>",
+    "<|special_115|>",
+    "<|special_116|>",
+    "<|special_117|>",
+    "<|special_118|>",
+    "<|special_119|>",
+    "<|special_120|>",
+    "<|special_121|>",
+    "<|special_122|>",
+    "<|special_123|>",
+    "<|special_124|>",
+    "<|special_125|>",
+    "<|special_126|>",
+    "<|special_127|>",
+    "<|im:start|>",
+    "<|im:end|>",
+    "<|im:content|>",
+    "<|tool:start|>",
+    "<|tool:end|>",
+    "<|tool_response:start|>",
+    "<|tool_response:end|>",
+    "<|special_142|>",
+    "<|special_143|>",
+    "<|special_144|>",
+    "<|special_145|>",
+    "<|special_146|>",
+    "<|special_147|>",
+    "<|special_148|>",
+    "<|special_149|>",
+    "<|special_150|>",
+    "<|special_151|>",
+    "<|special_152|>",
+    "<|special_153|>",
+    "<|special_154|>",
+    "<|special_155|>",
+    "<|special_156|>",
+    "<|special_157|>",
+    "<|special_158|>",
+    "<|special_159|>",
+    "<|special_160|>",
+    "<|special_161|>",
+    "<|special_162|>",
+    "<|special_163|>",
+    "<|special_164|>",
+    "<|special_165|>",
+    "<|special_166|>",
+    "<|special_167|>",
+    "<|special_168|>",
+    "<|special_169|>",
+    "<|special_170|>",
+    "<|special_171|>",
+    "<|special_172|>",
+    "<|special_173|>",
+    "<|special_174|>",
+    "<|special_175|>",
+    "<|special_176|>",
+    "<|special_177|>",
+    "<|special_178|>",
+    "<|special_179|>",
+    "<|special_180|>",
+    "<|special_181|>",
+    "<|special_182|>",
+    "<|special_183|>",
+    "<|special_184|>",
+    "<|special_185|>",
+    "<|special_186|>",
+    "<|special_187|>",
+    "<|special_188|>",
+    "<|special_189|>",
+    "<|special_190|>",
+    "<|special_191|>",
+    "<|special_192|>",
+    "<|special_193|>",
+    "<|special_194|>",
+    "<|special_195|>",
+    "<|special_196|>",
+    "<|special_197|>",
+    "<|special_198|>",
+    "<|special_199|>",
+    "<|special_200|>",
+    "<|special_201|>",
+    "<|special_202|>",
+    "<|special_203|>",
+    "<|special_204|>",
+    "<|special_205|>",
+    "<|special_206|>",
+    "<|special_207|>",
+    "<|special_208|>",
+    "<|special_209|>",
+    "<|special_210|>",
+    "<|special_211|>",
+    "<|special_212|>",
+    "<|special_213|>",
+    "<|special_214|>",
+    "<|special_215|>",
+    "<|special_216|>",
+    "<|special_217|>",
+    "<|special_218|>",
+    "<|special_219|>",
+    "<|special_220|>",
+    "<|special_221|>",
+    "<|special_222|>",
+    "<|special_223|>",
+    "<|special_224|>",
+    "<|special_225|>",
+    "<|special_226|>",
+    "<|special_227|>",
+    "<|special_228|>",
+    "<|special_229|>",
+    "<|special_230|>",
+    "<|special_231|>",
+    "<|special_232|>",
+    "<|special_233|>",
+    "<|special_234|>",
+    "<|special_235|>",
+    "<|special_236|>",
+    "<|special_237|>",
+    "<|special_238|>",
+    "<|special_239|>",
+    "<|special_240|>",
+    "<|special_241|>",
+    "<|special_242|>",
+    "<|special_243|>",
+    "<|special_244|>",
+    "<|special_245|>",
+    "<|special_246|>",
+    "<|special_247|>",
+    "<|special_248|>",
+    "<|special_249|>",
+    "<|special_250|>",
+    "<|special_251|>",
+    "<|special_252|>",
+    "<|special_253|>",
+    "<|special_254|>",
+    "<|special_255|>",
+    "<|special_256|>",
+    "<|special_257|>",
+    "<|special_258|>",
+    "<|special_259|>",
+    "<|special_260|>",
+    "<|special_261|>",
+    "<|special_262|>",
+    "<|special_263|>",
+    "<|special_264|>",
+    "<|special_265|>",
+    "<|special_266|>",
+    "<|special_267|>",
+    "<|special_268|>",
+    "<|special_269|>",
+    "<|special_270|>",
+    "<|special_271|>",
+    "<|special_272|>",
+    "<|special_273|>",
+    "<|special_274|>",
+    "<|special_275|>",
+    "<|special_276|>",
+    "<|special_277|>",
+    "<|special_278|>",
+    "<|special_279|>",
+    "<|special_280|>",
+    "<|special_281|>",
+    "<|special_282|>",
+    "<|special_283|>",
+    "<|special_284|>",
+    "<|special_285|>",
+    "<|special_286|>",
+    "<|special_287|>",
+    "<|special_288|>",
+    "<|special_289|>",
+    "<|special_290|>",
+    "<|special_291|>",
+    "<|special_292|>",
+    "<|special_293|>",
+    "<|special_294|>",
+    "<|special_295|>",
+    "<|special_296|>",
+    "<|special_297|>",
+    "<|special_298|>",
+    "<|special_299|>",
+    "<|special_300|>",
+    "<|special_301|>",
+    "<|special_302|>",
+    "<|special_303|>",
+    "<|special_304|>",
+    "<|special_305|>",
+    "<|special_306|>",
+    "<|special_307|>",
+    "<|special_308|>",
+    "<|special_309|>",
+    "<|special_310|>",
+    "<|special_311|>",
+    "<|special_312|>",
+    "<|special_313|>",
+    "<|special_314|>",
+    "<|special_315|>",
+    "<|special_316|>",
+    "<|special_317|>",
+    "<|special_318|>",
+    "<|special_319|>",
+    "<|special_320|>",
+    "<|special_321|>",
+    "<|special_322|>",
+    "<|special_323|>",
+    "<|special_324|>",
+    "<|special_325|>",
+    "<|special_326|>",
+    "<|special_327|>",
+    "<|special_328|>",
+    "<|special_329|>",
+    "<|special_330|>",
+    "<|special_331|>",
+    "<|special_332|>",
+    "<|special_333|>",
+    "<|special_334|>",
+    "<|special_335|>",
+    "<|special_336|>",
+    "<|special_337|>",
+    "<|special_338|>",
+    "<|special_339|>",
+    "<|special_340|>",
+    "<|special_341|>",
+    "<|special_342|>",
+    "<|special_343|>",
+    "<|special_344|>",
+    "<|special_345|>",
+    "<|special_346|>",
+    "<|special_347|>",
+    "<|special_348|>",
+    "<|special_349|>",
+    "<|special_350|>",
+    "<|special_351|>",
+    "<|special_352|>",
+    "<|special_353|>",
+    "<|special_354|>",
+    "<|special_355|>",
+    "<|special_356|>",
+    "<|special_357|>",
+    "<|special_358|>",
+    "<|special_359|>",
+    "<|special_360|>",
+    "<|special_361|>",
+    "<|special_362|>",
+    "<|special_363|>",
+    "<|special_364|>",
+    "<|special_365|>",
+    "<|special_366|>",
+    "<|special_367|>",
+    "<|special_368|>",
+    "<|special_369|>",
+    "<|special_370|>",
+    "<|special_371|>",
+    "<|special_372|>",
+    "<|special_373|>",
+    "<|special_374|>",
+    "<|special_375|>",
+    "<|special_376|>",
+    "<|special_377|>",
+    "<|special_378|>",
+    "<|special_379|>",
+    "<|special_380|>",
+    "<|special_381|>",
+    "<|special_382|>",
+    "<|special_383|>",
+    "<|special_384|>",
+    "<|special_385|>",
+    "<|special_386|>",
+    "<|special_387|>",
+    "<|special_388|>",
+    "<|special_389|>",
+    "<|special_390|>",
+    "<|special_391|>",
+    "<|special_392|>",
+    "<|special_393|>",
+    "<|special_394|>",
+    "<|special_395|>",
+    "<|special_396|>",
+    "<|special_397|>",
+    "<|special_398|>",
+    "<|special_399|>",
+    "<|special_400|>",
+    "<|special_401|>",
+    "<|special_402|>",
+    "<|special_403|>",
+    "<|special_404|>",
+    "<|special_405|>",
+    "<|special_406|>",
+    "<|special_407|>",
+    "<|special_408|>",
+    "<|special_409|>",
+    "<|special_410|>",
+    "<|special_411|>",
+    "<|special_412|>",
+    "<|special_413|>",
+    "<|special_414|>",
+    "<|special_415|>",
+    "<|special_416|>",
+    "<|special_417|>",
+    "<|special_418|>",
+    "<|special_419|>",
+    "<|special_420|>",
+    "<|special_421|>",
+    "<|special_422|>",
+    "<|special_423|>",
+    "<|special_424|>",
+    "<|special_425|>",
+    "<|special_426|>",
+    "<|special_427|>",
+    "<|special_428|>",
+    "<|special_429|>",
+    "<|special_430|>",
+    "<|special_431|>",
+    "<|special_432|>",
+    "<|special_433|>",
+    "<|special_434|>",
+    "<|special_435|>",
+    "<|special_436|>",
+    "<|special_437|>",
+    "<|special_438|>",
+    "<|special_439|>",
+    "<|special_440|>",
+    "<|special_441|>",
+    "<|special_442|>",
+    "<|special_443|>",
+    "<|special_444|>",
+    "<|special_445|>",
+    "<|special_446|>",
+    "<|special_447|>",
+    "<|special_448|>",
+    "<|special_449|>",
+    "<|special_450|>",
+    "<|special_451|>",
+    "<|special_452|>",
+    "<|special_453|>",
+    "<|special_454|>",
+    "<|special_455|>",
+    "<|special_456|>",
+    "<|special_457|>",
+    "<|special_458|>",
+    "<|special_459|>",
+    "<|special_460|>",
+    "<|special_461|>",
+    "<|special_462|>",
+    "<|special_463|>",
+    "<|special_464|>",
+    "<|special_465|>",
+    "<|special_466|>",
+    "<|special_467|>",
+    "<|special_468|>",
+    "<|special_469|>",
+    "<|special_470|>",
+    "<|special_471|>",
+    "<|special_472|>",
+    "<|special_473|>",
+    "<|special_474|>",
+    "<|special_475|>",
+    "<|special_476|>",
+    "<|special_477|>",
+    "<|special_478|>",
+    "<|special_479|>",
+    "<|special_480|>",
+    "<|special_481|>",
+    "<|special_482|>",
+    "<|special_483|>",
+    "<|special_484|>",
+    "<|special_485|>",
+    "<|special_486|>",
+    "<|special_487|>",
+    "<|special_488|>",
+    "<|special_489|>",
+    "<|special_490|>",
+    "<|special_491|>",
+    "<|special_492|>",
+    "<|special_493|>",
+    "<|special_494|>",
+    "<|special_495|>",
+    "<|special_496|>",
+    "<|special_497|>",
+    "<|special_498|>",
+    "<|special_499|>",
+    "<|special_500|>",
+    "<|special_501|>",
+    "<|special_502|>",
+    "<|special_503|>",
+    "<|special_504|>",
+    "<|special_505|>",
+    "<|special_506|>",
+    "<|special_507|>",
+    "<|special_508|>",
+    "<|special_509|>",
+    "<|special_510|>",
+    "<|special_511|>",
+    "<|special_625|>",
+    "<|special_626|>",
+    "<|special_627|>",
+    "<|special_628|>",
+    "<|special_629|>",
+    "<|special_630|>",
+    "<|special_631|>",
+    "<|special_632|>",
+    "<|special_633|>",
+    "<|special_634|>",
+    "<|special_635|>",
+    "<|special_636|>",
+    "<|special_637|>",
+    "<|special_638|>",
+    "<|special_639|>",
+    "<|special_640|>",
+    "<|special_641|>",
+    "<|special_642|>",
+    "<|special_643|>",
+    "<|special_644|>",
+    "<|special_645|>",
+    "<|special_646|>",
+    "<|special_647|>",
+    "<|special_648|>",
+    "<|special_649|>",
+    "<|special_650|>",
+    "<|special_651|>",
+    "<|special_652|>",
+    "<|special_653|>",
+    "<|special_654|>",
+    "<|special_655|>",
+    "<|special_656|>",
+    "<|special_657|>",
+    "<|special_658|>",
+    "<|special_659|>",
+    "<|special_660|>",
+    "<|special_661|>",
+    "<|special_662|>",
+    "<|special_663|>",
+    "<|special_664|>",
+    "<|special_665|>",
+    "<|special_666|>",
+    "<|special_667|>",
+    "<|special_668|>",
+    "<|special_669|>",
+    "<|special_670|>",
+    "<|special_671|>",
+    "<|special_672|>",
+    "<|special_673|>",
+    "<|special_674|>",
+    "<|special_675|>",
+    "<|special_676|>",
+    "<|special_677|>",
+    "<|special_678|>",
+    "<|special_679|>",
+    "<|special_680|>",
+    "<|special_681|>",
+    "<|special_682|>",
+    "<|special_683|>",
+    "<|special_684|>",
+    "<|special_685|>",
+    "<|special_686|>",
+    "<|special_687|>",
+    "<|special_688|>",
+    "<|special_689|>",
+    "<|special_690|>",
+    "<|special_691|>",
+    "<|special_692|>",
+    "<|special_693|>",
+    "<|special_694|>",
+    "<|special_695|>",
+    "<|special_696|>",
+    "<|special_697|>",
+    "<|special_698|>",
+    "<|special_699|>",
+    "<|special_700|>",
+    "<|special_701|>",
+    "<|special_702|>",
+    "<|special_703|>",
+    "<|special_704|>",
+    "<|special_705|>",
+    "<|special_706|>",
+    "<|special_707|>",
+    "<|special_708|>",
+    "<|special_709|>",
+    "<|special_710|>",
+    "<|special_711|>",
+    "<|special_712|>",
+    "<|special_713|>",
+    "<|special_714|>",
+    "<|special_715|>",
+    "<|special_716|>",
+    "<|special_717|>",
+    "<|special_718|>",
+    "<|special_719|>",
+    "<|special_720|>",
+    "<|special_721|>",
+    "<|special_722|>",
+    "<|special_723|>",
+    "<|special_724|>",
+    "<|special_725|>",
+    "<|special_726|>",
+    "<|special_727|>",
+    "<|special_728|>",
+    "<|special_729|>",
+    "<|special_730|>",
+    "<|special_731|>",
+    "<|special_732|>",
+    "<|special_733|>",
+    "<|special_734|>",
+    "<|special_735|>",
+    "<|special_736|>",
+    "<|special_737|>",
+    "<|special_738|>",
+    "<|special_739|>",
+    "<|special_740|>",
+    "<|special_741|>",
+    "<|special_742|>",
+    "<|special_743|>",
+    "<|special_744|>",
+    "<|special_745|>",
+    "<|special_746|>",
+    "<|special_747|>",
+    "<|special_748|>",
+    "<|special_749|>",
+    "<|special_750|>",
+    "<|special_751|>",
+    "<|special_752|>",
+    "<|special_753|>",
+    "<|special_754|>",
+    "<|special_755|>",
+    "<|special_756|>",
+    "<|special_757|>",
+    "<|special_758|>",
+    "<|special_759|>",
+    "<|special_760|>",
+    "<|special_761|>",
+    "<|special_762|>",
+    "<|special_763|>",
+    "<|special_764|>",
+    "<|special_765|>",
+    "<|special_766|>",
+    "<|special_767|>",
+    "<|special_768|>",
+    "<|special_769|>",
+    "<|special_770|>",
+    "<|special_771|>",
+    "<|special_772|>",
+    "<|special_773|>",
+    "<|special_774|>",
+    "<|special_775|>",
+    "<|special_776|>",
+    "<|special_777|>",
+    "<|special_778|>",
+    "<|special_779|>",
+    "<|special_780|>",
+    "<|special_781|>",
+    "<|special_782|>",
+    "<|special_783|>",
+    "<|special_784|>",
+    "<|special_785|>",
+    "<|special_786|>",
+    "<|special_787|>",
+    "<|special_788|>",
+    "<|special_789|>",
+    "<|special_790|>",
+    "<|special_791|>",
+    "<|special_792|>",
+    "<|special_793|>",
+    "<|special_794|>",
+    "<|special_795|>",
+    "<|special_796|>",
+    "<|special_797|>",
+    "<|special_798|>",
+    "<|special_799|>",
+    "<|special_800|>",
+    "<|special_801|>",
+    "<|special_802|>",
+    "<|special_803|>",
+    "<|special_804|>",
+    "<|special_805|>",
+    "<|special_806|>",
+    "<|special_807|>",
+    "<|special_808|>",
+    "<|special_809|>",
+    "<|special_810|>",
+    "<|special_811|>",
+    "<|special_812|>",
+    "<|special_813|>",
+    "<|special_814|>",
+    "<|special_815|>",
+    "<|special_816|>",
+    "<|special_817|>",
+    "<|special_818|>",
+    "<|special_819|>",
+    "<|special_820|>",
+    "<|special_821|>",
+    "<|special_822|>",
+    "<|special_823|>",
+    "<|special_824|>",
+    "<|special_825|>",
+    "<|special_826|>",
+    "<|special_827|>",
+    "<|special_828|>",
+    "<|special_829|>",
+    "<|special_830|>",
+    "<|special_831|>",
+    "<|special_832|>",
+    "<|special_833|>",
+    "<|special_834|>",
+    "<|special_835|>",
+    "<|special_836|>",
+    "<|special_837|>",
+    "<|special_838|>",
+    "<|special_839|>",
+    "<|special_840|>",
+    "<|special_841|>",
+    "<|special_842|>",
+    "<|special_843|>",
+    "<|special_844|>",
+    "<|special_845|>",
+    "<|special_846|>",
+    "<|special_847|>",
+    "<|special_848|>",
+    "<|special_849|>",
+    "<|special_850|>",
+    "<|special_851|>",
+    "<|special_852|>",
+    "<|special_853|>",
+    "<|special_854|>",
+    "<|special_855|>",
+    "<|special_856|>",
+    "<|special_857|>",
+    "<|special_858|>",
+    "<|special_859|>",
+    "<|special_860|>",
+    "<|special_861|>",
+    "<|special_862|>",
+    "<|special_863|>",
+    "<|special_864|>",
+    "<|special_865|>",
+    "<|special_866|>",
+    "<|special_867|>",
+    "<|special_868|>",
+    "<|special_869|>",
+    "<|special_870|>",
+    "<|special_871|>",
+    "<|special_872|>",
+    "<|special_873|>",
+    "<|special_874|>",
+    "<|special_875|>",
+    "<|special_876|>",
+    "<|special_877|>",
+    "<|special_878|>",
+    "<|special_879|>",
+    "<|special_880|>",
+    "<|special_881|>",
+    "<|special_882|>",
+    "<|special_883|>",
+    "<|special_884|>",
+    "<|special_885|>",
+    "<|special_886|>",
+    "<|special_887|>",
+    "<|special_888|>",
+    "<|special_889|>",
+    "<|special_890|>",
+    "<|special_891|>",
+    "<|special_892|>",
+    "<|special_893|>",
+    "<|special_894|>",
+    "<|special_895|>",
+    "<|special_896|>",
+    "<|special_897|>",
+    "<|special_898|>",
+    "<|special_899|>",
+    "<|special_900|>",
+    "<|special_901|>",
+    "<|special_902|>",
+    "<|special_903|>",
+    "<|special_904|>",
+    "<|special_905|>",
+    "<|special_906|>",
+    "<|special_907|>",
+    "<|special_908|>",
+    "<|special_909|>",
+    "<|special_910|>",
+    "<|special_911|>",
+    "<|special_912|>",
+    "<|special_913|>",
+    "<|special_914|>",
+    "<|special_915|>",
+    "<|special_916|>",
+    "<|special_917|>",
+    "<|special_918|>",
+    "<|special_919|>",
+    "<|special_920|>",
+    "<|special_921|>",
+    "<|special_922|>",
+    "<|special_923|>",
+    "<|special_924|>",
+    "<|special_925|>",
+    "<|special_926|>",
+    "<|special_927|>",
+    "<|special_928|>",
+    "<|special_929|>",
+    "<|special_930|>",
+    "<|special_931|>",
+    "<|special_932|>",
+    "<|special_933|>",
+    "<|special_934|>",
+    "<|special_935|>",
+    "<|special_936|>",
+    "<|special_937|>",
+    "<|special_938|>",
+    "<|special_939|>",
+    "<|special_940|>",
+    "<|special_941|>",
+    "<|special_942|>",
+    "<|special_943|>",
+    "<|special_944|>",
+    "<|special_945|>",
+    "<|special_946|>",
+    "<|special_947|>",
+    "<|special_948|>",
+    "<|special_949|>",
+    "<|special_950|>",
+    "<|special_951|>",
+    "<|special_952|>",
+    "<|special_953|>",
+    "<|special_954|>",
+    "<|special_955|>",
+    "<|special_956|>",
+    "<|special_957|>",
+    "<|special_958|>",
+    "<|special_959|>",
+    "<|special_960|>",
+    "<|special_961|>",
+    "<|special_962|>",
+    "<|special_963|>",
+    "<|special_964|>",
+    "<|special_965|>",
+    "<|special_966|>",
+    "<|special_967|>",
+    "<|special_968|>",
+    "<|special_969|>",
+    "<|special_970|>",
+    "<|special_971|>",
+    "<|special_972|>",
+    "<|special_973|>",
+    "<|special_974|>",
+    "<|special_975|>",
+    "<|special_976|>",
+    "<|special_977|>",
+    "<|special_978|>",
+    "<|special_979|>",
+    "<|special_980|>",
+    "<|special_981|>",
+    "<|special_982|>",
+    "<|special_983|>",
+    "<|special_984|>",
+    "<|special_985|>",
+    "<|special_986|>",
+    "<|special_987|>",
+    "<|special_988|>",
+    "<|special_989|>",
+    "<|special_990|>",
+    "<|special_991|>",
+    "<|special_992|>",
+    "<|special_993|>",
+    "<|special_994|>",
+    "<|special_995|>",
+    "<|special_996|>",
+    "<|special_997|>",
+    "<|special_998|>",
+    "<|special_999|>",
+    "<|special_1000|>",
+    "<|special_1001|>",
+    "<|special_1002|>",
+    "<|special_1003|>",
+    "<|special_1004|>",
+    "<|special_1005|>",
+    "<|special_1006|>",
+    "<|special_1007|>",
+    "<|special_1008|>",
+    "<|special_1009|>",
+    "<|special_1010|>",
+    "<|special_1011|>",
+    "<|special_1012|>",
+    "<|special_1013|>",
+    "<|special_1014|>",
+    "<|special_1015|>",
+    "<|special_1016|>",
+    "<|special_1017|>",
+    "<|special_1018|>",
+    "<|special_1019|>",
+    "<|special_1020|>",
+    "<|special_1021|>",
+    "<|special_1022|>",
+    "<|special_1023|>",
+    "<|special_1024|>",
+    "<|special_1025|>",
+    "<|special_1026|>",
+    "<|special_1027|>",
+    "<|special_1028|>",
+    "<|special_1029|>",
+    "<|special_1030|>",
+    "<|special_1031|>",
+    "<|special_1032|>",
+    "<|special_1033|>",
+    "<|special_1034|>",
+    "<|special_1035|>",
+    "<|special_1036|>",
+    "<|special_1037|>",
+    "<|special_1038|>",
+    "<|special_1039|>",
+    "<|special_1040|>",
+    "<|special_1041|>",
+    "<|special_1042|>",
+    "<|special_1043|>",
+    "<|special_1044|>",
+    "<|special_1045|>",
+    "<|special_1046|>",
+    "<|special_1047|>",
+    "<|special_1048|>",
+    "<|special_1049|>",
+    "<|special_1050|>",
+    "<|special_1051|>",
+    "<|special_1052|>",
+    "<|special_1053|>",
+    "<|special_1054|>",
+    "<|special_1055|>",
+    "<|special_1056|>",
+    "<|special_1057|>",
+    "<|special_1058|>",
+    "<|special_1059|>",
+    "<|special_1060|>",
+    "<|special_1061|>",
+    "<|special_1062|>",
+    "<|special_1063|>",
+    "<|special_1064|>",
+    "<|special_1065|>",
+    "<|special_1066|>",
+    "<|special_1067|>",
+    "<|special_1068|>",
+    "<|special_1069|>",
+    "<|special_1070|>",
+    "<|special_1071|>",
+    "<|special_1072|>",
+    "<|special_1073|>",
+    "<|special_1074|>",
+    "<|special_1075|>",
+    "<|special_1076|>",
+    "<|special_1077|>",
+    "<|special_1078|>",
+    "<|special_1079|>",
+    "<|special_1080|>",
+    "<|special_1081|>",
+    "<|special_1082|>",
+    "<|special_1083|>",
+    "<|special_1084|>",
+    "<|special_1085|>",
+    "<|special_1086|>",
+    "<|special_1087|>",
+    "<|special_1088|>",
+    "<|special_1089|>",
+    "<|special_1090|>",
+    "<|special_1091|>",
+    "<|special_1092|>",
+    "<|special_1093|>",
+    "<|special_1094|>",
+    "<|special_1095|>",
+    "<|special_1096|>",
+    "<|special_1097|>",
+    "<|special_1098|>",
+    "<|special_1099|>",
+    "<|special_1100|>",
+    "<|special_1101|>",
+    "<|special_1102|>",
+    "<|special_1103|>",
+    "<|special_1104|>",
+    "<|special_1105|>",
+    "<|special_1106|>",
+    "<|special_1107|>",
+    "<|special_1108|>",
+    "<|special_1109|>",
+    "<|special_1110|>",
+    "<|special_1111|>",
+    "<|special_1112|>",
+    "<|special_1113|>",
+    "<|special_1114|>",
+    "<|special_1115|>",
+    "<|special_1116|>",
+    "<|special_1117|>",
+    "<|special_1118|>",
+    "<|special_1119|>",
+    "<|special_1120|>",
+    "<|special_1121|>",
+    "<|special_1122|>",
+    "<|special_1123|>",
+    "<|special_1124|>",
+    "<|special_1125|>",
+    "<|special_1126|>",
+    "<|special_1127|>",
+    "<|special_1128|>",
+    "<|special_1129|>",
+    "<|special_1130|>",
+    "<|special_1131|>",
+    "<|special_1132|>",
+    "<|special_1133|>",
+    "<|special_1134|>",
+    "<|special_1135|>",
+    "<|special_1136|>",
+    "<|special_1137|>",
+    "<|special_1138|>",
+    "<|special_1139|>",
+    "<|special_1140|>",
+    "<|special_1141|>",
+    "<|special_1142|>",
+    "<|special_1143|>",
+    "<|special_1144|>",
+    "<|special_1145|>",
+    "<|special_1146|>",
+    "<|special_1147|>",
+    "<|special_1148|>",
+    "<|special_1149|>",
+    "<|special_1150|>",
+    "<|special_1151|>",
+    "<|special_1152|>",
+    "<|special_1153|>",
+    "<|special_1154|>",
+    "<|special_1155|>",
+    "<|special_1156|>",
+    "<|special_1157|>",
+    "<|special_1158|>",
+    "<|special_1159|>",
+    "<|special_1160|>",
+    "<|special_1161|>",
+    "<|special_1162|>",
+    "<|special_1163|>",
+    "<|special_1164|>",
+    "<|special_1165|>",
+    "<|special_1166|>",
+    "<|special_1167|>",
+    "<|special_1168|>",
+    "<|special_1169|>",
+    "<|special_1170|>",
+    "<|special_1171|>",
+    "<|special_1172|>",
+    "<|special_1173|>",
+    "<|special_1174|>",
+    "<|special_1175|>",
+    "<|special_1176|>",
+    "<|special_1177|>",
+    "<|special_1178|>",
+    "<|special_1179|>",
+    "<|special_1180|>",
+    "<|special_1181|>",
+    "<|special_1182|>",
+    "<|special_1183|>",
+    "<|special_1184|>",
+    "<|special_1185|>",
+    "<|special_1186|>",
+    "<|special_1187|>",
+    "<|special_1188|>",
+    "<|special_1189|>",
+    "<|special_1190|>",
+    "<|special_1191|>",
+    "<|special_1192|>",
+    "<|special_1193|>",
+    "<|special_1194|>",
+    "<|special_1195|>",
+    "<|special_1196|>",
+    "<|special_1197|>",
+    "<|special_1198|>",
+    "<|special_1199|>",
+    "<|special_1200|>",
+    "<|special_1201|>",
+    "<|special_1202|>",
+    "<|special_1203|>",
+    "<|special_1204|>",
+    "<|special_1205|>",
+    "<|special_1206|>",
+    "<|special_1207|>",
+    "<|special_1208|>",
+    "<|special_1209|>",
+    "<|special_1210|>",
+    "<|special_1211|>",
+    "<|special_1212|>",
+    "<|special_1213|>",
+    "<|special_1214|>",
+    "<|special_1215|>",
+    "<|special_1216|>",
+    "<|special_1217|>",
+    "<|special_1218|>",
+    "<|special_1219|>",
+    "<|special_1220|>",
+    "<|special_1221|>",
+    "<|special_1222|>",
+    "<|special_1223|>",
+    "<|special_1224|>",
+    "<|special_1225|>",
+    "<|special_1226|>",
+    "<|special_1227|>",
+    "<|special_1228|>",
+    "<|special_1229|>",
+    "<|special_1230|>",
+    "<|special_1231|>",
+    "<|special_1232|>",
+    "<|special_1233|>",
+    "<|special_1234|>",
+    "<|special_1235|>",
+    "<|special_1236|>",
+    "<|special_1237|>",
+    "<|special_1238|>",
+    "<|special_1239|>",
+    "<|special_1240|>",
+    "<|special_1241|>",
+    "<|special_1242|>",
+    "<|special_1243|>",
+    "<|special_1244|>",
+    "<|special_1245|>",
+    "<|special_1246|>",
+    "<|special_1247|>",
+    "<|special_1248|>",
+    "<|special_1249|>",
+    "<|special_1250|>",
+    "<|special_1251|>",
+    "<|special_1252|>",
+    "<|special_1253|>",
+    "<|special_1254|>",
+    "<|special_1255|>",
+    "<|special_1256|>",
+    "<|special_1257|>",
+    "<|special_1258|>",
+    "<|special_1259|>",
+    "<|special_1260|>",
+    "<|special_1261|>",
+    "<|special_1262|>",
+    "<|special_1263|>",
+    "<|special_1264|>",
+    "<|special_1265|>",
+    "<|special_1266|>",
+    "<|special_1267|>",
+    "<|special_1268|>",
+    "<|special_1269|>",
+    "<|special_1270|>",
+    "<|special_1271|>",
+    "<|special_1272|>",
+    "<|special_1273|>",
+    "<|special_1274|>",
+    "<|special_1275|>",
+    "<|special_1276|>",
+    "<|special_1277|>",
+    "<|special_1278|>",
+    "<|special_1279|>",
+    "<|special_1280|>",
+    "<|special_1281|>",
+    "<|special_1282|>",
+    "<|special_1283|>",
+    "<|special_1284|>",
+    "<|special_1285|>",
+    "<|special_1286|>",
+    "<|special_1287|>",
+    "<|special_1288|>",
+    "<|special_1289|>",
+    "<|special_1290|>",
+    "<|special_1291|>",
+    "<|special_1292|>",
+    "<|special_1293|>",
+    "<|special_1294|>",
+    "<|special_1295|>",
+    "<|special_1296|>",
+    "<|special_1297|>",
+    "<|special_1298|>",
+    "<|special_1299|>",
+    "<|special_1300|>",
+    "<|special_1301|>",
+    "<|special_1302|>",
+    "<|special_1303|>",
+    "<|special_1304|>",
+    "<|special_1305|>",
+    "<|special_1306|>",
+    "<|special_1307|>",
+    "<|special_1308|>",
+    "<|special_1309|>",
+    "<|special_1310|>",
+    "<|special_1311|>",
+    "<|special_1312|>",
+    "<|special_1313|>",
+    "<|special_1314|>",
+    "<|special_1315|>",
+    "<|special_1316|>",
+    "<|special_1317|>",
+    "<|special_1318|>",
+    "<|special_1319|>",
+    "<|special_1320|>",
+    "<|special_1321|>",
+    "<|special_1322|>",
+    "<|special_1323|>",
+    "<|special_1324|>",
+    "<|special_1325|>",
+    "<|special_1326|>",
+    "<|special_1327|>",
+    "<|special_1328|>",
+    "<|special_1329|>",
+    "<|special_1330|>",
+    "<|special_1331|>",
+    "<|special_1332|>",
+    "<|special_1333|>",
+    "<|special_1334|>",
+    "<|special_1335|>",
+    "<|special_1336|>",
+    "<|special_1337|>",
+    "<|special_1338|>",
+    "<|special_1339|>",
+    "<|special_1340|>",
+    "<|special_1341|>",
+    "<|special_1342|>",
+    "<|special_1343|>",
+    "<|special_1344|>",
+    "<|special_1345|>",
+    "<|special_1346|>",
+    "<|special_1347|>",
+    "<|special_1348|>",
+    "<|special_1349|>",
+    "<|special_1350|>",
+    "<|special_1351|>",
+    "<|special_1352|>",
+    "<|special_1353|>",
+    "<|special_1354|>",
+    "<|special_1355|>",
+    "<|special_1356|>",
+    "<|special_1357|>",
+    "<|special_1358|>",
+    "<|special_1359|>",
+    "<|special_1360|>",
+    "<|special_1361|>",
+    "<|special_1362|>",
+    "<|special_1363|>",
+    "<|special_1364|>",
+    "<|special_1365|>",
+    "<|special_1366|>",
+    "<|special_1367|>",
+    "<|special_1368|>",
+    "<|special_1369|>",
+    "<|special_1370|>",
+    "<|special_1371|>",
+    "<|special_1372|>",
+    "<|special_1373|>",
+    "<|special_1374|>",
+    "<|special_1375|>",
+    "<|special_1376|>",
+    "<|special_1377|>",
+    "<|special_1378|>",
+    "<|special_1379|>",
+    "<|special_1380|>",
+    "<|special_1381|>",
+    "<|special_1382|>",
+    "<|special_1383|>",
+    "<|special_1384|>",
+    "<|special_1385|>",
+    "<|special_1386|>",
+    "<|special_1387|>",
+    "<|special_1388|>",
+    "<|special_1389|>",
+    "<|special_1390|>",
+    "<|special_1391|>",
+    "<|special_1392|>",
+    "<|special_1393|>",
+    "<|special_1394|>",
+    "<|special_1395|>",
+    "<|special_1396|>",
+    "<|special_1397|>",
+    "<|special_1398|>",
+    "<|special_1399|>",
+    "<|special_1400|>",
+    "<|special_1401|>",
+    "<|special_1402|>",
+    "<|special_1403|>",
+    "<|special_1404|>",
+    "<|special_1405|>",
+    "<|special_1406|>",
+    "<|special_1407|>",
+    "<|special_1408|>",
+    "<|special_1409|>",
+    "<|special_1410|>",
+    "<|special_1411|>",
+    "<|special_1412|>",
+    "<|special_1413|>",
+    "<|special_1414|>",
+    "<|special_1415|>",
+    "<|special_1416|>",
+    "<|special_1417|>",
+    "<|special_1418|>",
+    "<|special_1419|>",
+    "<|special_1420|>",
+    "<|special_1421|>",
+    "<|special_1422|>",
+    "<|special_1423|>",
+    "<|special_1424|>",
+    "<|special_1425|>",
+    "<|special_1426|>",
+    "<|special_1427|>",
+    "<|special_1428|>",
+    "<|special_1429|>",
+    "<|special_1430|>",
+    "<|special_1431|>",
+    "<|special_1432|>",
+    "<|special_1433|>",
+    "<|special_1434|>",
+    "<|special_1435|>",
+    "<|special_1436|>",
+    "<|special_1437|>",
+    "<|special_1438|>",
+    "<|special_1439|>",
+    "<|special_1440|>",
+    "<|special_1441|>",
+    "<|special_1442|>",
+    "<|special_1443|>",
+    "<|special_1444|>",
+    "<|special_1445|>",
+    "<|special_1446|>",
+    "<|special_1447|>",
+    "<|special_1448|>",
+    "<|special_1449|>",
+    "<|special_1450|>",
+    "<|special_1451|>",
+    "<|special_1452|>",
+    "<|special_1453|>",
+    "<|special_1454|>",
+    "<|special_1455|>",
+    "<|special_1456|>",
+    "<|special_1457|>",
+    "<|special_1458|>",
+    "<|special_1459|>",
+    "<|special_1460|>",
+    "<|special_1461|>",
+    "<|special_1462|>",
+    "<|special_1463|>",
+    "<|special_1464|>",
+    "<|special_1465|>",
+    "<|special_1466|>",
+    "<|special_1467|>",
+    "<|special_1468|>",
+    "<|special_1469|>",
+    "<|special_1470|>",
+    "<|special_1471|>",
+    "<|special_1472|>",
+    "<|special_1473|>",
+    "<|special_1474|>",
+    "<|special_1475|>",
+    "<|special_1476|>",
+    "<|special_1477|>",
+    "<|special_1478|>",
+    "<|special_1479|>",
+    "<|special_1480|>",
+    "<|special_1481|>",
+    "<|special_1482|>",
+    "<|special_1483|>",
+    "<|special_1484|>",
+    "<|special_1485|>",
+    "<|special_1486|>",
+    "<|special_1487|>",
+    "<|special_1488|>",
+    "<|special_1489|>",
+    "<|special_1490|>",
+    "<|special_1491|>",
+    "<|special_1492|>",
+    "<|special_1493|>",
+    "<|special_1494|>",
+    "<|special_1495|>",
+    "<|special_1496|>",
+    "<|special_1497|>",
+    "<|special_1498|>",
+    "<|special_1499|>",
+    "<|special_1500|>",
+    "<|special_1501|>",
+    "<|special_1502|>",
+    "<|special_1503|>",
+    "<|special_1504|>",
+    "<|special_1505|>",
+    "<|special_1506|>",
+    "<|special_1507|>",
+    "<|special_1508|>",
+    "<|special_1509|>",
+    "<|special_1510|>",
+    "<|special_1511|>",
+    "<|special_1512|>",
+    "<|special_1513|>",
+    "<|special_1514|>",
+    "<|special_1515|>",
+    "<|special_1516|>",
+    "<|special_1517|>",
+    "<|special_1518|>",
+    "<|special_1519|>",
+    "<|special_1520|>",
+    "<|special_1521|>",
+    "<|special_1522|>",
+    "<|special_1523|>",
+    "<|special_1524|>",
+    "<|special_1525|>",
+    "<|special_1526|>",
+    "<|special_1527|>",
+    "<|special_1528|>",
+    "<|special_1529|>",
+    "<|special_1530|>",
+    "<|special_1531|>",
+    "<|special_1532|>",
+    "<|special_1533|>",
+    "<|special_1534|>",
+    "<|special_1535|>",
+    "<|special_1536|>",
+    "<|special_1537|>",
+    "<|special_1538|>",
+    "<|special_1539|>",
+    "<|special_1540|>",
+    "<|special_1541|>",
+    "<|special_1542|>",
+    "<|special_1543|>",
+    "<|special_1544|>",
+    "<|special_1545|>",
+    "<|special_1546|>",
+    "<|special_1547|>",
+    "<|special_1548|>",
+    "<|special_1549|>",
+    "<|special_1550|>",
+    "<|special_1551|>",
+    "<|special_1552|>",
+    "<|special_1553|>",
+    "<|special_1554|>",
+    "<|special_1555|>",
+    "<|special_1556|>",
+    "<|special_1557|>",
+    "<|special_1558|>",
+    "<|special_1559|>",
+    "<|special_1560|>",
+    "<|special_1561|>",
+    "<|special_1562|>",
+    "<|special_1563|>",
+    "<|special_1564|>",
+    "<|special_1565|>",
+    "<|special_1566|>",
+    "<|special_1567|>",
+    "<|special_1568|>",
+    "<|special_1569|>",
+    "<|special_1570|>",
+    "<|special_1571|>",
+    "<|special_1572|>",
+    "<|special_1573|>",
+    "<|special_1574|>",
+    "<|special_1575|>",
+    "<|special_1576|>",
+    "<|special_1577|>",
+    "<|special_1578|>",
+    "<|special_1579|>",
+    "<|special_1580|>",
+    "<|special_1581|>",
+    "<|special_1582|>",
+    "<|special_1583|>",
+    "<|special_1584|>",
+    "<|special_1585|>",
+    "<|special_1586|>",
+    "<|special_1587|>",
+    "<|special_1588|>",
+    "<|special_1589|>",
+    "<|special_1590|>",
+    "<|special_1591|>",
+    "<|special_1592|>",
+    "<|special_1593|>",
+    "<|special_1594|>",
+    "<|special_1595|>",
+    "<|special_1596|>",
+    "<|special_1597|>",
+    "<|special_1598|>",
+    "<|special_1599|>",
+    "<|special_1600|>",
+    "<|special_1601|>",
+    "<|special_1602|>",
+    "<|special_1603|>",
+    "<|special_1604|>",
+    "<|special_1605|>",
+    "<|special_1606|>",
+    "<|special_1607|>",
+    "<|special_1608|>",
+    "<|special_1609|>",
+    "<|special_1610|>",
+    "<|special_1611|>",
+    "<|special_1612|>",
+    "<|special_1613|>",
+    "<|special_1614|>",
+    "<|special_1615|>",
+    "<|special_1616|>",
+    "<|special_1617|>",
+    "<|special_1618|>",
+    "<|special_1619|>",
+    "<|special_1620|>",
+    "<|special_1621|>",
+    "<|special_1622|>",
+    "<|special_1623|>",
+    "<|special_1624|>",
+    "<|special_1625|>",
+    "<|special_1626|>",
+    "<|special_1627|>",
+    "<|special_1628|>",
+    "<|special_1629|>",
+    "<|special_1630|>",
+    "<|special_1631|>",
+    "<|special_1632|>",
+    "<|special_1633|>",
+    "<|special_1634|>",
+    "<|special_1635|>",
+    "<|special_1636|>",
+    "<|special_1637|>",
+    "<|special_1638|>",
+    "<|special_1639|>",
+    "<|special_1640|>",
+    "<|special_1641|>",
+    "<|special_1642|>",
+    "<|special_1643|>",
+    "<|special_1644|>",
+    "<|special_1645|>",
+    "<|special_1646|>",
+    "<|special_1647|>",
+    "<|special_1648|>",
+    "<|special_1649|>",
+    "<|special_1650|>",
+    "<|special_1651|>",
+    "<|special_1652|>",
+    "<|special_1653|>",
+    "<|special_1654|>",
+    "<|special_1655|>",
+    "<|special_1656|>",
+    "<|special_1657|>",
+    "<|special_1658|>",
+    "<|special_1659|>",
+    "<|special_1660|>",
+    "<|special_1661|>",
+    "<|special_1662|>",
+    "<|special_1663|>",
+    "<|special_1664|>",
+    "<|special_1665|>",
+    "<|special_1666|>",
+    "<|special_1667|>",
+    "<|special_1668|>",
+    "<|special_1669|>",
+    "<|special_1670|>",
+    "<|special_1671|>",
+    "<|special_1672|>",
+    "<|special_1673|>",
+    "<|special_1674|>",
+    "<|special_1675|>",
+    "<|special_1676|>",
+    "<|special_1677|>",
+    "<|special_1678|>",
+    "<|special_1679|>",
+    "<|special_1680|>",
+    "<|special_1681|>",
+    "<|special_1682|>",
+    "<|special_1683|>",
+    "<|special_1684|>",
+    "<|special_1685|>",
+    "<|special_1686|>",
+    "<|special_1687|>",
+    "<|special_1688|>",
+    "<|special_1689|>",
+    "<|special_1690|>",
+    "<|special_1691|>",
+    "<|special_1692|>",
+    "<|special_1693|>",
+    "<|special_1694|>",
+    "<|special_1695|>",
+    "<|special_1696|>",
+    "<|special_1697|>",
+    "<|special_1698|>",
+    "<|special_1699|>",
+    "<|special_1700|>",
+    "<|special_1701|>",
+    "<|special_1702|>",
+    "<|special_1703|>",
+    "<|special_1704|>",
+    "<|special_1705|>",
+    "<|special_1706|>",
+    "<|special_1707|>",
+    "<|special_1708|>",
+    "<|special_1709|>",
+    "<|special_1710|>",
+    "<|special_1711|>",
+    "<|special_1712|>",
+    "<|special_1713|>",
+    "<|special_1714|>",
+    "<|special_1715|>",
+    "<|special_1716|>",
+    "<|special_1717|>",
+    "<|special_1718|>",
+    "<|special_1719|>",
+    "<|special_1720|>",
+    "<|special_1721|>",
+    "<|special_1722|>",
+    "<|special_1723|>",
+    "<|special_1724|>",
+    "<|special_1725|>",
+    "<|special_1726|>",
+    "<|special_1727|>",
+    "<|special_1728|>",
+    "<|special_1729|>",
+    "<|special_1730|>",
+    "<|special_1731|>",
+    "<|special_1732|>",
+    "<|special_1733|>",
+    "<|special_1734|>",
+    "<|special_1735|>",
+    "<|special_1736|>",
+    "<|special_1737|>",
+    "<|special_1738|>",
+    "<|special_1739|>",
+    "<|special_1740|>",
+    "<|special_1741|>",
+    "<|special_1742|>",
+    "<|special_1743|>",
+    "<|special_1744|>",
+    "<|special_1745|>",
+    "<|special_1746|>",
+    "<|special_1747|>",
+    "<|special_1748|>",
+    "<|special_1749|>",
+    "<|special_1750|>",
+    "<|special_1751|>",
+    "<|special_1752|>",
+    "<|special_1753|>",
+    "<|special_1754|>",
+    "<|special_1755|>",
+    "<|special_1756|>",
+    "<|special_1757|>",
+    "<|special_1758|>",
+    "<|special_1759|>",
+    "<|special_1760|>",
+    "<|special_1761|>",
+    "<|special_1762|>",
+    "<|special_1763|>",
+    "<|special_1764|>",
+    "<|special_1765|>",
+    "<|special_1766|>",
+    "<|special_1767|>",
+    "<|special_1768|>",
+    "<|special_1769|>",
+    "<|special_1770|>",
+    "<|special_1771|>",
+    "<|special_1772|>",
+    "<|special_1773|>",
+    "<|special_1774|>",
+    "<|special_1775|>",
+    "<|special_1776|>",
+    "<|special_1777|>",
+    "<|special_1778|>",
+    "<|special_1779|>",
+    "<|special_1780|>",
+    "<|special_1781|>",
+    "<|special_1782|>",
+    "<|special_1783|>",
+    "<|special_1784|>",
+    "<|special_1785|>",
+    "<|special_1786|>",
+    "<|special_1787|>",
+    "<|special_1788|>",
+    "<|special_1789|>",
+    "<|special_1790|>",
+    "<|special_1791|>",
+    "<|special_1792|>",
+    "<|special_1793|>",
+    "<|special_1794|>",
+    "<|special_1795|>",
+    "<|special_1796|>",
+    "<|special_1797|>",
+    "<|special_1798|>",
+    "<|special_1799|>",
+    "<|special_1800|>",
+    "<|special_1801|>",
+    "<|special_1802|>",
+    "<|special_1803|>",
+    "<|special_1804|>",
+    "<|special_1805|>",
+    "<|special_1806|>",
+    "<|special_1807|>",
+    "<|special_1808|>",
+    "<|special_1809|>",
+    "<|special_1810|>",
+    "<|special_1811|>",
+    "<|special_1812|>",
+    "<|special_1813|>",
+    "<|special_1814|>",
+    "<|special_1815|>",
+    "<|special_1816|>",
+    "<|special_1817|>",
+    "<|special_1818|>",
+    "<|special_1819|>",
+    "<|special_1820|>",
+    "<|special_1821|>",
+    "<|special_1822|>",
+    "<|special_1823|>",
+    "<|special_1824|>",
+    "<|special_1825|>",
+    "<|special_1826|>",
+    "<|special_1827|>",
+    "<|special_1828|>",
+    "<|special_1829|>",
+    "<|special_1830|>",
+    "<|special_1831|>",
+    "<|special_1832|>",
+    "<|special_1833|>",
+    "<|special_1834|>",
+    "<|special_1835|>",
+    "<|special_1836|>",
+    "<|special_1837|>",
+    "<|special_1838|>",
+    "<|special_1839|>",
+    "<|special_1840|>",
+    "<|special_1841|>",
+    "<|special_1842|>",
+    "<|special_1843|>",
+    "<|special_1844|>",
+    "<|special_1845|>",
+    "<|special_1846|>",
+    "<|special_1847|>",
+    "<|special_1848|>",
+    "<|special_1849|>",
+    "<|special_1850|>",
+    "<|special_1851|>",
+    "<|special_1852|>",
+    "<|special_1853|>",
+    "<|special_1854|>",
+    "<|special_1855|>",
+    "<|special_1856|>",
+    "<|special_1857|>",
+    "<|special_1858|>",
+    "<|special_1859|>",
+    "<|special_1860|>",
+    "<|special_1861|>",
+    "<|special_1862|>",
+    "<|special_1863|>",
+    "<|special_1864|>",
+    "<|special_1865|>",
+    "<|special_1866|>",
+    "<|special_1867|>",
+    "<|special_1868|>",
+    "<|special_1869|>",
+    "<|special_1870|>",
+    "<|special_1871|>",
+    "<|special_1872|>",
+    "<|special_1873|>",
+    "<|special_1874|>",
+    "<|special_1875|>",
+    "<|special_1876|>",
+    "<|special_1877|>",
+    "<|special_1878|>",
+    "<|special_1879|>",
+    "<|special_1880|>",
+    "<|special_1881|>",
+    "<|special_1882|>",
+    "<|special_1883|>",
+    "<|special_1884|>",
+    "<|special_1885|>",
+    "<|special_1886|>",
+    "<|special_1887|>",
+    "<|special_1888|>",
+    "<|special_1889|>",
+    "<|special_1890|>",
+    "<|special_1891|>",
+    "<|special_1892|>",
+    "<|special_1893|>",
+    "<|special_1894|>",
+    "<|special_1895|>",
+    "<|special_1896|>",
+    "<|special_1897|>",
+    "<|special_1898|>",
+    "<|special_1899|>",
+    "<|special_1900|>",
+    "<|special_1901|>",
+    "<|special_1902|>",
+    "<|special_1903|>",
+    "<|special_1904|>",
+    "<|special_1905|>",
+    "<|special_1906|>",
+    "<|special_1907|>",
+    "<|special_1908|>",
+    "<|special_1909|>",
+    "<|special_1910|>",
+    "<|special_1911|>",
+    "<|special_1912|>",
+    "<|special_1913|>",
+    "<|special_1914|>",
+    "<|special_1915|>",
+    "<|special_1916|>",
+    "<|special_1917|>",
+    "<|special_1918|>",
+    "<|special_1919|>",
+    "<|special_1920|>",
+    "<|special_1921|>",
+    "<|special_1922|>",
+    "<|special_1923|>",
+    "<|special_1924|>",
+    "<|special_1925|>",
+    "<|special_1926|>",
+    "<|special_1927|>",
+    "<|special_1928|>",
+    "<|special_1929|>",
+    "<|special_1930|>",
+    "<|special_1931|>",
+    "<|special_1932|>",
+    "<|special_1933|>",
+    "<|special_1934|>",
+    "<|special_1935|>",
+    "<|special_1936|>",
+    "<|special_1937|>",
+    "<|special_1938|>",
+    "<|special_1939|>",
+    "<|special_1940|>",
+    "<|special_1941|>",
+    "<|special_1942|>",
+    "<|special_1943|>",
+    "<|special_1944|>",
+    "<|special_1945|>",
+    "<|special_1946|>",
+    "<|special_1947|>",
+    "<|special_1948|>",
+    "<|special_1949|>",
+    "<|special_1950|>",
+    "<|special_1951|>",
+    "<|special_1952|>",
+    "<|special_1953|>",
+    "<|special_1954|>",
+    "<|special_1955|>",
+    "<|special_1956|>",
+    "<|special_1957|>",
+    "<|special_1958|>",
+    "<|special_1959|>",
+    "<|special_1960|>",
+    "<|special_1961|>",
+    "<|special_1962|>",
+    "<|special_1963|>",
+    "<|special_1964|>",
+    "<|special_1965|>",
+    "<|special_1966|>",
+    "<|special_1967|>",
+    "<|special_1968|>",
+    "<|special_1969|>",
+    "<|special_1970|>",
+    "<|special_1971|>",
+    "<|special_1972|>",
+    "<|special_1973|>",
+    "<|special_1974|>",
+    "<|special_1975|>",
+    "<|special_1976|>",
+    "<|special_1977|>",
+    "<|special_1978|>",
+    "<|special_1979|>",
+    "<|special_1980|>",
+    "<|special_1981|>",
+    "<|special_1982|>",
+    "<|special_1983|>",
+    "<|special_1984|>",
+    "<|special_1985|>",
+    "<|special_1986|>",
+    "<|special_1987|>",
+    "<|special_1988|>",
+    "<|special_1989|>",
+    "<|special_1990|>",
+    "<|special_1991|>",
+    "<|special_1992|>",
+    "<|special_1993|>",
+    "<|special_1994|>",
+    "<|special_1995|>",
+    "<|special_1996|>",
+    "<|special_1997|>",
+    "<|special_1998|>",
+    "<|special_1999|>",
+    "<|special_2000|>",
+    "<|special_2001|>",
+    "<|special_2002|>",
+    "<|special_2003|>",
+    "<|special_2004|>",
+    "<|special_2005|>",
+    "<|special_2006|>",
+    "<|special_2007|>",
+    "<|special_2008|>",
+    "<|special_2009|>",
+    "<|special_2010|>",
+    "<|special_2011|>",
+    "<|special_2012|>",
+    "<|special_2013|>",
+    "<|special_2014|>",
+    "<|special_2015|>",
+    "<|special_2016|>",
+    "<|special_2017|>",
+    "<|special_2018|>",
+    "<|special_2019|>",
+    "<|special_2020|>",
+    "<|special_2021|>",
+    "<|special_2022|>",
+    "<|special_2023|>",
+    "<|special_2024|>",
+    "<|special_2025|>",
+    "<|special_2026|>",
+    "<|special_2027|>",
+    "<|special_2028|>",
+    "<|special_2029|>",
+    "<|special_2030|>",
+    "<|special_2031|>",
+    "<|special_2032|>",
+    "<|special_2033|>",
+    "<|special_2034|>",
+    "<|special_2035|>",
+    "<|special_2036|>",
+    "<|special_2037|>",
+    "<|special_2038|>",
+    "<|special_2039|>",
+    "<|special_2040|>",
+    "<|special_2041|>",
+    "<|special_2042|>",
+    "<|special_2043|>",
+    "<|special_2044|>",
+    "<|special_2045|>",
+    "<|special_2046|>",
+    "<|special_2047|>",
+    "<|special_2048|>",
+    "<|special_2049|>",
+    "<|special_2050|>",
+    "<|special_2051|>",
+    "<|special_2052|>",
+    "<|special_2053|>",
+    "<|special_2054|>",
+    "<|special_2055|>",
+    "<|special_2056|>",
+    "<|special_2057|>",
+    "<|special_2058|>",
+    "<|special_2059|>",
+    "<|special_2060|>",
+    "<|special_2061|>",
+    "<|special_2062|>",
+    "<|special_2063|>",
+    "<|special_2064|>",
+    "<|special_2065|>",
+    "<|special_2066|>",
+    "<|special_2067|>",
+    "<|special_2068|>",
+    "<|special_2069|>",
+    "<|special_2070|>",
+    "<|special_2071|>",
+    "<|special_2072|>",
+    "<|special_2073|>",
+    "<|special_2074|>",
+    "<|special_2075|>",
+    "<|special_2076|>",
+    "<|special_2077|>",
+    "<|special_2078|>",
+    "<|special_2079|>",
+    "<|special_2080|>",
+    "<|special_2081|>",
+    "<|special_2082|>",
+    "<|special_2083|>",
+    "<|special_2084|>",
+    "<|special_2085|>",
+    "<|special_2086|>",
+    "<|special_2087|>",
+    "<|special_2088|>",
+    "<|special_2089|>",
+    "<|special_2090|>",
+    "<|special_2091|>",
+    "<|special_2092|>",
+    "<|special_2093|>",
+    "<|special_2094|>",
+    "<|special_2095|>",
+    "<|special_2096|>",
+    "<|special_2097|>",
+    "<|special_2098|>",
+    "<|special_2099|>",
+    "<|special_2100|>",
+    "<|special_2101|>",
+    "<|special_2102|>",
+    "<|special_2103|>",
+    "<|special_2104|>",
+    "<|special_2105|>",
+    "<|special_2106|>",
+    "<|special_2107|>",
+    "<|special_2108|>",
+    "<|special_2109|>",
+    "<|special_2110|>",
+    "<|special_2111|>",
+    "<|special_2112|>",
+    "<|special_2113|>",
+    "<|special_2114|>",
+    "<|special_2115|>",
+    "<|special_2116|>",
+    "<|special_2117|>",
+    "<|special_2118|>",
+    "<|special_2119|>",
+    "<|special_2120|>",
+    "<|special_2121|>",
+    "<|special_2122|>",
+    "<|special_2123|>",
+    "<|special_2124|>",
+    "<|special_2125|>",
+    "<|special_2126|>",
+    "<|special_2127|>",
+    "<|special_2128|>",
+    "<|special_2129|>",
+    "<|special_2130|>",
+    "<|special_2131|>",
+    "<|special_2132|>",
+    "<|special_2133|>",
+    "<|special_2134|>",
+    "<|special_2135|>",
+    "<|special_2136|>",
+    "<|special_2137|>",
+    "<|special_2138|>",
+    "<|special_2139|>",
+    "<|special_2140|>",
+    "<|special_2141|>",
+    "<|special_2142|>",
+    "<|special_2143|>",
+    "<|special_2144|>",
+    "<|special_2145|>",
+    "<|special_2146|>",
+    "<|special_2147|>",
+    "<|special_2148|>",
+    "<|special_2149|>",
+    "<|special_2150|>",
+    "<|special_2151|>",
+    "<|special_2152|>",
+    "<|special_2153|>",
+    "<|special_2154|>",
+    "<|special_2155|>",
+    "<|special_2156|>",
+    "<|special_2157|>",
+    "<|special_2158|>",
+    "<|special_2159|>",
+    "<|special_2160|>",
+    "<|special_2161|>",
+    "<|special_2162|>",
+    "<|special_2163|>",
+    "<|special_2164|>",
+    "<|special_2165|>",
+    "<|special_2166|>",
+    "<|special_2167|>",
+    "<|special_2168|>",
+    "<|special_2169|>",
+    "<|special_2170|>",
+    "<|special_2171|>",
+    "<|special_2172|>",
+    "<|special_2173|>",
+    "<|special_2174|>",
+    "<|special_2175|>",
+    "<|special_2176|>",
+    "<|special_2177|>",
+    "<|special_2178|>",
+    "<|special_2179|>",
+    "<|special_2180|>",
+    "<|special_2181|>",
+    "<|special_2182|>",
+    "<|special_2183|>",
+    "<|special_2184|>",
+    "<|special_2185|>",
+    "<|special_2186|>",
+    "<|special_2187|>",
+    "<|special_2188|>",
+    "<|special_2189|>",
+    "<|special_2190|>",
+    "<|special_2191|>",
+    "<|special_2192|>",
+    "<|special_2193|>",
+    "<|special_2194|>",
+    "<|special_2195|>",
+    "<|special_2196|>",
+    "<|special_2197|>",
+    "<|special_2198|>",
+    "<|special_2199|>",
+    "<|special_2200|>",
+    "<|special_2201|>",
+    "<|special_2202|>",
+    "<|special_2203|>",
+    "<|special_2204|>",
+    "<|special_2205|>",
+    "<|special_2206|>",
+    "<|special_2207|>",
+    "<|special_2208|>",
+    "<|special_2209|>",
+    "<|special_2210|>",
+    "<|special_2211|>",
+    "<|special_2212|>",
+    "<|special_2213|>",
+    "<|special_2214|>",
+    "<|special_2215|>",
+    "<|special_2216|>",
+    "<|special_2217|>",
+    "<|special_2218|>",
+    "<|special_2219|>",
+    "<|special_2220|>",
+    "<|special_2221|>",
+    "<|special_2222|>",
+    "<|special_2223|>",
+    "<|special_2224|>",
+    "<|special_2225|>",
+    "<|special_2226|>",
+    "<|special_2227|>",
+    "<|special_2228|>",
+    "<|special_2229|>",
+    "<|special_2230|>",
+    "<|special_2231|>",
+    "<|special_2232|>",
+    "<|special_2233|>",
+    "<|special_2234|>",
+    "<|special_2235|>",
+    "<|special_2236|>",
+    "<|special_2237|>",
+    "<|special_2238|>",
+    "<|special_2239|>",
+    "<|special_2240|>",
+    "<|special_2241|>",
+    "<|special_2242|>",
+    "<|special_2243|>",
+    "<|special_2244|>",
+    "<|special_2245|>",
+    "<|special_2246|>",
+    "<|special_2247|>",
+    "<|special_2248|>",
+    "<|special_2249|>",
+    "<|special_2250|>",
+    "<|special_2251|>",
+    "<|special_2252|>",
+    "<|special_2253|>",
+    "<|special_2254|>",
+    "<|special_2255|>",
+    "<|special_2256|>",
+    "<|special_2257|>",
+    "<|special_2258|>",
+    "<|special_2259|>",
+    "<|special_2260|>",
+    "<|special_2261|>",
+    "<|special_2262|>",
+    "<|special_2263|>",
+    "<|special_2264|>",
+    "<|special_2265|>",
+    "<|special_2266|>",
+    "<|special_2267|>",
+    "<|special_2268|>",
+    "<|special_2269|>",
+    "<|special_2270|>",
+    "<|special_2271|>",
+    "<|special_2272|>",
+    "<|special_2273|>",
+    "<|special_2274|>",
+    "<|special_2275|>",
+    "<|special_2276|>",
+    "<|special_2277|>",
+    "<|special_2278|>",
+    "<|special_2279|>",
+    "<|special_2280|>",
+    "<|special_2281|>",
+    "<|special_2282|>",
+    "<|special_2283|>",
+    "<|special_2284|>",
+    "<|special_2285|>",
+    "<|special_2286|>",
+    "<|special_2287|>",
+    "<|special_2288|>",
+    "<|special_2289|>",
+    "<|special_2290|>",
+    "<|special_2291|>",
+    "<|special_2292|>",
+    "<|special_2293|>",
+    "<|special_2294|>",
+    "<|special_2295|>",
+    "<|special_2296|>",
+    "<|special_2297|>",
+    "<|special_2298|>",
+    "<|special_2299|>",
+    "<|special_2300|>",
+    "<|special_2301|>",
+    "<|special_2302|>",
+    "<|special_2303|>",
+    "<|special_2304|>",
+    "<|special_2305|>",
+    "<|special_2306|>",
+    "<|special_2307|>",
+    "<|special_2308|>",
+    "<|special_2309|>",
+    "<|special_2310|>",
+    "<|special_2311|>",
+    "<|special_2312|>",
+    "<|special_2313|>",
+    "<|special_2314|>",
+    "<|special_2315|>",
+    "<|special_2316|>",
+    "<|special_2317|>",
+    "<|special_2318|>",
+    "<|special_2319|>",
+    "<|special_2320|>",
+    "<|special_2321|>",
+    "<|special_2322|>",
+    "<|special_2323|>",
+    "<|special_2324|>",
+    "<|special_2325|>",
+    "<|special_2326|>",
+    "<|special_2327|>",
+    "<|special_2328|>",
+    "<|special_2329|>",
+    "<|special_2330|>",
+    "<|special_2331|>",
+    "<|special_2332|>",
+    "<|special_2333|>",
+    "<|special_2334|>",
+    "<|special_2335|>",
+    "<|special_2336|>",
+    "<|special_2337|>",
+    "<|special_2338|>",
+    "<|special_2339|>",
+    "<|special_2340|>",
+    "<|special_2341|>",
+    "<|special_2342|>",
+    "<|special_2343|>",
+    "<|special_2344|>",
+    "<|special_2345|>",
+    "<|special_2346|>",
+    "<|special_2347|>",
+    "<|special_2348|>",
+    "<|special_2349|>",
+    "<|special_2350|>",
+    "<|special_2351|>",
+    "<|special_2352|>",
+    "<|special_2353|>",
+    "<|special_2354|>",
+    "<|special_2355|>",
+    "<|special_2356|>",
+    "<|special_2357|>",
+    "<|special_2358|>",
+    "<|special_2359|>",
+    "<|special_2360|>",
+    "<|special_2361|>",
+    "<|special_2362|>",
+    "<|special_2363|>",
+    "<|special_2364|>",
+    "<|special_2365|>",
+    "<|special_2366|>",
+    "<|special_2367|>",
+    "<|special_2368|>",
+    "<|special_2369|>",
+    "<|special_2370|>",
+    "<|special_2371|>",
+    "<|special_2372|>",
+    "<|special_2373|>",
+    "<|special_2374|>",
+    "<|special_2375|>",
+    "<|special_2376|>",
+    "<|special_2377|>",
+    "<|special_2378|>",
+    "<|special_2379|>",
+    "<|special_2380|>",
+    "<|special_2381|>",
+    "<|special_2382|>",
+    "<|special_2383|>",
+    "<|special_2384|>",
+    "<|special_2385|>",
+    "<|special_2386|>",
+    "<|special_2387|>",
+    "<|special_2388|>",
+    "<|special_2389|>",
+    "<|special_2390|>",
+    "<|special_2391|>",
+    "<|special_2392|>",
+    "<|special_2393|>",
+    "<|special_2394|>",
+    "<|special_2395|>",
+    "<|special_2396|>",
+    "<|special_2397|>",
+    "<|special_2398|>",
+    "<|special_2399|>",
+    "<|special_2400|>",
+    "<|special_2401|>",
+    "<|special_2402|>",
+    "<|special_2403|>",
+    "<|special_2404|>",
+    "<|special_2405|>",
+    "<|special_2406|>",
+    "<|special_2407|>",
+    "<|special_2408|>",
+    "<|special_2409|>",
+    "<|special_2410|>",
+    "<|special_2411|>",
+    "<|special_2412|>",
+    "<|special_2413|>",
+    "<|special_2414|>",
+    "<|special_2415|>",
+    "<|special_2416|>",
+    "<|special_2417|>",
+    "<|special_2418|>",
+    "<|special_2419|>",
+    "<|special_2420|>",
+    "<|special_2421|>",
+    "<|special_2422|>",
+    "<|special_2423|>",
+    "<|special_2424|>",
+    "<|special_2425|>",
+    "<|special_2426|>",
+    "<|special_2427|>",
+    "<|special_2428|>",
+    "<|special_2429|>",
+    "<|special_2430|>",
+    "<|special_2431|>",
+    "<|special_2432|>",
+    "<|special_2433|>",
+    "<|special_2434|>",
+    "<|special_2435|>",
+    "<|special_2436|>",
+    "<|special_2437|>",
+    "<|special_2438|>",
+    "<|special_2439|>",
+    "<|special_2440|>",
+    "<|special_2441|>",
+    "<|special_2442|>",
+    "<|special_2443|>",
+    "<|special_2444|>",
+    "<|special_2445|>",
+    "<|special_2446|>",
+    "<|special_2447|>",
+    "<|special_2448|>",
+    "<|special_2449|>",
+    "<|special_2450|>",
+    "<|special_2451|>",
+    "<|special_2452|>",
+    "<|special_2453|>",
+    "<|special_2454|>",
+    "<|special_2455|>",
+    "<|special_2456|>",
+    "<|special_2457|>",
+    "<|special_2458|>",
+    "<|special_2459|>",
+    "<|special_2460|>",
+    "<|special_2461|>",
+    "<|special_2462|>",
+    "<|special_2463|>",
+    "<|special_2464|>",
+    "<|special_2465|>",
+    "<|special_2466|>",
+    "<|special_2467|>",
+    "<|special_2468|>",
+    "<|special_2469|>",
+    "<|special_2470|>",
+    "<|special_2471|>",
+    "<|special_2472|>",
+    "<|special_2473|>",
+    "<|special_2474|>",
+    "<|special_2475|>",
+    "<|special_2476|>",
+    "<|special_2477|>",
+    "<|special_2478|>",
+    "<|special_2479|>",
+    "<|special_2480|>",
+    "<|special_2481|>",
+    "<|special_2482|>",
+    "<|special_2483|>",
+    "<|special_2484|>",
+    "<|special_2485|>",
+    "<|special_2486|>",
+    "<|special_2487|>",
+    "<|special_2488|>",
+    "<|special_2489|>",
+    "<|special_2490|>",
+    "<|special_2491|>",
+    "<|special_2492|>",
+    "<|special_2493|>",
+    "<|special_2494|>",
+    "<|special_2495|>",
+    "<|special_2496|>",
+    "<|special_2497|>",
+    "<|special_2498|>",
+    "<|special_2499|>",
+    "<|special_2500|>",
+    "<|special_2501|>",
+    "<|special_2502|>",
+    "<|special_2503|>",
+    "<|special_2504|>",
+    "<|special_2505|>",
+    "<|special_2506|>",
+    "<|special_2507|>",
+    "<|special_2508|>",
+    "<|special_2509|>",
+    "<|special_2510|>",
+    "<|special_2511|>",
+    "<|special_2512|>",
+    "<|special_2513|>",
+    "<|special_2514|>",
+    "<|special_2515|>",
+    "<|special_2516|>",
+    "<|special_2517|>",
+    "<|special_2518|>",
+    "<|special_2519|>",
+    "<|special_2520|>",
+    "<|special_2521|>",
+    "<|special_2522|>",
+    "<|special_2523|>",
+    "<|special_2524|>",
+    "<|special_2525|>",
+    "<|special_2526|>",
+    "<|special_2527|>",
+    "<|special_2528|>",
+    "<|special_2529|>",
+    "<|special_2530|>",
+    "<|special_2531|>",
+    "<|special_2532|>",
+    "<|special_2533|>",
+    "<|special_2534|>",
+    "<|special_2535|>",
+    "<|special_2536|>",
+    "<|special_2537|>",
+    "<|special_2538|>",
+    "<|special_2539|>",
+    "<|special_2540|>",
+    "<|special_2541|>",
+    "<|special_2542|>",
+    "<|special_2543|>",
+    "<|special_2544|>",
+    "<|special_2545|>",
+    "<|special_2546|>",
+    "<|special_2547|>",
+    "<|special_2548|>",
+    "<|special_2549|>",
+    "<|special_2550|>",
+    "<|special_2551|>",
+    "<|special_2552|>",
+    "<|special_2553|>",
+    "<|special_2554|>",
+    "<|special_2555|>",
+    "<|special_2556|>",
+    "<|special_2557|>",
+    "<|special_2558|>",
+    "<|special_2559|>",
+    "<|special_2560|>",
+    "<|special_2561|>",
+    "<|special_2562|>",
+    "<|special_2563|>",
+    "<|special_2564|>",
+    "<|special_2565|>",
+    "<|special_2566|>",
+    "<|special_2567|>",
+    "<|special_2568|>",
+    "<|special_2569|>",
+    "<|special_2570|>",
+    "<|special_2571|>",
+    "<|special_2572|>",
+    "<|special_2573|>",
+    "<|special_2574|>",
+    "<|special_2575|>",
+    "<|special_2576|>",
+    "<|special_2577|>",
+    "<|special_2578|>",
+    "<|special_2579|>",
+    "<|special_2580|>",
+    "<|special_2581|>",
+    "<|special_2582|>",
+    "<|special_2583|>",
+    "<|special_2584|>",
+    "<|special_2585|>",
+    "<|special_2586|>",
+    "<|special_2587|>",
+    "<|special_2588|>",
+    "<|special_2589|>",
+    "<|special_2590|>",
+    "<|special_2591|>",
+    "<|special_2592|>",
+    "<|special_2593|>",
+    "<|special_2594|>",
+    "<|special_2595|>",
+    "<|special_2596|>",
+    "<|special_2597|>",
+    "<|special_2598|>",
+    "<|special_2599|>",
+    "<|special_2600|>",
+    "<|special_2601|>",
+    "<|special_2602|>",
+    "<|special_2603|>",
+    "<|special_2604|>",
+    "<|special_2605|>",
+    "<|special_2606|>",
+    "<|special_2607|>",
+    "<|special_2608|>",
+    "<|special_2609|>",
+    "<|special_2610|>",
+    "<|special_2611|>",
+    "<|special_2612|>",
+    "<|special_2613|>",
+    "<|special_2614|>",
+    "<|special_2615|>",
+    "<|special_2616|>",
+    "<|special_2617|>",
+    "<|special_2618|>",
+    "<|special_2619|>",
+    "<|special_2620|>",
+    "<|special_2621|>",
+    "<|special_2622|>",
+    "<|special_2623|>",
+    "<|special_2624|>",
+    "<|special_2625|>",
+    "<|special_2626|>",
+    "<|special_2627|>",
+    "<|special_2628|>",
+    "<|special_2629|>",
+    "<|special_2630|>",
+    "<|special_2631|>",
+    "<|special_2632|>",
+    "<|special_2633|>",
+    "<|special_2634|>",
+    "<|special_2635|>",
+    "<|special_2636|>",
+    "<|special_2637|>",
+    "<|special_2638|>",
+    "<|special_2639|>",
+    "<|special_2640|>",
+    "<|special_2641|>",
+    "<|special_2642|>",
+    "<|special_2643|>",
+    "<|special_2644|>",
+    "<|special_2645|>",
+    "<|special_2646|>",
+    "<|special_2647|>",
+    "<|special_2648|>",
+    "<|special_2649|>",
+    "<|special_2650|>",
+    "<|special_2651|>",
+    "<|special_2652|>",
+    "<|special_2653|>",
+    "<|special_2654|>",
+    "<|special_2655|>",
+    "<|special_2656|>",
+    "<|special_2657|>",
+    "<|special_2658|>",
+    "<|special_2659|>",
+    "<|special_2660|>",
+    "<|special_2661|>",
+    "<|special_2662|>",
+    "<|special_2663|>",
+    "<|special_2664|>",
+    "<|special_2665|>",
+    "<|special_2666|>",
+    "<|special_2667|>",
+    "<|special_2668|>",
+    "<|special_2669|>",
+    "<|special_2670|>",
+    "<|special_2671|>",
+    "<|special_2672|>",
+    "<|special_2673|>",
+    "<|special_2674|>",
+    "<|special_2675|>",
+    "<|special_2676|>",
+    "<|special_2677|>",
+    "<|special_2678|>",
+    "<|special_2679|>",
+    "<|special_2680|>",
+    "<|special_2681|>",
+    "<|special_2682|>",
+    "<|special_2683|>",
+    "<|special_2684|>",
+    "<|special_2685|>",
+    "<|special_2686|>",
+    "<|special_2687|>",
+    "<|special_2688|>",
+    "<|special_2689|>",
+    "<|special_2690|>",
+    "<|special_2691|>",
+    "<|special_2692|>",
+    "<|special_2693|>",
+    "<|special_2694|>",
+    "<|special_2695|>",
+    "<|special_2696|>",
+    "<|special_2697|>",
+    "<|special_2698|>",
+    "<|special_2699|>",
+    "<|special_2700|>",
+    "<|special_2701|>",
+    "<|special_2702|>",
+    "<|special_2703|>",
+    "<|special_2704|>",
+    "<|special_2705|>",
+    "<|special_2706|>",
+    "<|special_2707|>",
+    "<|special_2708|>",
+    "<|special_2709|>",
+    "<|special_2710|>",
+    "<|special_2711|>",
+    "<|special_2712|>",
+    "<|special_2713|>",
+    "<|special_2714|>",
+    "<|special_2715|>",
+    "<|special_2716|>",
+    "<|special_2717|>",
+    "<|special_2718|>",
+    "<|special_2719|>",
+    "<|special_2720|>",
+    "<|special_2721|>",
+    "<|special_2722|>",
+    "<|special_2723|>",
+    "<|special_2724|>",
+    "<|special_2725|>",
+    "<|special_2726|>",
+    "<|special_2727|>",
+    "<|special_2728|>",
+    "<|special_2729|>",
+    "<|special_2730|>",
+    "<|special_2731|>",
+    "<|special_2732|>",
+    "<|special_2733|>",
+    "<|special_2734|>",
+    "<|special_2735|>",
+    "<|special_2736|>",
+    "<|special_2737|>",
+    "<|special_2738|>",
+    "<|special_2739|>",
+    "<|special_2740|>",
+    "<|special_2741|>",
+    "<|special_2742|>",
+    "<|special_2743|>",
+    "<|special_2744|>",
+    "<|special_2745|>",
+    "<|special_2746|>",
+    "<|special_2747|>",
+    "<|special_2748|>",
+    "<|special_2749|>",
+    "<|special_2750|>",
+    "<|special_2751|>",
+    "<|special_2752|>",
+    "<|special_2753|>",
+    "<|special_2754|>",
+    "<|special_2755|>",
+    "<|special_2756|>",
+    "<|special_2757|>",
+    "<|special_2758|>",
+    "<|special_2759|>",
+    "<|special_2760|>",
+    "<|special_2761|>",
+    "<|special_2762|>",
+    "<|special_2763|>",
+    "<|special_2764|>",
+    "<|special_2765|>",
+    "<|special_2766|>",
+    "<|special_2767|>",
+    "<|special_2768|>",
+    "<|special_2769|>",
+    "<|special_2770|>",
+    "<|special_2771|>",
+    "<|special_2772|>",
+    "<|special_2773|>",
+    "<|special_2774|>",
+    "<|special_2775|>",
+    "<|special_2776|>",
+    "<|special_2777|>",
+    "<|special_2778|>",
+    "<|special_2779|>",
+    "<|special_2780|>",
+    "<|special_2781|>",
+    "<|special_2782|>",
+    "<|special_2783|>",
+    "<|special_2784|>",
+    "<|special_2785|>",
+    "<|special_2786|>",
+    "<|special_2787|>",
+    "<|special_2788|>",
+    "<|special_2789|>",
+    "<|special_2790|>",
+    "<|special_2791|>",
+    "<|special_2792|>",
+    "<|special_2793|>",
+    "<|special_2794|>",
+    "<|special_2795|>",
+    "<|special_2796|>",
+    "<|special_2797|>",
+    "<|special_2798|>",
+    "<|special_2799|>",
+    "<|special_2800|>",
+    "<|special_2801|>",
+    "<|special_2802|>",
+    "<|special_2803|>",
+    "<|special_2804|>",
+    "<|special_2805|>",
+    "<|special_2806|>",
+    "<|special_2807|>",
+    "<|special_2808|>",
+    "<|special_2809|>",
+    "<|special_2810|>",
+    "<|special_2811|>",
+    "<|special_2812|>",
+    "<|special_2813|>",
+    "<|special_2814|>",
+    "<|special_2815|>",
+    "<|special_2816|>",
+    "<|special_2817|>",
+    "<|special_2818|>",
+    "<|special_2819|>",
+    "<|special_2820|>",
+    "<|special_2821|>",
+    "<|special_2822|>",
+    "<|special_2823|>",
+    "<|special_2824|>",
+    "<|special_2825|>",
+    "<|special_2826|>",
+    "<|special_2827|>",
+    "<|special_2828|>",
+    "<|special_2829|>",
+    "<|special_2830|>",
+    "<|special_2831|>",
+    "<|special_2832|>",
+    "<|special_2833|>",
+    "<|special_2834|>",
+    "<|special_2835|>",
+    "<|special_2836|>",
+    "<|special_2837|>",
+    "<|special_2838|>",
+    "<|special_2839|>",
+    "<|special_2840|>",
+    "<|special_2841|>",
+    "<|special_2842|>",
+    "<|special_2843|>",
+    "<|special_2844|>",
+    "<|special_2845|>",
+    "<|special_2846|>",
+    "<|special_2847|>",
+    "<|special_2848|>",
+    "<|special_2849|>",
+    "<|special_2850|>",
+    "<|special_2851|>",
+    "<|special_2852|>",
+    "<|special_2853|>",
+    "<|special_2854|>",
+    "<|special_2855|>",
+    "<|special_2856|>",
+    "<|special_2857|>",
+    "<|special_2858|>",
+    "<|special_2859|>",
+    "<|special_2860|>",
+    "<|special_2861|>",
+    "<|special_2862|>",
+    "<|special_2863|>",
+    "<|special_2864|>",
+    "<|special_2865|>",
+    "<|special_2866|>",
+    "<|special_2867|>",
+    "<|special_2868|>",
+    "<|special_2869|>",
+    "<|special_2870|>",
+    "<|special_2871|>",
+    "<|special_2872|>",
+    "<|special_2873|>",
+    "<|special_2874|>",
+    "<|special_2875|>",
+    "<|special_2876|>",
+    "<|special_2877|>",
+    "<|special_2878|>",
+    "<|special_2879|>",
+    "<|special_2880|>",
+    "<|special_2881|>",
+    "<|special_2882|>",
+    "<|special_2883|>",
+    "<|special_2884|>",
+    "<|special_2885|>",
+    "<|special_2886|>",
+    "<|special_2887|>",
+    "<|special_2888|>",
+    "<|special_2889|>",
+    "<|special_2890|>",
+    "<|special_2891|>",
+    "<|special_2892|>",
+    "<|special_2893|>",
+    "<|special_2894|>",
+    "<|special_2895|>",
+    "<|special_2896|>",
+    "<|special_2897|>",
+    "<|special_2898|>",
+    "<|special_2899|>",
+    "<|special_2900|>",
+    "<|special_2901|>",
+    "<|special_2902|>",
+    "<|special_2903|>",
+    "<|special_2904|>",
+    "<|special_2905|>",
+    "<|special_2906|>",
+    "<|special_2907|>",
+    "<|special_2908|>",
+    "<|special_2909|>",
+    "<|special_2910|>",
+    "<|special_2911|>",
+    "<|special_2912|>",
+    "<|special_2913|>",
+    "<|special_2914|>",
+    "<|special_2915|>",
+    "<|special_2916|>",
+    "<|special_2917|>",
+    "<|special_2918|>",
+    "<|special_2919|>",
+    "<|special_2920|>",
+    "<|special_2921|>",
+    "<|special_2922|>",
+    "<|special_2923|>",
+    "<|special_2924|>",
+    "<|special_2925|>",
+    "<|special_2926|>",
+    "<|special_2927|>",
+    "<|special_2928|>",
+    "<|special_2929|>",
+    "<|special_2930|>",
+    "<|special_2931|>",
+    "<|special_2932|>",
+    "<|special_2933|>",
+    "<|special_2934|>",
+    "<|special_2935|>",
+    "<|special_2936|>",
+    "<|special_2937|>",
+    "<|special_2938|>",
+    "<|special_2939|>",
+    "<|special_2940|>",
+    "<|special_2941|>",
+    "<|special_2942|>",
+    "<|special_2943|>",
+    "<|special_2944|>",
+    "<|special_2945|>",
+    "<|special_2946|>",
+    "<|special_2947|>",
+    "<|special_2948|>",
+    "<|special_2949|>",
+    "<|special_2950|>",
+    "<|special_2951|>",
+    "<|special_2952|>",
+    "<|special_2953|>",
+    "<|special_2954|>",
+    "<|special_2955|>",
+    "<|special_2956|>",
+    "<|special_2957|>",
+    "<|special_2958|>",
+    "<|special_2959|>",
+    "<|special_2960|>",
+    "<|special_2961|>",
+    "<|special_2962|>",
+    "<|special_2963|>",
+    "<|special_2964|>",
+    "<|special_2965|>",
+    "<|special_2966|>",
+    "<|special_2967|>",
+    "<|special_2968|>",
+    "<|special_2969|>",
+    "<|special_2970|>",
+    "<|special_2971|>",
+    "<|special_2972|>",
+    "<|special_2973|>",
+    "<|special_2974|>",
+    "<|special_2975|>",
+    "<|special_2976|>",
+    "<|special_2977|>",
+    "<|special_2978|>",
+    "<|special_2979|>",
+    "<|special_2980|>",
+    "<|special_2981|>",
+    "<|special_2982|>",
+    "<|special_2983|>",
+    "<|special_2984|>",
+    "<|special_2985|>",
+    "<|special_2986|>",
+    "<|special_2987|>",
+    "<|special_2988|>",
+    "<|special_2989|>",
+    "<|special_2990|>",
+    "<|special_2991|>",
+    "<|special_2992|>",
+    "<|special_2993|>",
+    "<|special_2994|>",
+    "<|special_2995|>",
+    "<|special_2996|>",
+    "<|special_2997|>",
+    "<|special_2998|>",
+    "<|special_2999|>",
+    "<|special_3000|>",
+    "<|special_3001|>",
+    "<|special_3002|>",
+    "<|special_3003|>",
+    "<|special_3004|>",
+    "<|special_3005|>",
+    "<|special_3006|>",
+    "<|special_3007|>",
+    "<|special_3008|>",
+    "<|special_3009|>",
+    "<|special_3010|>",
+    "<|special_3011|>",
+    "<|special_3012|>",
+    "<|special_3013|>",
+    "<|special_3014|>",
+    "<|special_3015|>",
+    "<|special_3016|>",
+    "<|special_3017|>",
+    "<|special_3018|>",
+    "<|special_3019|>",
+    "<|special_3020|>",
+    "<|special_3021|>",
+    "<|special_3022|>",
+    "<|special_3023|>",
+    "<|special_3024|>",
+    "<|special_3025|>",
+    "<|special_3026|>",
+    "<|special_3027|>",
+    "<|special_3028|>",
+    "<|special_3029|>",
+    "<|special_3030|>",
+    "<|special_3031|>",
+    "<|special_3032|>",
+    "<|special_3033|>",
+    "<|special_3034|>",
+    "<|special_3035|>",
+    "<|special_3036|>",
+    "<|special_3037|>",
+    "<|special_3038|>",
+    "<|special_3039|>",
+    "<|special_3040|>",
+    "<|special_3041|>",
+    "<|special_3042|>",
+    "<|special_3043|>",
+    "<|special_3044|>",
+    "<|special_3045|>",
+    "<|special_3046|>",
+    "<|special_3047|>",
+    "<|special_3048|>",
+    "<|special_3049|>",
+    "<|special_3050|>",
+    "<|special_3051|>",
+    "<|special_3052|>",
+    "<|special_3053|>",
+    "<|special_3054|>",
+    "<|special_3055|>",
+    "<|special_3056|>",
+    "<|special_3057|>",
+    "<|special_3058|>",
+    "<|special_3059|>",
+    "<|special_3060|>",
+    "<|special_3061|>",
+    "<|special_3062|>",
+    "<|special_3063|>",
+    "<|special_3064|>",
+    "<|special_3065|>",
+    "<|special_3066|>",
+    "<|special_3067|>",
+    "<|special_3068|>",
+    "<|special_3069|>",
+    "<|special_3070|>",
+    "<|special_3071|>",
+    "<|special_3072|>",
+    "<|special_3073|>",
+    "<|special_3074|>",
+    "<|special_3075|>",
+    "<|special_3076|>",
+    "<|special_3077|>",
+    "<|special_3078|>",
+    "<|special_3079|>",
+    "<|special_3080|>",
+    "<|special_3081|>",
+    "<|special_3082|>",
+    "<|special_3083|>",
+    "<|special_3084|>",
+    "<|special_3085|>",
+    "<|special_3086|>",
+    "<|special_3087|>",
+    "<|special_3088|>",
+    "<|special_3089|>",
+    "<|special_3090|>",
+    "<|special_3091|>",
+    "<|special_3092|>",
+    "<|special_3093|>",
+    "<|special_3094|>",
+    "<|special_3095|>",
+    "<|special_3096|>",
+    "<|special_3097|>",
+    "<|special_3098|>",
+    "<|special_3099|>",
+    "<|special_3100|>",
+    "<|special_3101|>",
+    "<|special_3102|>",
+    "<|special_3103|>",
+    "<|special_3104|>",
+    "<|special_3105|>",
+    "<|special_3106|>",
+    "<|special_3107|>",
+    "<|special_3108|>",
+    "<|special_3109|>",
+    "<|special_3110|>",
+    "<|special_3111|>",
+    "<|special_3112|>",
+    "<|special_3113|>",
+    "<|special_3114|>",
+    "<|special_3115|>",
+    "<|special_3116|>",
+    "<|special_3117|>",
+    "<|special_3118|>",
+    "<|special_3119|>",
+    "<|special_3120|>",
+    "<|special_3121|>",
+    "<|special_3122|>",
+    "<|special_3123|>",
+    "<|special_3124|>",
+    "<|special_3125|>",
+    "<|special_3126|>",
+    "<|special_3127|>",
+    "<|special_3128|>",
+    "<|special_3129|>",
+    "<|special_3130|>",
+    "<|special_3131|>",
+    "<|special_3132|>",
+    "<|special_3133|>",
+    "<|special_3134|>",
+    "<|special_3135|>",
+    "<|special_3136|>",
+    "<|special_3137|>",
+    "<|special_3138|>",
+    "<|special_3139|>",
+    "<|special_3140|>",
+    "<|special_3141|>",
+    "<|special_3142|>",
+    "<|special_3143|>",
+    "<|special_3144|>",
+    "<|special_3145|>",
+    "<|special_3146|>",
+    "<|special_3147|>",
+    "<|special_3148|>",
+    "<|special_3149|>",
+    "<|special_3150|>",
+    "<|special_3151|>",
+    "<|special_3152|>",
+    "<|special_3153|>",
+    "<|special_3154|>",
+    "<|special_3155|>",
+    "<|special_3156|>",
+    "<|special_3157|>",
+    "<|special_3158|>",
+    "<|special_3159|>",
+    "<|special_3160|>",
+    "<|special_3161|>",
+    "<|special_3162|>",
+    "<|special_3163|>",
+    "<|special_3164|>",
+    "<|special_3165|>",
+    "<|special_3166|>",
+    "<|special_3167|>",
+    "<|special_3168|>",
+    "<|special_3169|>",
+    "<|special_3170|>",
+    "<|special_3171|>",
+    "<|special_3172|>",
+    "<|special_3173|>",
+    "<|special_3174|>",
+    "<|special_3175|>",
+    "<|special_3176|>",
+    "<|special_3177|>",
+    "<|special_3178|>",
+    "<|special_3179|>",
+    "<|special_3180|>",
+    "<|special_3181|>",
+    "<|special_3182|>",
+    "<|special_3183|>",
+    "<|special_3184|>",
+    "<|special_3185|>",
+    "<|special_3186|>",
+    "<|special_3187|>",
+    "<|special_3188|>",
+    "<|special_3189|>",
+    "<|special_3190|>",
+    "<|special_3191|>",
+    "<|special_3192|>",
+    "<|special_3193|>",
+    "<|special_3194|>",
+    "<|special_3195|>",
+    "<|special_3196|>",
+    "<|special_3197|>",
+    "<|special_3198|>",
+    "<|special_3199|>",
+    "<|special_3200|>",
+    "<|special_3201|>",
+    "<|special_3202|>",
+    "<|special_3203|>",
+    "<|special_3204|>",
+    "<|special_3205|>",
+    "<|special_3206|>",
+    "<|special_3207|>",
+    "<|special_3208|>",
+    "<|special_3209|>",
+    "<|special_3210|>",
+    "<|special_3211|>",
+    "<|special_3212|>",
+    "<|special_3213|>",
+    "<|special_3214|>",
+    "<|special_3215|>",
+    "<|special_3216|>",
+    "<|special_3217|>",
+    "<|special_3218|>",
+    "<|special_3219|>",
+    "<|special_3220|>",
+    "<|special_3221|>",
+    "<|special_3222|>",
+    "<|special_3223|>",
+    "<|special_3224|>",
+    "<|special_3225|>",
+    "<|special_3226|>",
+    "<|special_3227|>",
+    "<|special_3228|>",
+    "<|special_3229|>",
+    "<|special_3230|>",
+    "<|special_3231|>",
+    "<|special_3232|>",
+    "<|special_3233|>",
+    "<|special_3234|>",
+    "<|special_3235|>",
+    "<|special_3236|>",
+    "<|special_3237|>",
+    "<|special_3238|>",
+    "<|special_3239|>",
+    "<|special_3240|>",
+    "<|special_3241|>",
+    "<|special_3242|>",
+    "<|special_3243|>",
+    "<|special_3244|>",
+    "<|special_3245|>",
+    "<|special_3246|>",
+    "<|special_3247|>",
+    "<|special_3248|>",
+    "<|special_3249|>",
+    "<|special_3250|>",
+    "<|special_3251|>",
+    "<|special_3252|>",
+    "<|special_3253|>",
+    "<|special_3254|>",
+    "<|special_3255|>",
+    "<|special_3256|>",
+    "<|special_3257|>",
+    "<|special_3258|>",
+    "<|special_3259|>",
+    "<|special_3260|>",
+    "<|special_3261|>",
+    "<|special_3262|>",
+    "<|special_3263|>",
+    "<|special_3264|>",
+    "<|special_3265|>",
+    "<|special_3266|>",
+    "<|special_3267|>",
+    "<|special_3268|>",
+    "<|special_3269|>",
+    "<|special_3270|>",
+    "<|special_3271|>",
+    "<|special_3272|>",
+    "<|special_3273|>",
+    "<|special_3274|>",
+    "<|special_3275|>",
+    "<|special_3276|>",
+    "<|special_3277|>",
+    "<|special_3278|>",
+    "<|special_3279|>",
+    "<|special_3280|>",
+    "<|special_3281|>",
+    "<|special_3282|>",
+    "<|special_3283|>",
+    "<|special_3284|>",
+    "<|special_3285|>",
+    "<|special_3286|>",
+    "<|special_3287|>",
+    "<|special_3288|>",
+    "<|special_3289|>",
+    "<|special_3290|>",
+    "<|special_3291|>",
+    "<|special_3292|>",
+    "<|special_3293|>",
+    "<|special_3294|>",
+    "<|special_3295|>",
+    "<|special_3296|>",
+    "<|special_3297|>",
+    "<|special_3298|>",
+    "<|special_3299|>",
+    "<|special_3300|>",
+    "<|special_3301|>",
+    "<|special_3302|>",
+    "<|special_3303|>",
+    "<|special_3304|>",
+    "<|special_3305|>",
+    "<|special_3306|>",
+    "<|special_3307|>",
+    "<|special_3308|>",
+    "<|special_3309|>",
+    "<|special_3310|>",
+    "<|special_3311|>",
+    "<|special_3312|>",
+    "<|special_3313|>",
+    "<|special_3314|>",
+    "<|special_3315|>",
+    "<|special_3316|>",
+    "<|special_3317|>",
+    "<|special_3318|>",
+    "<|special_3319|>",
+    "<|special_3320|>",
+    "<|special_3321|>",
+    "<|special_3322|>",
+    "<|special_3323|>",
+    "<|special_3324|>",
+    "<|special_3325|>",
+    "<|special_3326|>",
+    "<|special_3327|>",
+    "<|special_3328|>",
+    "<|special_3329|>",
+    "<|special_3330|>",
+    "<|special_3331|>",
+    "<|special_3332|>",
+    "<|special_3333|>",
+    "<|special_3334|>",
+    "<|special_3335|>",
+    "<|special_3336|>",
+    "<|special_3337|>",
+    "<|special_3338|>",
+    "<|special_3339|>",
+    "<|special_3340|>",
+    "<|special_3341|>",
+    "<|special_3342|>",
+    "<|special_3343|>",
+    "<|special_3344|>",
+    "<|special_3345|>",
+    "<|special_3346|>",
+    "<|special_3347|>",
+    "<|special_3348|>",
+    "<|special_3349|>",
+    "<|special_3350|>",
+    "<|special_3351|>",
+    "<|special_3352|>",
+    "<|special_3353|>",
+    "<|special_3354|>",
+    "<|special_3355|>",
+    "<|special_3356|>",
+    "<|special_3357|>",
+    "<|special_3358|>",
+    "<|special_3359|>",
+    "<|special_3360|>",
+    "<|special_3361|>",
+    "<|special_3362|>",
+    "<|special_3363|>",
+    "<|special_3364|>",
+    "<|special_3365|>",
+    "<|special_3366|>",
+    "<|special_3367|>",
+    "<|special_3368|>",
+    "<|special_3369|>",
+    "<|special_3370|>",
+    "<|special_3371|>",
+    "<|special_3372|>",
+    "<|special_3373|>",
+    "<|special_3374|>",
+    "<|special_3375|>",
+    "<|special_3376|>",
+    "<|special_3377|>",
+    "<|special_3378|>",
+    "<|special_3379|>",
+    "<|special_3380|>",
+    "<|special_3381|>",
+    "<|special_3382|>",
+    "<|special_3383|>",
+    "<|special_3384|>",
+    "<|special_3385|>",
+    "<|special_3386|>",
+    "<|special_3387|>",
+    "<|special_3388|>",
+    "<|special_3389|>",
+    "<|special_3390|>",
+    "<|special_3391|>",
+    "<|special_3392|>",
+    "<|special_3393|>",
+    "<|special_3394|>",
+    "<|special_3395|>",
+    "<|special_3396|>",
+    "<|special_3397|>",
+    "<|special_3398|>",
+    "<|special_3399|>",
+    "<|special_3400|>",
+    "<|special_3401|>",
+    "<|special_3402|>",
+    "<|special_3403|>",
+    "<|special_3404|>",
+    "<|special_3405|>",
+    "<|special_3406|>",
+    "<|special_3407|>",
+    "<|special_3408|>",
+    "<|special_3409|>",
+    "<|special_3410|>",
+    "<|special_3411|>",
+    "<|special_3412|>",
+    "<|special_3413|>",
+    "<|special_3414|>",
+    "<|special_3415|>",
+    "<|special_3416|>",
+    "<|special_3417|>",
+    "<|special_3418|>",
+    "<|special_3419|>",
+    "<|special_3420|>",
+    "<|special_3421|>",
+    "<|special_3422|>",
+    "<|special_3423|>",
+    "<|special_3424|>",
+    "<|special_3425|>",
+    "<|special_3426|>",
+    "<|special_3427|>",
+    "<|special_3428|>",
+    "<|special_3429|>",
+    "<|special_3430|>",
+    "<|special_3431|>",
+    "<|special_3432|>",
+    "<|special_3433|>",
+    "<|special_3434|>",
+    "<|special_3435|>",
+    "<|special_3436|>",
+    "<|special_3437|>",
+    "<|special_3438|>",
+    "<|special_3439|>",
+    "<|special_3440|>",
+    "<|special_3441|>",
+    "<|special_3442|>",
+    "<|special_3443|>",
+    "<|special_3444|>",
+    "<|special_3445|>",
+    "<|special_3446|>",
+    "<|special_3447|>",
+    "<|special_3448|>",
+    "<|special_3449|>",
+    "<|special_3450|>",
+    "<|special_3451|>",
+    "<|special_3452|>",
+    "<|special_3453|>",
+    "<|special_3454|>",
+    "<|special_3455|>",
+    "<|special_3456|>",
+    "<|special_3457|>",
+    "<|special_3458|>",
+    "<|special_3459|>",
+    "<|special_3460|>",
+    "<|special_3461|>",
+    "<|special_3462|>",
+    "<|special_3463|>",
+    "<|special_3464|>",
+    "<|special_3465|>",
+    "<|special_3466|>",
+    "<|special_3467|>",
+    "<|special_3468|>",
+    "<|special_3469|>",
+    "<|special_3470|>",
+    "<|special_3471|>",
+    "<|special_3472|>",
+    "<|special_3473|>",
+    "<|special_3474|>",
+    "<|special_3475|>",
+    "<|special_3476|>",
+    "<|special_3477|>",
+    "<|special_3478|>",
+    "<|special_3479|>",
+    "<|special_3480|>",
+    "<|special_3481|>",
+    "<|special_3482|>",
+    "<|special_3483|>",
+    "<|special_3484|>",
+    "<|special_3485|>",
+    "<|special_3486|>",
+    "<|special_3487|>",
+    "<|special_3488|>",
+    "<|special_3489|>",
+    "<|special_3490|>",
+    "<|special_3491|>",
+    "<|special_3492|>",
+    "<|special_3493|>",
+    "<|special_3494|>",
+    "<|special_3495|>",
+    "<|special_3496|>",
+    "<|special_3497|>",
+    "<|special_3498|>",
+    "<|special_3499|>",
+    "<|special_3500|>",
+    "<|special_3501|>",
+    "<|special_3502|>",
+    "<|special_3503|>",
+    "<|special_3504|>",
+    "<|special_3505|>",
+    "<|special_3506|>",
+    "<|special_3507|>",
+    "<|special_3508|>",
+    "<|special_3509|>",
+    "<|special_3510|>",
+    "<|special_3511|>",
+    "<|special_3512|>",
+    "<|special_3513|>",
+    "<|special_3514|>",
+    "<|special_3515|>",
+    "<|special_3516|>",
+    "<|special_3517|>",
+    "<|special_3518|>",
+    "<|special_3519|>",
+    "<|special_3520|>",
+    "<|special_3521|>",
+    "<|special_3522|>",
+    "<|special_3523|>",
+    "<|special_3524|>",
+    "<|special_3525|>",
+    "<|special_3526|>",
+    "<|special_3527|>",
+    "<|special_3528|>",
+    "<|special_3529|>",
+    "<|special_3530|>",
+    "<|special_3531|>",
+    "<|special_3532|>",
+    "<|special_3533|>",
+    "<|special_3534|>",
+    "<|special_3535|>",
+    "<|special_3536|>",
+    "<|special_3537|>",
+    "<|special_3538|>",
+    "<|special_3539|>",
+    "<|special_3540|>",
+    "<|special_3541|>",
+    "<|special_3542|>",
+    "<|special_3543|>",
+    "<|special_3544|>",
+    "<|special_3545|>",
+    "<|special_3546|>",
+    "<|special_3547|>",
+    "<|special_3548|>",
+    "<|special_3549|>",
+    "<|special_3550|>",
+    "<|special_3551|>",
+    "<|special_3552|>",
+    "<|special_3553|>",
+    "<|special_3554|>",
+    "<|special_3555|>",
+    "<|special_3556|>",
+    "<|special_3557|>",
+    "<|special_3558|>",
+    "<|special_3559|>",
+    "<|special_3560|>",
+    "<|special_3561|>",
+    "<|special_3562|>",
+    "<|special_3563|>",
+    "<|special_3564|>",
+    "<|special_3565|>",
+    "<|special_3566|>",
+    "<|special_3567|>",
+    "<|special_3568|>",
+    "<|special_3569|>",
+    "<|special_3570|>",
+    "<|special_3571|>",
+    "<|special_3572|>",
+    "<|special_3573|>",
+    "<|special_3574|>",
+    "<|special_3575|>",
+    "<|special_3576|>",
+    "<|special_3577|>",
+    "<|special_3578|>",
+    "<|special_3579|>",
+    "<|special_3580|>",
+    "<|special_3581|>",
+    "<|special_3582|>",
+    "<|special_3583|>",
+    "<|special_3584|>",
+    "<|special_3585|>",
+    "<|special_3586|>",
+    "<|special_3587|>",
+    "<|special_3588|>",
+    "<|special_3589|>",
+    "<|special_3590|>",
+    "<|special_3591|>",
+    "<|special_3592|>",
+    "<|special_3593|>",
+    "<|special_3594|>",
+    "<|special_3595|>",
+    "<|special_3596|>",
+    "<|special_3597|>",
+    "<|special_3598|>",
+    "<|special_3599|>",
+    "<|special_3600|>",
+    "<|special_3601|>",
+    "<|special_3602|>",
+    "<|special_3603|>",
+    "<|special_3604|>",
+    "<|special_3605|>",
+    "<|special_3606|>",
+    "<|special_3607|>",
+    "<|special_3608|>",
+    "<|special_3609|>",
+    "<|special_3610|>",
+    "<|special_3611|>",
+    "<|special_3612|>",
+    "<|special_3613|>",
+    "<|special_3614|>",
+    "<|special_3615|>",
+    "<|special_3616|>",
+    "<|special_3617|>",
+    "<|special_3618|>",
+    "<|special_3619|>",
+    "<|special_3620|>",
+    "<|special_3621|>",
+    "<|special_3622|>",
+    "<|special_3623|>",
+    "<|special_3624|>",
+    "<|special_3625|>",
+    "<|special_3626|>",
+    "<|special_3627|>",
+    "<|special_3628|>",
+    "<|special_3629|>",
+    "<|special_3630|>",
+    "<|special_3631|>",
+    "<|special_3632|>",
+    "<|special_3633|>",
+    "<|special_3634|>",
+    "<|special_3635|>",
+    "<|special_3636|>",
+    "<|special_3637|>",
+    "<|special_3638|>",
+    "<|special_3639|>",
+    "<|special_3640|>",
+    "<|special_3641|>",
+    "<|special_3642|>",
+    "<|special_3643|>",
+    "<|special_3644|>",
+    "<|special_3645|>",
+    "<|special_3646|>",
+    "<|special_3647|>",
+    "<|special_3648|>",
+    "<|special_3649|>",
+    "<|special_3650|>",
+    "<|special_3651|>",
+    "<|special_3652|>",
+    "<|special_3653|>",
+    "<|special_3654|>",
+    "<|special_3655|>",
+    "<|special_3656|>",
+    "<|special_3657|>",
+    "<|special_3658|>",
+    "<|special_3659|>",
+    "<|special_3660|>",
+    "<|special_3661|>",
+    "<|special_3662|>",
+    "<|special_3663|>",
+    "<|special_3664|>",
+    "<|special_3665|>",
+    "<|special_3666|>",
+    "<|special_3667|>",
+    "<|special_3668|>",
+    "<|special_3669|>",
+    "<|special_3670|>",
+    "<|special_3671|>",
+    "<|special_3672|>",
+    "<|special_3673|>",
+    "<|special_3674|>",
+    "<|special_3675|>",
+    "<|special_3676|>",
+    "<|special_3677|>",
+    "<|special_3678|>",
+    "<|special_3679|>",
+    "<|special_3680|>",
+    "<|special_3681|>",
+    "<|special_3682|>",
+    "<|special_3683|>",
+    "<|special_3684|>",
+    "<|special_3685|>",
+    "<|special_3686|>",
+    "<|special_3687|>",
+    "<|special_3688|>",
+    "<|special_3689|>",
+    "<|special_3690|>",
+    "<|special_3691|>",
+    "<|special_3692|>",
+    "<|special_3693|>",
+    "<|special_3694|>",
+    "<|special_3695|>",
+    "<|special_3696|>",
+    "<|special_3697|>",
+    "<|special_3698|>",
+    "<|special_3699|>",
+    "<|special_3700|>",
+    "<|special_3701|>",
+    "<|special_3702|>",
+    "<|special_3703|>",
+    "<|special_3704|>",
+    "<|special_3705|>",
+    "<|special_3706|>",
+    "<|special_3707|>",
+    "<|special_3708|>",
+    "<|special_3709|>",
+    "<|special_3710|>",
+    "<|special_3711|>",
+    "<|special_3712|>",
+    "<|special_3713|>",
+    "<|special_3714|>",
+    "<|special_3715|>",
+    "<|special_3716|>",
+    "<|special_3717|>",
+    "<|special_3718|>",
+    "<|special_3719|>",
+    "<|special_3720|>",
+    "<|special_3721|>",
+    "<|special_3722|>",
+    "<|special_3723|>",
+    "<|special_3724|>",
+    "<|special_3725|>",
+    "<|special_3726|>",
+    "<|special_3727|>",
+    "<|special_3728|>",
+    "<|special_3729|>",
+    "<|special_3730|>",
+    "<|special_3731|>",
+    "<|special_3732|>",
+    "<|special_3733|>",
+    "<|special_3734|>",
+    "<|special_3735|>",
+    "<|special_3736|>",
+    "<|special_3737|>",
+    "<|special_3738|>",
+    "<|special_3739|>",
+    "<|special_3740|>",
+    "<|special_3741|>",
+    "<|special_3742|>",
+    "<|special_3743|>",
+    "<|special_3744|>",
+    "<|special_3745|>",
+    "<|special_3746|>",
+    "<|special_3747|>",
+    "<|special_3748|>",
+    "<|special_3749|>",
+    "<|special_3750|>",
+    "<|special_3751|>",
+    "<|special_3752|>",
+    "<|special_3753|>",
+    "<|special_3754|>",
+    "<|special_3755|>",
+    "<|special_3756|>",
+    "<|special_3757|>",
+    "<|special_3758|>",
+    "<|special_3759|>",
+    "<|special_3760|>",
+    "<|special_3761|>",
+    "<|special_3762|>",
+    "<|special_3763|>",
+    "<|special_3764|>",
+    "<|special_3765|>",
+    "<|special_3766|>",
+    "<|special_3767|>",
+    "<|special_3768|>",
+    "<|special_3769|>",
+    "<|special_3770|>",
+    "<|special_3771|>",
+    "<|special_3772|>",
+    "<|special_3773|>",
+    "<|special_3774|>",
+    "<|special_3775|>",
+    "<|special_3776|>",
+    "<|special_3777|>",
+    "<|special_3778|>",
+    "<|special_3779|>",
+    "<|special_3780|>",
+    "<|special_3781|>",
+    "<|special_3782|>",
+    "<|special_3783|>",
+    "<|special_3784|>",
+    "<|special_3785|>",
+    "<|special_3786|>",
+    "<|special_3787|>",
+    "<|special_3788|>",
+    "<|special_3789|>",
+    "<|special_3790|>",
+    "<|special_3791|>",
+    "<|special_3792|>",
+    "<|special_3793|>",
+    "<|special_3794|>",
+    "<|special_3795|>",
+    "<|special_3796|>",
+    "<|special_3797|>",
+    "<|special_3798|>",
+    "<|special_3799|>",
+    "<|special_3800|>",
+    "<|special_3801|>",
+    "<|special_3802|>",
+    "<|special_3803|>",
+    "<|special_3804|>",
+    "<|special_3805|>",
+    "<|special_3806|>",
+    "<|special_3807|>",
+    "<|special_3808|>",
+    "<|special_3809|>",
+    "<|special_3810|>",
+    "<|special_3811|>",
+    "<|special_3812|>",
+    "<|special_3813|>",
+    "<|special_3814|>",
+    "<|special_3815|>",
+    "<|special_3816|>",
+    "<|special_3817|>",
+    "<|special_3818|>",
+    "<|special_3819|>",
+    "<|special_3820|>",
+    "<|special_3821|>",
+    "<|special_3822|>",
+    "<|special_3823|>",
+    "<|special_3824|>",
+    "<|special_3825|>",
+    "<|special_3826|>",
+    "<|special_3827|>",
+    "<|special_3828|>",
+    "<|special_3829|>",
+    "<|special_3830|>",
+    "<|special_3831|>",
+    "<|special_3832|>",
+    "<|special_3833|>",
+    "<|special_3834|>",
+    "<|special_3835|>",
+    "<|special_3836|>",
+    "<|special_3837|>",
+    "<|special_3838|>",
+    "<|special_3839|>",
+    "<|special_3840|>",
+    "<|special_3841|>",
+    "<|special_3842|>",
+    "<|special_3843|>",
+    "<|special_3844|>",
+    "<|special_3845|>",
+    "<|special_3846|>",
+    "<|special_3847|>",
+    "<|special_3848|>",
+    "<|special_3849|>",
+    "<|special_3850|>",
+    "<|special_3851|>",
+    "<|special_3852|>",
+    "<|special_3853|>",
+    "<|special_3854|>",
+    "<|special_3855|>",
+    "<|special_3856|>",
+    "<|special_3857|>",
+    "<|special_3858|>",
+    "<|special_3859|>",
+    "<|special_3860|>",
+    "<|special_3861|>",
+    "<|special_3862|>",
+    "<|special_3863|>",
+    "<|special_3864|>",
+    "<|special_3865|>",
+    "<|special_3866|>",
+    "<|special_3867|>",
+    "<|special_3868|>",
+    "<|special_3869|>",
+    "<|special_3870|>",
+    "<|special_3871|>",
+    "<|special_3872|>",
+    "<|special_3873|>",
+    "<|special_3874|>",
+    "<|special_3875|>",
+    "<|special_3876|>",
+    "<|special_3877|>",
+    "<|special_3878|>",
+    "<|special_3879|>",
+    "<|special_3880|>",
+    "<|special_3881|>",
+    "<|special_3882|>",
+    "<|special_3883|>",
+    "<|special_3884|>",
+    "<|special_3885|>",
+    "<|special_3886|>",
+    "<|special_3887|>",
+    "<|special_3888|>",
+    "<|special_3889|>",
+    "<|special_3890|>",
+    "<|special_3891|>",
+    "<|special_3892|>",
+    "<|special_3893|>",
+    "<|special_3894|>",
+    "<|special_3895|>",
+    "<|special_3896|>",
+    "<|special_3897|>",
+    "<|special_3898|>",
+    "<|special_3899|>",
+    "<|special_3900|>",
+    "<|special_3901|>",
+    "<|special_3902|>",
+    "<|special_3903|>",
+    "<|special_3904|>",
+    "<|special_3905|>",
+    "<|special_3906|>",
+    "<|special_3907|>",
+    "<|special_3908|>",
+    "<|special_3909|>",
+    "<|special_3910|>",
+    "<|special_3911|>",
+    "<|special_3912|>",
+    "<|special_3913|>",
+    "<|special_3914|>",
+    "<|special_3915|>",
+    "<|special_3916|>",
+    "<|special_3917|>",
+    "<|special_3918|>",
+    "<|special_3919|>",
+    "<|special_3920|>",
+    "<|special_3921|>",
+    "<|special_3922|>",
+    "<|special_3923|>",
+    "<|special_3924|>",
+    "<|special_3925|>",
+    "<|special_3926|>",
+    "<|special_3927|>",
+    "<|special_3928|>",
+    "<|special_3929|>",
+    "<|special_3930|>",
+    "<|special_3931|>",
+    "<|special_3932|>",
+    "<|special_3933|>",
+    "<|special_3934|>",
+    "<|special_3935|>",
+    "<|special_3936|>",
+    "<|special_3937|>",
+    "<|special_3938|>",
+    "<|special_3939|>",
+    "<|special_3940|>",
+    "<|special_3941|>",
+    "<|special_3942|>",
+    "<|special_3943|>",
+    "<|special_3944|>",
+    "<|special_3945|>",
+    "<|special_3946|>",
+    "<|special_3947|>",
+    "<|special_3948|>",
+    "<|special_3949|>",
+    "<|special_3950|>",
+    "<|special_3951|>",
+    "<|special_3952|>",
+    "<|special_3953|>",
+    "<|special_3954|>",
+    "<|special_3955|>",
+    "<|special_3956|>",
+    "<|special_3957|>",
+    "<|special_3958|>",
+    "<|special_3959|>",
+    "<|special_3960|>",
+    "<|special_3961|>",
+    "<|special_3962|>",
+    "<|special_3963|>",
+    "<|special_3964|>",
+    "<|special_3965|>",
+    "<|special_3966|>",
+    "<|special_3967|>",
+    "<|special_3968|>",
+    "<|special_3969|>",
+    "<|special_3970|>",
+    "<|special_3971|>",
+    "<|special_3972|>",
+    "<|special_3973|>",
+    "<|special_3974|>",
+    "<|special_3975|>",
+    "<|special_3976|>",
+    "<|special_3977|>",
+    "<|special_3978|>",
+    "<|special_3979|>",
+    "<|special_3980|>",
+    "<|special_3981|>",
+    "<|special_3982|>",
+    "<|special_3983|>",
+    "<|special_3984|>",
+    "<|special_3985|>",
+    "<|special_3986|>",
+    "<|special_3987|>",
+    "<|special_3988|>",
+    "<|special_3989|>",
+    "<|special_3990|>",
+    "<|special_3991|>",
+    "<|special_3992|>",
+    "<|special_3993|>",
+    "<|special_3994|>",
+    "<|special_3995|>",
+    "<|special_3996|>",
+    "<|special_3997|>",
+    "<|special_3998|>",
+    "<|special_3999|>",
+    "<|special_4000|>",
+    "<|special_4001|>",
+    "<|special_4002|>",
+    "<|special_4003|>",
+    "<|special_4004|>",
+    "<|special_4005|>",
+    "<|special_4006|>",
+    "<|special_4007|>",
+    "<|special_4008|>",
+    "<|special_4009|>",
+    "<|special_4010|>",
+    "<|special_4011|>",
+    "<|special_4012|>",
+    "<|special_4013|>",
+    "<|special_4014|>",
+    "<|special_4015|>",
+    "<|special_4016|>",
+    "<|special_4017|>",
+    "<|special_4018|>",
+    "<|special_4019|>",
+    "<|special_4020|>",
+    "<|special_4021|>",
+    "<|special_4022|>",
+    "<|special_4023|>",
+    "<|special_4024|>",
+    "<|special_4025|>",
+    "<|special_4026|>",
+    "<|special_4027|>",
+    "<|special_4028|>",
+    "<|special_4029|>",
+    "<|special_4030|>",
+    "<|special_4031|>",
+    "<|special_4032|>",
+    "<|special_4033|>",
+    "<|special_4034|>",
+    "<|special_4035|>",
+    "<|special_4036|>",
+    "<|special_4037|>",
+    "<|special_4038|>",
+    "<|special_4039|>",
+    "<|special_4040|>",
+    "<|special_4041|>",
+    "<|special_4042|>",
+    "<|special_4043|>",
+    "<|special_4044|>",
+    "<|special_4045|>",
+    "<|special_4046|>",
+    "<|special_4047|>",
+    "<|special_4048|>",
+    "<|special_4049|>",
+    "<|special_4050|>",
+    "<|special_4051|>",
+    "<|special_4052|>",
+    "<|special_4053|>",
+    "<|special_4054|>",
+    "<|special_4055|>",
+    "<|special_4056|>",
+    "<|special_4057|>",
+    "<|special_4058|>",
+    "<|special_4059|>",
+    "<|special_4060|>",
+    "<|special_4061|>",
+    "<|special_4062|>",
+    "<|special_4063|>",
+    "<|special_4064|>",
+    "<|special_4065|>",
+    "<|special_4066|>",
+    "<|special_4067|>",
+    "<|special_4068|>",
+    "<|special_4069|>",
+    "<|special_4070|>",
+    "<|special_4071|>",
+    "<|special_4072|>",
+    "<|special_4073|>",
+    "<|special_4074|>",
+    "<|special_4075|>",
+    "<|special_4076|>",
+    "<|special_4077|>",
+    "<|special_4078|>",
+    "<|special_4079|>",
+    "<|special_4080|>",
+    "<|special_4081|>",
+    "<|special_4082|>",
+    "<|special_4083|>",
+    "<|special_4084|>",
+    "<|special_4085|>",
+    "<|special_4086|>",
+    "<|special_4087|>",
+    "<|special_4088|>",
+    "<|special_4089|>",
+    "<|special_4090|>",
+    "<|special_4091|>",
+    "<|special_4092|>",
+    "<|special_4093|>",
+    "<|special_4094|>",
+    "<|special_4095|>"
+  ],
+  "bos_token": {
+    "content": "<|startoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7b7cb72447bbe1f8d856adaf34933acfdedfa89c7fe1d95d0fdabedf9d19a94
+size 16473585