Initial release of Luciole-23B-Instruct-1.1

Co-authored-by: Kate Thompson <Kqte@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,229 @@
+---
+license: apache-2.0
+language:
+- fr
+- en
+pipeline_tag: text-generation
+tags:
+- openllm-france
+base_model:
+- OpenLLM-France/Luciole-23B-Base
+---
+![luciole_logo.png](luciole_logo.png)
+
+# Model Card for Luciole-23B-Instruct-1.1
+
+* [Model Description](#model-description)
+  * [Bias, Risks, and Limitations](#bias-risks-and-limitations)
+  * [Recommendations](#recommendations)
+* [Training Details](#training-details)
+  * [Training Data](#training-data)
+  * [Instruction template](#instruction-template)
+  * [Training Procedure](#training-procedure)
+* [Evaluation](#evaluation)
+* [Testing the model](#testing-the-model)
+  * [Test with ollama](#test-with-ollama)
+  * [Test with vLLM](#test-with-vllm)
+* [Citation](#citation)
+* [Acknowledgements](#acknowledgements)
+* [Contact](#contact)
+
+## Model Description
+
+Luciole-23B-Instruct-1.1 is a fine-tuned and aligned version of [Luciole-23B-Base](https://huggingface.co/OpenLLM-France/Luciole-23B-Base), an open-source, multilingual causal language model created by OpenLLM-France. Luciole-23B-Instruct-1.1  was developed by [LINAGORA](https://labs.linagora.com/) and the [OpenLLM-France](https://openllm-france.fr/) consortium as a part of the OpenLLM France project, funded by [BPI France](https://www.bpifrance.fr/) through the [France 2030](https://www.info.gouv.fr/grand-dossier/france-2030) program.
+
+Training of Luciole-23B-Instruct-1.1 was conducted on Jean Zay in three phases: (i) a supervised fine-tuning (SFT) phase on instruction data with thinking traces, (ii) an SFT phase on instruction data without thinking traces, and (iii) a final preference alignment phase using Direct Preference Optimization (DPO). The training data covers topics in math, science, coding, general chat, RAG and translation. 
+
+Note that Luciole-23B-Instruct-1.1 is only the first iteration of post-trained models based on [Luciole-23B-Base](https://huggingface.co/OpenLLM-France/Luciole-23B-Base). Development of the Luciole models is an active, ongoing project and further iterations, including models trained on higher proportions of French data, are planned. In the spirit of open-source, we share the model weights and training recipes to facilitate research and to provide open source building blocks for other training projects.   
+
+If you are interested in contributing to the Luciole project, contact us at contact@openllm-france.fr. 
+
+* License: [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
+* Training repository: [Luciole-Training](https://github.com/OpenLLM-France/Luciole-Training/tree/main/finetune)
+* Training data: coming soon.
+* Technical report: coming soon.
+
+
+### Bias, Risks, and Limitations
+Luciole-23B-Instruct-1.1 is the result of a first phase of fine-tuning and alignment to human preferences. Efforts to improve the model are active and ongoing and the model should be thoroughly tested for target use cases before being incorporated in industrial pipelines. Training has mostly focused on instruction following, and additional training would be necessary for tasks specifically focused on code generation or mathematical problem solving. It is also susceptible to hallucinations; that is, producing false answers that result from its training on massive amounts of diverse text. 
+
+Luciole-23B-Instruct-1.1 was post-trained almost entirely on English data (in contrast to its base model, [OpenLLM-France/Luciole-23B-Base](https://huggingface.co/OpenLLM-France/Luciole-1B-Base), which was trained on roughly 30% French data). Future post-training phases will focus specifically on increasing the proportion of French data and testing the impact that this has on model performance. 
+
+Due to its size, Luciole-23B-Instruct-1.1 is limited in the information that it can memorize; its ability to produce correct answers could be improved by implementing the model in a retrieval augmented generation pipeline.
+
+Finally,  Luciole-23B-Instruct-1.1 was trained on sequences of 16,384 tokens (in contrast to its base model, whose context window was extended to 131,000 tokens).
+
+### Recommendations
+* Further train Luciole-23B-Instruct-1.1 for specific use cases.
+* Integrate the model in a RAG pipeline to augment its knowledge base.
+* Extend training with longer sequences to increase context length.
+
+
+## Training details
+
+### Training data
+The post-training data and dataset description will be published soon.
+
+### Instruction template
+Luciole-23B-Instruct-1.1 was trained on the chat template inspired from [Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B/). In our chat template, we also use a default system prompt.
+
+
+An example: 
+
+```
+from transformers import AutoTokenizer
+tokenizer = AutoTokenizer.from_pretrained("OpenLLM-France/Luciole-23B-Instruct-1.1")
+chat = [
+  {
+    "content": "Who was Molière?",
+    "role": "user"
+  },
+  {
+    "content": "Molière was a 17th-century French playwright, actor, and comedian.",
+    "role": "assistant"
+  },
+  {
+    "content": "What are his best-known works?",
+    "role": "user"
+  }
+]
+print(tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True))
+```
+gives the output
+```
+<|im_start|>system
+You are a helpful AI assistant named Luciole, trained by LINAGORA and OpenLLM France.<|im_end|>
+<|im_start|>user
+Who was Molière?<|im_end|>
+<|im_start|>assistant
+Molière was a 17th-century French playwright, actor, and comedian.<|im_end|>
+<|im_start|>user
+What are his best-known works?<|im_end|>
+<|im_start|>assistant
+```
+
+### Training procedure
+
+Luciole-23B-Instruct-1.1 was trained in three phases. The model was first finetuned on thinking data, and then on data without thinking traces. The resulting instruct model was aligned during a final DPO finetuning phase.
+
+
+|                            | **SFT with thinking** | **SFT without thinking**  | **DPO without thinking** |  
+|----------------------------|:----------:|:-----------:|:------------:|
+| **Num training samples**   |     2.6M     |     2.1M   |      284K   |  
+| **Num epochs**             |       4      |      4     |      1      |  
+| **Max learning rate**      |     2.0e-5     |    2.0e-5       |      3.0e-7      |  
+| **Min learning rate**      |      2.0e-6  |       2.0e-6    |     0       |  
+| **LR scheduler**           |      cosine    |      cosine     |     linear       |  
+| **Max sequence length**    |     16384    |     16384      |    16384        | 
+| **Batch size**             |    2048      |      2048     |     128       |  
+| **DPO β**            |     n/a     |     n/a      |     5       |  
+
+Links to the interim models: [SFT with thinking](https://dl.labs.linagora.com/files/models/OpenLLM-France/Luciole-23B-SFT-Thinking-1.1/), [SFT without thinking](https://dl.labs.linagora.com/files/models/OpenLLM-France/Luciole-23B-SFT-1.1/)
+<!--<sup>*</sup>As noted above, while Luciole-23B-SFT-1.1 is trained on sequences of 16384 tokens, it maintains the capacity of the original base model, [OpenLLM-France/Luciole-1B-Base](https://huggingface.co/OpenLLM-France/Luciole-1B-Base), to handle context sizes of up to 131K tokens.-->
+
+## Evaluation
+Overall, Luciole-23B-Instruct-1.1 performs comparably with other open source instruct models of the same size. The model performs particularly well on instruction following benchmarks. Its performance on French language instruction following (ifeval-fr) is notable since Luciole-23B-Instruct-1.1 was finetuned on mostly English language data. This is likely the effect of seeing a larger proportion of French data during pretraining.  
+
+<center>
+   <img src='benchmark_eval.png' width='800px'>
+</center>
+
+## Testing the model
+
+### Test with ollama
+Note: for best performance, please use the [GGUF files](https://huggingface.co/OpenLLM-France/Luciole-23B-Instruct-1.1-GGUF) released officially by OpenLLM.
+
+* Download and install [Ollama](https://ollama.com/download)
+* Download the [GGUF model](https://huggingface.co/OpenLLM-France/Luciole-23B-Instruct-1.1-GGUF)
+* Copy the [`Modelfile`](https://huggingface.co/OpenLLM-France/Luciole-23B-Instruct-1.1-GGUF/blob/main/Modelfile), adpating if necessary the path to the GGUF file (line starting with `FROM`).
+* Run in a shell:
+    * `ollama create -f Modelfile Luciole`
+    * `ollama run Luciole`
+* Once ">>>" appears, type your prompt(s) and press Enter.
+* Optionally, restart a conversation by typing "`/clear`"
+* End the session by typing "`/bye`".
+
+Useful for debug:
+* [How to print input requests and output responses in Ollama server?](https://stackoverflow.com/a/78831840)
+* [Documentation on Modelfile](https://github.com/ollama/ollama/blob/main/docs/modelfile.mdx#parameter)
+   * Examples: [Ollama model library](https://github.com/ollama/ollama#model-library)
+      * Llama 3 example: https://ollama.com/library/llama3.1
+* Add GUI : https://docs.openwebui.com/
+
+### Test with vLLM
+
+#### 1. Run vLLM Docker Container
+
+Use the following command to deploy the model,
+replacing `INSERT_YOUR_HF_TOKEN` with your Hugging Face Hub token.
+
+```bash
+docker run --runtime nvidia --gpus=all \
+    --env "HUGGING_FACE_HUB_TOKEN=INSERT_YOUR_HF_TOKEN" \
+    -p 8000:8000 \
+    --ipc=host \
+    vllm/vllm-openai:latest \
+    --model OpenLLM-France/Luciole-23B-Instruct-1.1
+```
+
+#### 2. Test using OpenAI Client in Python
+
+To test the deployed model, use the OpenAI Python client as follows:
+
+```python
+from openai import OpenAI
+
+# Initialize the client
+client = OpenAI(base_url='http://localhost:8000/v1', api_key='empty')
+
+# Define the input content
+content = "Hello Luciole"
+
+# Generate a response
+chat_response = client.chat.completions.create(
+    model="OpenLLM-France/Luciole-23B-Instruct-1.1",
+    messages=[
+        {"role": "user", "content": content}
+    ],
+)
+print(chat_response.choices[0].message.content)
+```
+
+## Citation
+
+✍ Paper coming soon!
+
+
+## Acknowledgements
+
+We gratefully acknowledge BPI France for funding the OpenLLM France project under the [call](https://www.bpifrance.fr/nos-appels-a-projets-concours/appel-a-projets-communs-numeriques-pour-lintelligence-artificielle-generative) "Communs numériques pour l’intelligence artificielle générative" ("Digital commons for generative artificial intelligence") as a part of the [France 2030](https://www.info.gouv.fr/grand-dossier/france-2030) program.
+
+Training of Luciole-23B-Instruct-1.1 was made possible by computing AI and storage resources by [GENCI](https://www.genci.fr/) at [IDRIS](http://www.idris.fr/) thanks to the project A0181016189 on the supercomputer Jean Zay’s H100 partition. We gratefully acknowledge their support. 
+
+
+Luciole-23B-Instruct-1.1 was created by members of [LINAGORA](https://labs.linagora.com/) and the [OpenLLM-France](https://www.openllm-france.fr/) community, including in alphabetical order:
+
+Akshay Chaturvedi (LINAGORA)  
+Liam Duignan (CEA List)  
+Olivier Ferret (CEA List)  
+Olivier Gouvert (LINAGORA)  
+Émile Hazard (Opsci)  
+Julie Hunter (LINAGORA)   
+Jean-Pierre Lorré (LINAGORA)  
+Jérôme Louradour (LINAGORA)   
+Michel-Marie Maudet (LINAGORA)  
+Kate Thompson (LINAGORA)  
+Matteo Van Ypersele de Strihou (LINAGORA) 
+
+Particular thanks to the following OpenLLM France members for their valuable input: Clément Bénesse (Opsci), Christophe Cerisara (LORIA) and Gabriel Lauzzana (LORIA).  
+  
+We thank the support teams from IDRIS and NVIDIA for technical guidance throughout the project.
+
+We would also like to thank members of the [Gaperon](https://huggingface.co/collections/almanach/gaperon) and [Salamandra](https://huggingface.co/collections/BSC-LT/salamandra)  projects for sharing their insights with us. We also acknowledge the numerous open source actors whose resources have guided us throughout the training process, with particular thanks to [Nvidia](https://www.nvidia.com/en-eu/),  [HuggingFace](https://huggingface.co/) and [Allen AI](https://allenai.org/). 
+
+
+Finally, we thank the entire OpenLLM-France community, whose members have helped in diverse ways.
+
+## Contact
+
+contact@openllm-france.fr

chat_template.jinja

@@ -0,0 +1,81 @@
+{%- if messages[0].role == 'system' and messages[0].content.strip() != "" %}
+    {%- set system_message = messages[0].content %}
+{%- else %}
+    {%- set system_message = 'You are a helpful AI assistant named Luciole, trained by LINAGORA and OpenLLM France.' %}
+{%- endif %}
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {{- system_message + '\n\n' }}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {{- '<|im_start|>system\n' + system_message + '<|im_end|>\n' }}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
+        {%- set ns.multi_step_tool = false %}
+        {%- set ns.last_query_index = index %}
+    {%- endif %}
+{%- endfor %}
+{%- for message in messages %}
+    {%- if message.content is string %}
+        {%- set content = message.content %}
+    {%- else %}
+        {%- set content = '' %}
+    {%- endif %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- generation %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+                {{- '<|im_start|>' + message.role + '\n' + content.lstrip('\n') }}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{\"name\": \"' }}
+                {{- tool_call.name }}
+                {{- '\", \"arguments\": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+        {%- endgeneration %}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

config.json

@@ -0,0 +1,28 @@
+{
+  "architectures": [
+    "NemotronForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 2,
+  "dtype": "bfloat16",
+  "eos_token_id": 261,
+  "head_dim": 128,
+  "hidden_act": "relu2",
+  "hidden_size": 6144,
+  "initializer_range": 0.008,
+  "intermediate_size": 36864,
+  "max_position_embeddings": 131072,
+  "mlp_bias": false,
+  "model_type": "nemotron",
+  "norm_eps": 1e-05,
+  "num_attention_heads": 48,
+  "num_hidden_layers": 40,
+  "num_key_value_heads": 8,
+  "partial_rotary_factor": 0.5,
+  "rope_theta": 2000000.0,
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.1",
+  "use_cache": true,
+  "vocab_size": 128000
+}

generation_config.json

@@ -0,0 +1,10 @@
+{
+  "_from_model_config": true,
+  "do_sample": true,
+  "bos_token_id": 0,
+  "eos_token_id": [1, 3, 261],
+  "pad_token_id": 3,
+  "temperature": 0.7,
+  "top_p": 0.9,
+  "transformers_version": "4.57.6"
+}

model-00001-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d55d64a9ad0075837f994d5e6115ca7200baefbab42b93c177f323e90a42c100
+size 4995567560

model-00002-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b751912850f3e67e3f40f0c2cc532dd95f89a4bb188abae1132b81d6d149215
+size 4781707872

model-00003-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8576b8a9195e07fffa260e27be2269ebd88889ffad23392559698ffe9abf7103
+size 4957918744

model-00004-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:86d97526297e1c0e585a5f89f0b0c1fe750c6986373af7dc0a598294692035cf
+size 4781707912

model-00005-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:05ebf9297805f15cceb43cdda50cc420ea97664039544880f346464efbfe7219
+size 4957918768

model-00006-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c6a2a60404d852f8b346515ea6563be7280574bf5beff5990d5476d2fdd004c
+size 4781707912

model-00007-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7fd051347fa12dd92760e9267e7ac3834c9f14ba37f5d176570baa4f76b55878
+size 4957918768

model-00008-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:508dbf36ac35383940fe1785e10ef9aa1e94c06a69b1d2fe61acd34bf796170a
+size 4781707912

model-00009-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:740e767fa23e3ce41d161945e1b00c49e32477a1a75d52646202eed096e7594a
+size 4957918768

model-00010-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:38f1a618887de1cbe8cdc30770c524d48bce6ad9250ec50aedd6e59974872a64
+size 2478908360

model.safetensors.index.json

@@ -0,0 +1,411 @@
+{
+    "metadata": {
+        "total_size": 46432935936
+    },
+    "weight_map": {
+        "model.layers.33.self_attn.k_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.10.input_layernorm.weight": "model-00003-of-00010.safetensors",
+        "model.layers.9.post_attention_layernorm.bias": "model-00003-of-00010.safetensors",
+        "model.layers.9.mlp.up_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.35.input_layernorm.bias": "model-00009-of-00010.safetensors",
+        "model.layers.32.mlp.down_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.38.mlp.up_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.2.input_layernorm.weight": "model-00001-of-00010.safetensors",
+        "model.layers.28.self_attn.q_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.7.self_attn.k_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.6.self_attn.k_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.6.self_attn.v_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.14.self_attn.o_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.20.input_layernorm.weight": "model-00005-of-00010.safetensors",
+        "model.layers.35.post_attention_layernorm.weight": "model-00009-of-00010.safetensors",
+        "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00010.safetensors",
+        "model.layers.21.self_attn.k_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.19.input_layernorm.bias": "model-00005-of-00010.safetensors",
+        "model.layers.2.post_attention_layernorm.bias": "model-00001-of-00010.safetensors",
+        "model.layers.8.post_attention_layernorm.weight": "model-00003-of-00010.safetensors",
+        "model.layers.8.input_layernorm.bias": "model-00003-of-00010.safetensors",
+        "model.layers.4.mlp.up_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00010.safetensors",
+        "model.layers.25.input_layernorm.bias": "model-00007-of-00010.safetensors",
+        "model.layers.0.mlp.down_proj.weight": "model-00001-of-00010.safetensors",
+        "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00010.safetensors",
+        "model.layers.9.self_attn.o_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.31.input_layernorm.bias": "model-00008-of-00010.safetensors",
+        "model.layers.5.post_attention_layernorm.weight": "model-00002-of-00010.safetensors",
+        "model.layers.13.self_attn.k_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.26.input_layernorm.bias": "model-00007-of-00010.safetensors",
+        "model.layers.9.self_attn.k_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.15.input_layernorm.bias": "model-00004-of-00010.safetensors",
+        "model.layers.16.self_attn.q_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.36.input_layernorm.bias": "model-00009-of-00010.safetensors",
+        "model.layers.12.mlp.up_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.2.mlp.down_proj.weight": "model-00001-of-00010.safetensors",
+        "model.layers.17.self_attn.o_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.23.mlp.up_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.39.mlp.up_proj.weight": "model-00010-of-00010.safetensors",
+        "model.layers.37.post_attention_layernorm.bias": "model-00009-of-00010.safetensors",
+        "model.layers.10.mlp.up_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.14.self_attn.q_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.15.mlp.down_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.1.input_layernorm.weight": "model-00001-of-00010.safetensors",
+        "model.layers.38.mlp.down_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.20.post_attention_layernorm.bias": "model-00005-of-00010.safetensors",
+        "model.layers.10.self_attn.q_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.26.mlp.down_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.28.input_layernorm.weight": "model-00007-of-00010.safetensors",
+        "model.layers.9.self_attn.q_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.31.post_attention_layernorm.weight": "model-00008-of-00010.safetensors",
+        "model.layers.12.mlp.down_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.14.post_attention_layernorm.bias": "model-00004-of-00010.safetensors",
+        "model.layers.21.input_layernorm.bias": "model-00006-of-00010.safetensors",
+        "model.layers.32.post_attention_layernorm.weight": "model-00008-of-00010.safetensors",
+        "model.layers.4.self_attn.o_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00010.safetensors",
+        "model.layers.14.input_layernorm.weight": "model-00004-of-00010.safetensors",
+        "model.layers.35.mlp.down_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00010.safetensors",
+        "model.layers.14.input_layernorm.bias": "model-00004-of-00010.safetensors",
+        "model.layers.14.self_attn.k_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.22.self_attn.q_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.15.mlp.up_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.33.input_layernorm.weight": "model-00008-of-00010.safetensors",
+        "model.layers.23.input_layernorm.bias": "model-00006-of-00010.safetensors",
+        "model.layers.31.self_attn.k_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.21.input_layernorm.weight": "model-00006-of-00010.safetensors",
+        "model.layers.10.mlp.down_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00010.safetensors",
+        "model.layers.5.self_attn.v_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.32.self_attn.k_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.8.self_attn.q_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.15.input_layernorm.weight": "model-00004-of-00010.safetensors",
+        "model.layers.16.post_attention_layernorm.weight": "model-00005-of-00010.safetensors",
+        "model.layers.37.post_attention_layernorm.weight": "model-00009-of-00010.safetensors",
+        "model.layers.24.post_attention_layernorm.bias": "model-00006-of-00010.safetensors",
+        "model.layers.36.self_attn.v_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.10.self_attn.o_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.16.post_attention_layernorm.bias": "model-00005-of-00010.safetensors",
+        "model.layers.23.post_attention_layernorm.weight": "model-00006-of-00010.safetensors",
+        "model.layers.39.post_attention_layernorm.bias": "model-00010-of-00010.safetensors",
+        "model.layers.36.mlp.down_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.12.post_attention_layernorm.weight": "model-00004-of-00010.safetensors",
+        "model.layers.22.post_attention_layernorm.weight": "model-00006-of-00010.safetensors",
+        "model.layers.26.post_attention_layernorm.bias": "model-00007-of-00010.safetensors",
+        "model.layers.21.self_attn.o_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.30.input_layernorm.bias": "model-00008-of-00010.safetensors",
+        "model.layers.18.self_attn.q_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.29.mlp.up_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.16.self_attn.k_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.6.mlp.up_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.13.self_attn.q_proj.weight": "model-00004-of-00010.safetensors",
+        "model.norm.weight": "model-00010-of-00010.safetensors",
+        "model.layers.11.input_layernorm.bias": "model-00003-of-00010.safetensors",
+        "model.layers.7.post_attention_layernorm.weight": "model-00003-of-00010.safetensors",
+        "model.layers.21.self_attn.v_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.26.self_attn.v_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.10.self_attn.k_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.12.input_layernorm.bias": "model-00004-of-00010.safetensors",
+        "model.layers.12.self_attn.o_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.13.post_attention_layernorm.bias": "model-00004-of-00010.safetensors",
+        "model.layers.2.mlp.up_proj.weight": "model-00001-of-00010.safetensors",
+        "model.layers.30.self_attn.q_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.15.self_attn.k_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.37.input_layernorm.bias": "model-00009-of-00010.safetensors",
+        "model.layers.35.self_attn.k_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.18.input_layernorm.bias": "model-00005-of-00010.safetensors",
+        "model.layers.36.mlp.up_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.23.self_attn.k_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.24.mlp.down_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.1.mlp.down_proj.weight": "model-00001-of-00010.safetensors",
+        "model.layers.22.mlp.down_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.24.self_attn.o_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.11.self_attn.q_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.17.post_attention_layernorm.weight": "model-00005-of-00010.safetensors",
+        "model.layers.14.self_attn.v_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.8.post_attention_layernorm.bias": "model-00003-of-00010.safetensors",
+        "model.layers.17.mlp.down_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.20.self_attn.v_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.36.post_attention_layernorm.weight": "model-00009-of-00010.safetensors",
+        "model.layers.38.input_layernorm.bias": "model-00009-of-00010.safetensors",
+        "model.layers.35.mlp.up_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.34.self_attn.v_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.30.self_attn.k_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.32.self_attn.o_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.12.self_attn.q_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00010.safetensors",
+        "model.layers.32.self_attn.q_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.24.self_attn.k_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.0.input_layernorm.weight": "model-00001-of-00010.safetensors",
+        "model.layers.32.input_layernorm.bias": "model-00008-of-00010.safetensors",
+        "model.layers.12.post_attention_layernorm.bias": "model-00004-of-00010.safetensors",
+        "model.layers.4.self_attn.k_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.19.input_layernorm.weight": "model-00005-of-00010.safetensors",
+        "model.layers.32.post_attention_layernorm.bias": "model-00008-of-00010.safetensors",
+        "model.layers.17.mlp.up_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.27.mlp.down_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.21.post_attention_layernorm.bias": "model-00006-of-00010.safetensors",
+        "model.layers.27.input_layernorm.bias": "model-00007-of-00010.safetensors",
+        "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00010.safetensors",
+        "model.layers.27.self_attn.q_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.7.self_attn.o_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.38.self_attn.o_proj.weight": "model-00009-of-00010.safetensors",
+        "model.embed_tokens.weight": "model-00001-of-00010.safetensors",
+        "model.layers.9.post_attention_layernorm.weight": "model-00003-of-00010.safetensors",
+        "model.layers.20.self_attn.q_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.26.input_layernorm.weight": "model-00007-of-00010.safetensors",
+        "model.layers.22.input_layernorm.bias": "model-00006-of-00010.safetensors",
+        "lm_head.weight": "model-00010-of-00010.safetensors",
+        "model.layers.24.input_layernorm.bias": "model-00006-of-00010.safetensors",
+        "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00010.safetensors",
+        "model.layers.4.post_attention_layernorm.bias": "model-00002-of-00010.safetensors",
+        "model.layers.10.self_attn.v_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00010.safetensors",
+        "model.layers.13.input_layernorm.weight": "model-00004-of-00010.safetensors",
+        "model.layers.25.self_attn.v_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.5.post_attention_layernorm.bias": "model-00002-of-00010.safetensors",
+        "model.layers.5.mlp.down_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.16.input_layernorm.bias": "model-00005-of-00010.safetensors",
+        "model.layers.32.input_layernorm.weight": "model-00008-of-00010.safetensors",
+        "model.layers.39.input_layernorm.weight": "model-00010-of-00010.safetensors",
+        "model.layers.25.self_attn.k_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.8.mlp.down_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.20.input_layernorm.bias": "model-00005-of-00010.safetensors",
+        "model.layers.36.input_layernorm.weight": "model-00009-of-00010.safetensors",
+        "model.layers.17.post_attention_layernorm.bias": "model-00005-of-00010.safetensors",
+        "model.layers.29.mlp.down_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.11.post_attention_layernorm.bias": "model-00003-of-00010.safetensors",
+        "model.layers.29.post_attention_layernorm.weight": "model-00007-of-00010.safetensors",
+        "model.layers.39.mlp.down_proj.weight": "model-00010-of-00010.safetensors",
+        "model.layers.23.self_attn.o_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.18.self_attn.o_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.14.post_attention_layernorm.weight": "model-00004-of-00010.safetensors",
+        "model.layers.8.self_attn.k_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.26.post_attention_layernorm.weight": "model-00007-of-00010.safetensors",
+        "model.layers.25.mlp.down_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.9.self_attn.v_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.38.self_attn.v_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.39.input_layernorm.bias": "model-00010-of-00010.safetensors",
+        "model.layers.36.post_attention_layernorm.bias": "model-00009-of-00010.safetensors",
+        "model.layers.28.self_attn.v_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.6.post_attention_layernorm.weight": "model-00002-of-00010.safetensors",
+        "model.layers.17.self_attn.v_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.19.post_attention_layernorm.weight": "model-00005-of-00010.safetensors",
+        "model.layers.21.post_attention_layernorm.weight": "model-00006-of-00010.safetensors",
+        "model.layers.12.input_layernorm.weight": "model-00004-of-00010.safetensors",
+        "model.layers.9.input_layernorm.weight": "model-00003-of-00010.safetensors",
+        "model.layers.15.post_attention_layernorm.bias": "model-00004-of-00010.safetensors",
+        "model.layers.18.post_attention_layernorm.weight": "model-00005-of-00010.safetensors",
+        "model.layers.33.post_attention_layernorm.weight": "model-00008-of-00010.safetensors",
+        "model.layers.27.self_attn.o_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.6.post_attention_layernorm.bias": "model-00002-of-00010.safetensors",
+        "model.layers.36.self_attn.o_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.18.post_attention_layernorm.bias": "model-00005-of-00010.safetensors",
+        "model.layers.21.self_attn.q_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.3.input_layernorm.weight": "model-00002-of-00010.safetensors",
+        "model.layers.25.post_attention_layernorm.bias": "model-00007-of-00010.safetensors",
+        "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00010.safetensors",
+        "model.layers.5.self_attn.q_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.11.self_attn.k_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.31.post_attention_layernorm.bias": "model-00008-of-00010.safetensors",
+        "model.layers.27.input_layernorm.weight": "model-00007-of-00010.safetensors",
+        "model.layers.25.self_attn.o_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.28.self_attn.o_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.15.self_attn.q_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.1.post_attention_layernorm.bias": "model-00001-of-00010.safetensors",
+        "model.norm.bias": "model-00010-of-00010.safetensors",
+        "model.layers.20.mlp.down_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.4.mlp.down_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.19.self_attn.q_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.18.input_layernorm.weight": "model-00005-of-00010.safetensors",
+        "model.layers.3.post_attention_layernorm.weight": "model-00002-of-00010.safetensors",
+        "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00010.safetensors",
+        "model.layers.35.self_attn.q_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.25.post_attention_layernorm.weight": "model-00007-of-00010.safetensors",
+        "model.layers.10.input_layernorm.bias": "model-00003-of-00010.safetensors",
+        "model.layers.26.self_attn.k_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.24.self_attn.v_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.36.self_attn.q_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00010.safetensors",
+        "model.layers.20.self_attn.k_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.6.input_layernorm.bias": "model-00002-of-00010.safetensors",
+        "model.layers.25.mlp.up_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.33.self_attn.v_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00010.safetensors",
+        "model.layers.13.input_layernorm.bias": "model-00004-of-00010.safetensors",
+        "model.layers.19.mlp.down_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.39.self_attn.k_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.19.self_attn.v_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.21.mlp.down_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.23.post_attention_layernorm.bias": "model-00006-of-00010.safetensors",
+        "model.layers.8.self_attn.o_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.28.mlp.up_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00010.safetensors",
+        "model.layers.22.self_attn.o_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.24.self_attn.q_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.28.mlp.down_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.19.self_attn.o_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.31.self_attn.o_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.4.input_layernorm.weight": "model-00002-of-00010.safetensors",
+        "model.layers.7.input_layernorm.weight": "model-00003-of-00010.safetensors",
+        "model.layers.28.post_attention_layernorm.weight": "model-00007-of-00010.safetensors",
+        "model.layers.13.mlp.up_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.28.input_layernorm.bias": "model-00007-of-00010.safetensors",
+        "model.layers.31.mlp.down_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.38.post_attention_layernorm.bias": "model-00009-of-00010.safetensors",
+        "model.layers.11.mlp.up_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.39.self_attn.q_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.7.self_attn.q_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.29.post_attention_layernorm.bias": "model-00007-of-00010.safetensors",
+        "model.layers.22.input_layernorm.weight": "model-00006-of-00010.safetensors",
+        "model.layers.5.input_layernorm.weight": "model-00002-of-00010.safetensors",
+        "model.layers.2.input_layernorm.bias": "model-00001-of-00010.safetensors",
+        "model.layers.13.self_attn.o_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.34.post_attention_layernorm.weight": "model-00009-of-00010.safetensors",
+        "model.layers.6.mlp.down_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.18.mlp.up_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.29.self_attn.k_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.29.self_attn.o_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.4.self_attn.v_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.6.input_layernorm.weight": "model-00002-of-00010.safetensors",
+        "model.layers.15.self_attn.v_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.14.mlp.down_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.19.post_attention_layernorm.bias": "model-00005-of-00010.safetensors",
+        "model.layers.20.self_attn.o_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.33.mlp.up_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.24.input_layernorm.weight": "model-00006-of-00010.safetensors",
+        "model.layers.23.input_layernorm.weight": "model-00006-of-00010.safetensors",
+        "model.layers.6.self_attn.o_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.33.self_attn.o_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.30.mlp.down_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.7.self_attn.v_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.3.input_layernorm.bias": "model-00002-of-00010.safetensors",
+        "model.layers.27.self_attn.v_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.10.post_attention_layernorm.weight": "model-00003-of-00010.safetensors",
+        "model.layers.27.post_attention_layernorm.weight": "model-00007-of-00010.safetensors",
+        "model.layers.12.self_attn.v_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.15.post_attention_layernorm.weight": "model-00004-of-00010.safetensors",
+        "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00010.safetensors",
+        "model.layers.38.post_attention_layernorm.weight": "model-00009-of-00010.safetensors",
+        "model.layers.35.input_layernorm.weight": "model-00009-of-00010.safetensors",
+        "model.layers.34.mlp.up_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.35.post_attention_layernorm.bias": "model-00009-of-00010.safetensors",
+        "model.layers.0.input_layernorm.bias": "model-00001-of-00010.safetensors",
+        "model.layers.12.self_attn.k_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.3.post_attention_layernorm.bias": "model-00002-of-00010.safetensors",
+        "model.layers.31.self_attn.v_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.27.post_attention_layernorm.bias": "model-00007-of-00010.safetensors",
+        "model.layers.34.mlp.down_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.14.mlp.up_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.23.self_attn.v_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.24.post_attention_layernorm.weight": "model-00006-of-00010.safetensors",
+        "model.layers.5.self_attn.k_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.1.input_layernorm.bias": "model-00001-of-00010.safetensors",
+        "model.layers.38.input_layernorm.weight": "model-00009-of-00010.safetensors",
+        "model.layers.5.input_layernorm.bias": "model-00002-of-00010.safetensors",
+        "model.layers.17.input_layernorm.weight": "model-00005-of-00010.safetensors",
+        "model.layers.32.self_attn.v_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.7.mlp.down_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.27.self_attn.k_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.17.self_attn.k_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.37.mlp.down_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.30.post_attention_layernorm.weight": "model-00008-of-00010.safetensors",
+        "model.layers.28.self_attn.k_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.37.input_layernorm.weight": "model-00009-of-00010.safetensors",
+        "model.layers.16.input_layernorm.weight": "model-00005-of-00010.safetensors",
+        "model.layers.27.mlp.up_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.31.input_layernorm.weight": "model-00008-of-00010.safetensors",
+        "model.layers.34.input_layernorm.weight": "model-00009-of-00010.safetensors",
+        "model.layers.1.mlp.up_proj.weight": "model-00001-of-00010.safetensors",
+        "model.layers.18.self_attn.v_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.28.post_attention_layernorm.bias": "model-00007-of-00010.safetensors",
+        "model.layers.25.input_layernorm.weight": "model-00007-of-00010.safetensors",
+        "model.layers.30.input_layernorm.weight": "model-00008-of-00010.safetensors",
+        "model.layers.35.self_attn.o_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.11.self_attn.v_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.32.mlp.up_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.4.input_layernorm.bias": "model-00002-of-00010.safetensors",
+        "model.layers.4.post_attention_layernorm.weight": "model-00002-of-00010.safetensors",
+        "model.layers.29.self_attn.q_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.33.self_attn.q_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.36.self_attn.k_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.37.self_attn.q_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.34.post_attention_layernorm.bias": "model-00009-of-00010.safetensors",
+        "model.layers.37.mlp.up_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.3.mlp.down_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.7.mlp.up_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.39.self_attn.o_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.8.self_attn.v_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.37.self_attn.o_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.21.mlp.up_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.31.self_attn.q_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.16.mlp.up_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.35.self_attn.v_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.22.self_attn.v_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.5.mlp.up_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.26.mlp.up_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.11.mlp.down_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.37.self_attn.v_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.11.input_layernorm.weight": "model-00003-of-00010.safetensors",
+        "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00010.safetensors",
+        "model.layers.39.post_attention_layernorm.weight": "model-00010-of-00010.safetensors",
+        "model.layers.24.mlp.up_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.23.mlp.down_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.34.self_attn.k_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.13.mlp.down_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.13.self_attn.v_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.8.mlp.up_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.9.input_layernorm.bias": "model-00003-of-00010.safetensors",
+        "model.layers.39.self_attn.v_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.13.post_attention_layernorm.weight": "model-00004-of-00010.safetensors",
+        "model.layers.34.self_attn.q_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.23.self_attn.q_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.22.post_attention_layernorm.bias": "model-00006-of-00010.safetensors",
+        "model.layers.20.mlp.up_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.22.self_attn.k_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.37.self_attn.k_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.20.post_attention_layernorm.weight": "model-00005-of-00010.safetensors",
+        "model.layers.19.self_attn.k_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.16.self_attn.v_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.29.input_layernorm.weight": "model-00007-of-00010.safetensors",
+        "model.layers.7.post_attention_layernorm.bias": "model-00003-of-00010.safetensors",
+        "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00010.safetensors",
+        "model.layers.7.input_layernorm.bias": "model-00003-of-00010.safetensors",
+        "model.layers.11.self_attn.o_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.15.self_attn.o_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.9.mlp.down_proj.weight": "model-00003-of-00010.safetensors",
+        "model.layers.22.mlp.up_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.30.self_attn.o_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.30.mlp.up_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.8.input_layernorm.weight": "model-00003-of-00010.safetensors",
+        "model.layers.6.self_attn.q_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.30.post_attention_layernorm.bias": "model-00008-of-00010.safetensors",
+        "model.layers.3.mlp.up_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.34.self_attn.o_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.11.post_attention_layernorm.weight": "model-00003-of-00010.safetensors",
+        "model.layers.29.input_layernorm.bias": "model-00007-of-00010.safetensors",
+        "model.layers.38.self_attn.k_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.31.mlp.up_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.17.self_attn.q_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.26.self_attn.q_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.38.self_attn.q_proj.weight": "model-00009-of-00010.safetensors",
+        "model.layers.10.post_attention_layernorm.bias": "model-00003-of-00010.safetensors",
+        "model.layers.18.self_attn.k_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.29.self_attn.v_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.16.mlp.down_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.0.post_attention_layernorm.bias": "model-00001-of-00010.safetensors",
+        "model.layers.19.mlp.up_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00010.safetensors",
+        "model.layers.26.self_attn.o_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.5.self_attn.o_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.17.input_layernorm.bias": "model-00005-of-00010.safetensors",
+        "model.layers.18.mlp.down_proj.weight": "model-00005-of-00010.safetensors",
+        "model.layers.0.mlp.up_proj.weight": "model-00001-of-00010.safetensors",
+        "model.layers.16.self_attn.o_proj.weight": "model-00004-of-00010.safetensors",
+        "model.layers.33.mlp.down_proj.weight": "model-00008-of-00010.safetensors",
+        "model.layers.33.post_attention_layernorm.bias": "model-00008-of-00010.safetensors",
+        "model.layers.33.input_layernorm.bias": "model-00008-of-00010.safetensors",
+        "model.layers.25.self_attn.q_proj.weight": "model-00006-of-00010.safetensors",
+        "model.layers.4.self_attn.q_proj.weight": "model-00002-of-00010.safetensors",
+        "model.layers.30.self_attn.v_proj.weight": "model-00007-of-00010.safetensors",
+        "model.layers.34.input_layernorm.bias": "model-00009-of-00010.safetensors"
+    }
+}

special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,113 @@
+{
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "add_prefix_space": null,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "260": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "262": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "263": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "264": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "265": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "266": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "267": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "extra_special_tokens": {},
+  "legacy": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "tokenizer_class": "LlamaTokenizerFast",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}