Add files using upload-large-folder tool

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,119 @@
+---
+license: apache-2.0
+base_model: Kwaipilot/KAT-Coder-V2.5-Dev
+tags:
+  - colibri
+  - kat-coder
+  - qwen3.6-35b-a3b
+  - moe
+  - int4
+  - gs64
+  - code
+library_name: colibri
+---
+
+# KAT-Coder-V2.5-Dev — colibri int4 **gs64** container (group-scaled)
+
+[colibri](https://github.com/JustVugg/colibri) container for
+[**Kwaipilot/KAT-Coder-V2.5-Dev**](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev),
+a coding model that shares the Qwen3.6-35B-A3B architecture (`qwen3_5_moe`,
+35B total / 3B active, 40 layers, 256 experts top-8, hybrid Gated Attention +
+Gated DeltaNet). Experts are quantized to **group-scaled int4** (`expert_gs=64`,
+one f32 scale per 64 input elements per row, ~22 GB), the higher-accuracy
+variant of colibri's int4 format: on the sibling base model the same pipeline
+cut first-token logit error by **~44 %** vs per-row scaling (see
+[`qwen36-35b-a3b-colibri-i4-gs64`](https://huggingface.co/Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64)).
+Self-contained: bundled `tokenizer.json`, flat `config.json`.
+
+This runs a 35B-class coding MoE interactively on one or two 8 GB GPUs plus
+~30–40 GB RAM (see numbers below).
+
+## ⚠️ Two KAT-Coder-V2.5 versions exist — don't mix up the scores
+
+Kwaipilot's paper evaluates the **commercial flagship** KAT-Coder-V2.5; the
+open weights (and therefore this container) are **KAT-Coder-V2.5-Dev**, a
+separate open release. Reported numbers differ substantially:
+
+| | KAT-Coder-V2.5 (commercial flagship) | KAT-Coder-V2.5-Dev (open weights, this container) |
+|---|---|---|
+| Source | paper evaluation | HuggingFace open weights |
+| Architecture | MoE (undisclosed) | MoE (`qwen3_5_moe`) |
+| Parameters | undisclosed | 35B total / 3B active |
+| Base model | undisclosed | Qwen3.6-35B-A3B |
+| Modality | includes vision/multimodal | text-only |
+| Context | unspecified | 262,144 tokens (YaRN up to ~1M) |
+| License | commercial, closed | Apache 2.0 |
+| SWE-Bench Pro | 65.2 | 45.96 |
+| KAT Code Bench | 53.1 | 46.21 |
+| PinchBench | 94.9 | 93.43 |
+| Terminal-Bench 2.1 | 60.7 | 41.02 |
+| SciCode | 50.3 | 44.20 |
+| SWE-Bench Verified | not reported | 69.40 |
+| SWE-Bench Multilingual | not reported | 63.00 |
+
+(Scores as reported by Kwaipilot for the respective version.)
+
+## Which commits you need
+
+The qwen36 engine is not merged upstream yet; pick the branch matching your
+setup (all on [github.com/kreuzzelg/colibri](https://github.com/kreuzzelg/colibri)):
+
+| Setup | Branch / commit | Upstream status |
+|---|---|---|
+| **CPU, gs64 (this container)** | [`gs64-ab`](https://github.com/kreuzzelg/colibri/tree/gs64-ab) (`2f9aab2`) — engine + group-scaled GEMV `matmul_q_gs` | engine submitted as [#712](https://github.com/JustVugg/colibri/pull/712) |
+| **CUDA VRAM expert tier + gs64 (recommended, 1–2 GPUs)** | [`gs64-gpu`](https://github.com/kreuzzelg/colibri/tree/gs64-gpu) (`de5dde7`) = gs64 tier upload (`90450ec`) + upstream-dev backend incl. the async grouped-int4 fix | tier submitted as [#713](https://github.com/JustVugg/colibri/pull/713); backend fix as [#762](https://github.com/JustVugg/colibri/pull/762) |
+| Per-row containers on the plain engine | [`upstream-pr1-qwen36-engine`](https://github.com/kreuzzelg/colibri/tree/upstream-pr1-qwen36-engine) (`16b945a`) | = #712 |
+
+The backend fix **#762 is required for correct gs64 output on the GPU tier** —
+without it the async decode path silently applies per-row scales to grouped
+containers. Validated on this exact combination: CPU-vs-GPU-tier logits cosine
+1.0 over 24 greedy tokens, token-identical output.
+
+## Run it
+
+```bash
+git clone -b gs64-gpu https://github.com/kreuzzelg/colibri && cd colibri/c
+# GPU tier (CUDA):
+nvcc -O3 -std=c++17 -arch=native -c backend_cuda.cu -o backend_cuda.o
+gcc -O3 -march=x86-64-v3 -fopenmp -pthread qwen36.c qwen36_tier.c vulkan_gemv.c \
+    backend_cuda.o -o qwen36 -lm -lcudart -lstdc++
+
+SNAP=$(python -c "from huggingface_hub import snapshot_download; \
+print(snapshot_download('Kreuzzelg/kat-coder-v2.5-dev-colibri-i4-gs64'))")
+printf 'Write a Python function that returns the n-th Fibonacci number using memoization.\n' > prompt.txt
+SNAP=$SNAP N_NEW=300 COLI_CUDA=1 HEAT_FILE=heat.bin \
+  OMP_NUM_THREADS=<physical cores> OMP_WAIT_POLICY=ACTIVE OMP_PROC_BIND=close \
+  ./qwen36 256 4 prompt.txt
+# CPU only: build from the gs64-ab branch with `make -C c qwen36` and drop COLI_CUDA.
+```
+
+The engine reads `expert_gs` from `qwen36_meta.json` and picks the group-scaled
+GEMV automatically. `HEAT_FILE` persists expert-routing heat: the second run
+starts with the right experts in VRAM.
+
+## Measured (Threadripper PRO 3945WX 12C, RTX 3070 8 GB + Quadro RTX 4000 8 GB, 200-token decode)
+
+| | 2×8 GB GPUs | 1×8 GB GPU (Quadro) |
+|---|---|---|
+| tok/s cold / warm | 10.4 / **11.3** | 9.3 / **10.4** |
+| VRAM hit rate cold / warm | 75 % / 100 % | 39 % / 97 % |
+| VRAM used | 13.2 GiB total | 6.6 GiB |
+| peak RSS | 30 GB | 40 GB |
+| TTFT (49-token prompt) | 1.9 s | 2.1 s |
+
+Verified qualitatively: correct, clean Python (memoization, docstring, type
+hints, edge cases) on the Fibonacci prompt; coherent 200-token continuations
+under the benchmark prompt.
+
+## Caveats
+
+- **Dev checkpoint:** for serious use, handle the chat template and EOS
+  yourself — in raw-completion mode the model keeps writing after the code.
+- The 44 % accuracy A/B was measured on the sibling Qwen3.6-35B-A3B base with
+  the same converter/engine; no separate A/B was run for the KAT finetune.
+
+## Credits
+
+- Model: [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) (Apache-2.0), a finetune of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)
+- colibri engine & concept: [JustVugg/colibri](https://github.com/JustVugg/colibri)
+- Original qwen36 engine + converter: [@minne100](https://huggingface.co/minne100) ([PR #602](https://github.com/JustVugg/colibri/pull/602)); group-scaled quantization, CUDA VRAM tier and the grouped-int4 backend fix added on top.

config.hf.json

@@ -0,0 +1,119 @@
+{
+    "architectures": [
+        "Qwen3_5MoeForConditionalGeneration"
+    ],
+    "image_token_id": 248056,
+    "model_type": "qwen3_5_moe",
+    "text_config": {
+        "attention_bias": false,
+        "attention_dropout": 0.0,
+        "attn_output_gate": true,
+        "bos_token_id": 248044,
+        "dtype": "bfloat16",
+        "eos_token_id": 248044,
+        "full_attention_interval": 4,
+        "head_dim": 256,
+        "hidden_act": "silu",
+        "hidden_size": 2048,
+        "initializer_range": 0.02,
+        "layer_types": [
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention",
+            "linear_attention",
+            "linear_attention",
+            "linear_attention",
+            "full_attention"
+        ],
+        "linear_conv_kernel_dim": 4,
+        "linear_key_head_dim": 128,
+        "linear_num_key_heads": 16,
+        "linear_num_value_heads": 32,
+        "linear_value_head_dim": 128,
+        "mamba_ssm_dtype": "float32",
+        "max_position_embeddings": 262144,
+        "model_type": "qwen3_5_moe_text",
+        "moe_intermediate_size": 512,
+        "mtp_num_hidden_layers": 0,
+        "mtp_use_dedicated_embeddings": false,
+        "num_attention_heads": 16,
+        "num_experts": 256,
+        "num_experts_per_tok": 8,
+        "num_hidden_layers": 40,
+        "num_key_value_heads": 2,
+        "output_router_logits": false,
+        "pad_token_id": null,
+        "partial_rotary_factor": 0.25,
+        "rms_norm_eps": 1e-06,
+        "rope_parameters": {
+            "mrope_interleaved": true,
+            "mrope_section": [
+                11,
+                11,
+                10
+            ],
+            "partial_rotary_factor": 0.25,
+            "rope_theta": 10000000,
+            "rope_type": "default"
+        },
+        "router_aux_loss_coef": 0.001,
+        "shared_expert_intermediate_size": 512,
+        "tie_word_embeddings": false,
+        "use_cache": true,
+        "vocab_size": 248320
+    },
+    "tie_word_embeddings": false,
+    "transformers_version": "4.57.1",
+    "video_token_id": 248057,
+    "vision_config": {
+        "deepstack_visual_indexes": [],
+        "depth": 27,
+        "hidden_act": "gelu_pytorch_tanh",
+        "hidden_size": 1152,
+        "in_channels": 3,
+        "initializer_range": 0.02,
+        "intermediate_size": 4304,
+        "model_type": "qwen3_5_moe",
+        "num_heads": 16,
+        "num_position_embeddings": 2304,
+        "out_hidden_size": 2048,
+        "patch_size": 16,
+        "spatial_merge_size": 2,
+        "temporal_patch_size": 2
+    },
+    "vision_end_token_id": 248054,
+    "vision_start_token_id": 248053
+}

config.json

@@ -0,0 +1,91 @@
+{
+ "attention_bias": false,
+ "attention_dropout": 0.0,
+ "attn_output_gate": true,
+ "bos_token_id": 248044,
+ "dtype": "bfloat16",
+ "eos_token_id": 248044,
+ "full_attention_interval": 4,
+ "head_dim": 256,
+ "hidden_act": "silu",
+ "hidden_size": 2048,
+ "initializer_range": 0.02,
+ "layer_types": [
+  "linear_attention",
+  "linear_attention",
+  "linear_attention",
+  "full_attention",
+  "linear_attention",
+  "linear_attention",
+  "linear_attention",
+  "full_attention",
+  "linear_attention",
+  "linear_attention",
+  "linear_attention",
+  "full_attention",
+  "linear_attention",
+  "linear_attention",
+  "linear_attention",
+  "full_attention",
+  "linear_attention",
+  "linear_attention",
+  "linear_attention",
+  "full_attention",
+  "linear_attention",
+  "linear_attention",
+  "linear_attention",
+  "full_attention",
+  "linear_attention",
+  "linear_attention",
+  "linear_attention",
+  "full_attention",
+  "linear_attention",
+  "linear_attention",
+  "linear_attention",
+  "full_attention",
+  "linear_attention",
+  "linear_attention",
+  "linear_attention",
+  "full_attention",
+  "linear_attention",
+  "linear_attention",
+  "linear_attention",
+  "full_attention"
+ ],
+ "linear_conv_kernel_dim": 4,
+ "linear_key_head_dim": 128,
+ "linear_num_key_heads": 16,
+ "linear_num_value_heads": 32,
+ "linear_value_head_dim": 128,
+ "mamba_ssm_dtype": "float32",
+ "max_position_embeddings": 262144,
+ "model_type": "qwen3_5_moe_text",
+ "moe_intermediate_size": 512,
+ "mtp_num_hidden_layers": 0,
+ "mtp_use_dedicated_embeddings": false,
+ "num_attention_heads": 16,
+ "num_experts": 256,
+ "num_experts_per_tok": 8,
+ "num_hidden_layers": 40,
+ "num_key_value_heads": 2,
+ "output_router_logits": false,
+ "pad_token_id": null,
+ "partial_rotary_factor": 0.25,
+ "rms_norm_eps": 1e-06,
+ "rope_parameters": {
+  "mrope_interleaved": true,
+  "mrope_section": [
+   11,
+   11,
+   10
+  ],
+  "partial_rotary_factor": 0.25,
+  "rope_theta": 10000000,
+  "rope_type": "default"
+ },
+ "router_aux_loss_coef": 0.001,
+ "shared_expert_intermediate_size": 512,
+ "tie_word_embeddings": false,
+ "use_cache": true,
+ "vocab_size": 248320
+}

model-00000.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8d07f15637a9efcd78be0c310df53fd07e1f71b6e4004096bb7c0286f7b3b0b9
+size 527831896

model-00001.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a2dccde923e63ad00ed4b2b95afa462dab0daea6bceb755119a32590cea1ee59
+size 527831896

model-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b02dc5c8d787225bb1437ffebd2033a6f89739d4cc99a9d2253bdc81993da0d
+size 527831896

model-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e314ce0090ae636817e98e25c54096285fecc4174949dceb64961fe39fcb1d05
+size 514921584

model-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ef270056016186336c1ea1dcaf07903dc10a2aab9ffc0aef151390f599592539
+size 527831896

model-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e8e7797ae3a8803e4e1c1326f6f3b02f376dad69494fc52518720ddad118bf83
+size 527831896

model-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:61b44f61faca5abdede78098463bd62301054499225a8f205142537dddcc8eb8
+size 527831896

model-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3457957a2797d85717138f8b336301f5f64e2609d86e15ac6f7d0126d2fdbdce
+size 514921584

model-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:45aa6626f41f42b342ea2f069d8a4208bb57a7622992959687d49701cddf25cb
+size 527831896

model-00009.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:15b29be78ca2b7a0412d7416d91b49a30a06a5a1b0a47bc4e65781797b85d2b0
+size 527831896

model-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7d540236b1a72249696ccd0831f060b6c539c21ba80fa17caed5fcd12dfa7e69
+size 527832424

model-00011.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:471c3caee7080c717ea6191db2f297eb533cb3c71f951cbee22cb0156083492a
+size 514922112

model-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50b4d81e7a6e4697840e44875e362d14b50b2a23e7a2afd16d4848ba1bf48e80
+size 527832424

model-00013.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4b311c0e910158a89a4d837b0e8fc1acdf34ddf7bbe0827b42efe679ca33d3c1
+size 527832424

model-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:87ae031ac9d08261f1279bf4989c22f48e6029df79ce5228bbb10ed92f72fe21
+size 527832424

model-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23a7e44bf3f4968feb99122c9bb8b988e1017ca9dc90c968250225a89086987b
+size 514922112

model-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:44f8c62ddb255b3edd75e97978ca1bc905fbd2fc8ccdd4e72ec83d41f19d18a9
+size 527832424

model-00017.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8a7403f4fe20c880360bbfef72873ed5094d0870312c1efbcafc5db3ecc43708
+size 527832424

model-00018.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b473fe9d19447c40bbf2d30df9ec9855d3aa8faa1d3165f90a5d1bb00be3a625
+size 527832424

model-00019.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b931b6ea90517130f57a893d893b97d59655918fe33453b2e1b542a02600ddaf
+size 514922112

model-00020.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0db48fd3902850ee723928122057582125bf6362fcd85c8f107d579f9681fc12
+size 527832424

model-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b787b919b17e4ff827e49b5ffc77b639d99b2fe18d1ea555df1636738a07c75
+size 527832424

model-00022.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:95162a7fb80ab2efffdaed06dae2b974dcb713c8a0ad1e9043145e56b34562a7
+size 527832424

model-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:46df91b0421b214d7710f8a171c1194d976da731b887b1a01811fe023de7cf1c
+size 514922112

model-00024.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f30691a1703665074ec096db56eb85a03a25a03f745e3684b4055b4da67364cd
+size 527832424

model-00025.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:960eff38e1448dd43e1c19fec2c110a78b3916acee0a617e1759d3f3e30f7be2
+size 527832424

model-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0674a4c897e78176c9cc5acb21b1e920d6414f213e693d9313be6c9aabf3937b
+size 527832424

model-00027.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b97c96416db7b8de1b47cb8712b9c552ef9cc94e7f6c0587855f7bfe6d1f80ec
+size 514922112

model-00028.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2b8d7d953102f2d122f67b31bbbff91659373ffcb49aa63b4dbccfb5bb18bb15
+size 527832424

model-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a5965d5b79ab2ff13a2ecff677c789709f42d45261c774295627c4ea93d38d6f
+size 527832424

model-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:281198cb453370c15c3beda8bc8dbf2a672e2e668f70924050a99b2c01beda1d
+size 527832424

model-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2ec09280f7051647ddcfe029ac4cbcf419310ddfce8b53907b3d66971623ca91
+size 514922112

model-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:efa09875392b994cc0f4d343e5f03d1d392fe9cc1b4f5ccf68bbddad4249cde1
+size 527832424

model-00033.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:27cde38fff534aef07710b7754096fb053e12286c79939d5fc4e25bad26bd0cc
+size 527832424

model-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:713732a554d1e7dda7c4614f5ecd678f1d0766c083f1e5dabf71e134c7572671
+size 527832424

model-00035.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:44da6d9ce691b7e6893c946d77cf875bca84d5d6f0da4fb7a497cdbf22c0d4a1
+size 514922112

model-00036.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3cc9e5dc92893b454142c3728ba3cbc7cdf79fb4f0aca9828128b565eca0a913
+size 527832424

model-00037.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7a1c858e882bb09d6ea1294ad1a1767ec3b95096b530ba3d4c2425f6e935ddd7
+size 527832424

model-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e70a8bad77ef93b1676868a61876e1eb7161163628e053c709a01d5929c6b254
+size 527832424

model-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c33f647a3511c676210d7cc5fa08e2e80402bcbe824dda7ce92705c8dd8e42af
+size 514922112

model-globals.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:78d4d8b57ffb78be6a890469f4238d3a6a8a3b9b6c1aae9582a391ff23818c22
+size 2034241832

qwen36_meta.json

@@ -0,0 +1,82 @@
+{
+  "model_type": "qwen3_5_moe",
+  "hidden": 2048,
+  "n_layers": 40,
+  "n_active": 40,
+  "layer_types": [
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention"
+  ],
+  "num_experts": 256,
+  "topk": 8,
+  "moe_inter": 512,
+  "shared_inter": 512,
+  "rms_eps": 1e-06,
+  "ebits": 4,
+  "scoring_func": "softmax",
+  "n_group": 1,
+  "topk_group": 1,
+  "norm_topk_prob": false,
+  "attn_output_gate": true,
+  "rope_theta": 10000000.0,
+  "mrope_section": [
+    11,
+    11,
+    10
+  ],
+  "partial_rotary_factor": 0.25,
+  "q_heads": 16,
+  "kv_heads": 2,
+  "q_head_dim": 512,
+  "k_head_dim": 256,
+  "v_head_dim": 256,
+  "o_in": 4096,
+  "qk_rope_head_dim": 256,
+  "expert_gs": 64,
+  "head_dim": 256,
+  "rope_dim": 256,
+  "dn_vheads": 32,
+  "dn_kheads": 16,
+  "dn_kdim": 128,
+  "dn_vdim": 128,
+  "dn_convk": 4,
+  "dn_conv_dim": 8192
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42
+size 12807982