README.md

---
license: apache-2.0
base_model: Kwaipilot/KAT-Coder-V2.5-Dev
tags:
  - colibri
  - kat-coder
  - qwen3.6-35b-a3b
  - moe
  - int4
  - gs64
  - code
library_name: colibri
---

# KAT-Coder-V2.5-Dev — colibri int4 **gs64** container (group-scaled)

[colibri](https://github.com/JustVugg/colibri) container for
[**Kwaipilot/KAT-Coder-V2.5-Dev**](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev),
a coding model that shares the Qwen3.6-35B-A3B architecture (`qwen3_5_moe`,
35B total / 3B active, 40 layers, 256 experts top-8, hybrid Gated Attention +
Gated DeltaNet). Experts are quantized to **group-scaled int4** (`expert_gs=64`,
one f32 scale per 64 input elements per row, about 22 GB), the higher-accuracy
variant of colibri's int4 format: on the sibling base model the same pipeline
cut first-token logit error by **~44 %** vs per-row scaling (see
[`qwen36-35b-a3b-colibri-i4-gs64`](https://huggingface.co/Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64)).
Self-contained: bundled `tokenizer.json`, flat `config.json`.

This runs a 35B-class coding MoE interactively on one or two 8 GB GPUs plus
~30–40 GB RAM (see numbers below).

## ⚠️ Two KAT-Coder-V2.5 versions exist — don't mix up the scores

Kwaipilot's paper evaluates the **commercial flagship** KAT-Coder-V2.5; the
open weights (and therefore this container) are **KAT-Coder-V2.5-Dev**, a
separate open release. Reported numbers differ substantially:

| | KAT-Coder-V2.5 (commercial flagship) | KAT-Coder-V2.5-Dev (open weights, this container) |
|---|---|---|
| Source | paper evaluation | HuggingFace open weights |
| Architecture | MoE (undisclosed) | MoE (`qwen3_5_moe`) |
| Parameters | undisclosed | 35B total / 3B active |
| Base model | undisclosed | Qwen3.6-35B-A3B |
| Modality | includes vision/multimodal | text-only |
| Context | unspecified | 262,144 tokens (YaRN up to ~1M) |
| License | commercial, closed | Apache 2.0 |
| SWE-Bench Pro | 65.2 | 45.96 |
| KAT Code Bench | 53.1 | 46.21 |
| PinchBench | 94.9 | 93.43 |
| Terminal-Bench 2.1 | 60.7 | 41.02 |
| SciCode | 50.3 | 44.20 |
| SWE-Bench Verified | not reported | 69.40 |
| SWE-Bench Multilingual | not reported | 63.00 |

(Scores as reported by Kwaipilot for the respective version.)

## Which commits you need

The qwen3.6 engine is not merged upstream yet; pick the branch matching your
setup (all on [github.com/kreuzzelg/colibri](https://github.com/kreuzzelg/colibri)):

| Setup | Branch / commit | Upstream status |
|---|---|---|
| **CPU, gs64 (this container)** | [`gs64-ab`](https://github.com/kreuzzelg/colibri/tree/gs64-ab) (`2f9aab2`) — engine + group-scaled GEMV `matmul_q_gs` | engine submitted as [#712](https://github.com/JustVugg/colibri/pull/712) |
| **CUDA VRAM expert tier + gs64 (recommended, 1–2 GPUs)** | [`gs64-gpu`](https://github.com/kreuzzelg/colibri/tree/gs64-gpu) (`de5dde7`) = gs64 tier upload (`90450ec`) + upstream-dev backend incl. the async grouped-int4 fix | tier submitted as [#713](https://github.com/JustVugg/colibri/pull/713); backend fix as [#762](https://github.com/JustVugg/colibri/pull/762) |
| Per-row containers on the plain engine | [`upstream-pr1-qwen36-engine`](https://github.com/kreuzzelg/colibri/tree/upstream-pr1-qwen36-engine) (`16b945a`) | = #712 |

The backend fix **#762 is required for correct gs64 output on the GPU tier** —
without it the async decode path silently applies per-row scales to grouped
containers. Validated on this exact combination: CPU-vs-GPU-tier logits cosine
1.0 over 24 greedy tokens, token-identical output.

## Run it

```bash
git clone -b gs64-gpu https://github.com/kreuzzelg/colibri && cd colibri/c
# GPU tier (CUDA):
nvcc -O3 -std=c++17 -arch=native -c backend_cuda.cu -o backend_cuda.o
gcc -O3 -march=x86-64-v3 -fopenmp -pthread qwen36.c qwen36_tier.c vulkan_gemv.c \
    backend_cuda.o -o qwen36 -lm -lcudart -lstdc++

SNAP=$(python -c "from huggingface_hub import snapshot_download; \
print(snapshot_download('Kreuzzelg/kat-coder-v2.5-dev-colibri-i4-gs64'))")
printf 'Write a Python function that returns the n-th Fibonacci number using memoization.\n' > prompt.txt
SNAP=$SNAP N_NEW=300 COLI_CUDA=1 HEAT_FILE=heat.bin \
  OMP_NUM_THREADS=<physical cores> OMP_WAIT_POLICY=ACTIVE OMP_PROC_BIND=close \
  ./qwen36 256 4 prompt.txt
# CPU only: build from the gs64-ab branch with `make -C c qwen36` and drop COLI_CUDA.
```

The engine reads `expert_gs` from `qwen36_meta.json` and picks the group-scaled
GEMV automatically. `HEAT_FILE` persists expert-routing heat: the second run
starts with the right experts in VRAM.

## Measured (Threadripper PRO 3945WX 12C, RTX 3070 8 GB + Quadro RTX 4000 8 GB, 200-token decode)

| | 2×8 GB GPUs | 1×8 GB GPU (Quadro) |
|---|---|---|
| tok/s cold / warm | 10.4 / **11.3** | 9.3 / **10.4** |
| VRAM hit rate cold / warm | 75 % / 100 % | 39 % / 97 % |
| VRAM used | 13.2 GiB total | 6.6 GiB |
| peak RSS | 30 GB | 40 GB |
| TTFT (49-token prompt) | 1.9 s | 2.1 s |

Verified qualitatively: correct, clean Python (memoization, docstring, type
hints, edge cases) on the Fibonacci prompt; coherent 200-token continuations
under the benchmark prompt.

## Caveats

- **Dev checkpoint:** for serious use, handle the chat template and EOS
  yourself — in raw-completion mode the model keeps writing after the code.
- The 44 % accuracy A/B was measured on the sibling Qwen3.6-35B-A3B base with
  the same converter/engine; no separate A/B was run for the KAT finetune.

## Credits

- Model: [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) (Apache-2.0), a finetune of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)
- colibri engine & concept: [JustVugg/colibri](https://github.com/JustVugg/colibri)
- Original qwen36 engine + converter: [@minne100](https://huggingface.co/minne100) ([PR #602](https://github.com/JustVugg/colibri/pull/602)); group-scaled quantization, CUDA VRAM tier and the grouped-int4 backend fix added on top.