File size: 6,345 Bytes
d0888d9 2c9b9ef 84e1b65 d0888d9 2c9b9ef d0888d9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 | ---
license: apache-2.0
library_name: transformers
base_model:
- Kwaipilot/KAT-Coder-V2.5-Dev
inference: false
tags:
- dflash2
- speculative-decoding
- draft-model
- moe
- code
- speculators
- vllm
---
# KAT-Coder-V2.5-Dev-DFlash2-2.6B-A0.3B
A Mixture-of-Experts DFlash2 drafter for [`Kwaipilot/KAT-Coder-V2.5-Dev`](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev).
Modern MoE target models are well suited to expert offloading on memory-constrained systems and large-scale expert parallelism at datacenter scale. Applying the same sparse structure to speculative drafters is therefore a natural direction. Strong precedents include [`DeepSeek-V3`](https://huggingface.co/deepseek-ai/DeepSeek-V3) MTP and [`DeepSeek-V4-Flash-0731`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) DSpark.
This release explores the same direction with a DFlash2 drafter: approximately 2.6B total parameters with approximately 0.3B activated per token.
## Results
| Setting | Value |
| --------------------- | ------------------------------------------------------------------- |
| Engine | vLLM |
| Hardware | NVIDIA H200, TP1 |
| Concurrency | 1 |
| Target | `Kwaipilot/KAT-Coder-V2.5-Dev` BF16 |
| Thinking sampling | `temperature=1.0`, `top_p=0.95`, `top_k=20`, `presence_penalty=1.5` |
| Thinking mode | `enable_thinking=true` |
| Non-thinking sampling | `temperature=0.7`, `top_p=0.8`, `top_k=20`, `presence_penalty=1.5` |
| Non-thinking mode | `enable_thinking=false` |
| Max new tokens | 4,096 |
| Context | native 262,144 |
| Draft tokens | 7 per verification step |
| Benchmark | Samples | Thinking AL | Non-thinking AL |
| -------------------------- | ------: | ----------: | --------------: |
| AA-LCR | 100 | 2.8870 | 3.3894 |
| AIME26 | 30 | 2.8959 | 4.1591 |
| GSM8K | 1,319 | 4.2453 | 5.2930 |
| HumanEval | 164 | 4.0568 | 5.9836 |
| LiveCodeBench | 1,055 | 2.7090 | 3.5688 |
| LongBench-v2 | 402 | 2.4484 | 2.8255 |
| MATH500 | 500 | 3.6183 | 4.7679 |
| MBPP | 257 | 3.6543 | 4.5800 |
| MT-Bench | 80 | 2.5998 | 2.8009 |
| SWE-bench Pro | 731 | 3.1425 | 3.4428 |
| SpeedBench / coding | 80 | 3.3456 | 4.0887 |
| SpeedBench / humanities | 80 | 2.2735 | 2.7807 |
| SpeedBench / math | 80 | 2.4900 | 3.3639 |
| SpeedBench / multilingual | 80 | 3.0767 | 3.0240 |
| SpeedBench / qa | 80 | 2.5138 | 2.4313 |
| SpeedBench / rag | 80 | 3.3953 | 3.5579 |
| SpeedBench / reasoning | 80 | 2.8214 | 3.1751 |
| SpeedBench / roleplay | 80 | 1.8487 | 1.8848 |
| SpeedBench / stem | 80 | 2.3774 | 3.0061 |
| SpeedBench / summarization | 80 | 3.0177 | 2.6967 |
| SpeedBench / writing | 80 | 2.2223 | 2.3711 |
Accepted length is calculated from the raw server counters:
```text
accepted_length = 1 + accepted_tokens / draft_calls
```
## Architecture
| Setting | Value |
| --- | --- |
| Draft layers | 3 |
| Auxiliary hidden-state IDs | `[3, 12, 21, 30, 39]` |
| Hidden size | 2,048 |
| MoE FFN | 256 routed experts, top-8 routing |
| Routed expert intermediate | 512 |
| Shared expert | 1 expert, intermediate size 512 |
| Router | softmax, top-k normalization |
| Shared-expert gate | sigmoid |
| Attention | 32 Q heads / 4 KV heads |
| Head dimension | 128 |
| Attention window | sliding window 2,048 |
| Draft attention | non-causal inside the draft block |
| RoPE theta | 1e7 |
| Norm / activation | RMSNorm eps 1e-6 / SiLU |
| Dynamic convolution | kernel 2, group 16 |
| Candidate selector | rank 256, top-k 16 |
| Block | 7 draft tokens, `sample_from_anchor=true` |
| Draft vocabulary | 248,320 |
| Parameters | 2,638,551,808 total / 298,130,176 activated per token |
## Example: Run with vLLM
This is a drafter checkpoint, not a standalone language model. Pair it with KAT-Coder or a compatible quantized version:
```bash
vllm serve Kwaipilot/KAT-Coder-V2.5-Dev \
--tensor-parallel-size 1 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--language-model-only \
--speculative-config '{
"method": "dflash",
"model": "DaoCloud/KAT-Coder-V2.5-Dev-DFlash2-2.6B-A0.3B",
"num_speculative_tokens": 7
}'
```
This checkpoint currently requires a vLLM build containing both [vLLM PR #54154](https://github.com/vllm-project/vllm/pull/54154), which adds the `sample_from_anchor=true` DFlash query layout, and [vLLM PR #54475](https://github.com/vllm-project/vllm/pull/54475), which adds MoE DFlash2.
## Checkpoint
The BF16 checkpoint contains 63 tensors and 2,638,551,808 parameters.
| File | SHA256 |
| --- | --- |
| `model.safetensors` | `8e96fdfa924405443b566be3a961eb087cd6991665040b05d7c3ec739d5469ee` |
## Future work
- Adaptive verification: explore confidence-head-driven adaptive draft budgets for high-concurrency serving.
## Acknowledgements
This work builds on [`Kwaipilot/KAT-Coder-V2.5-Dev`](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev), [`vllm-project/speculators`](https://github.com/vllm-project/speculators), and [`vllm-project/vllm`](https://github.com/vllm-project/vllm).
## License
Apache-2.0.
|