Upload folder using huggingface_hub

README.md

@@ -0,0 +1,134 @@
+---
+license: apache-2.0
+library_name: transformers
+pipeline_tag: text-generation
+base_model:
+  - Kwaipilot/KAT-Coder-V2.5-Dev
+inference: false
+tags:
+  - dflash2
+  - speculative-decoding
+  - draft-model
+  - moe
+  - code
+  - speculators
+  - vllm
+---
+
+# KAT-Coder-V2.5-Dev-DFlash2-2.6B-A0.3B
+
+A Mixture-of-Experts DFlash2 drafter for [`Kwaipilot/KAT-Coder-V2.5-Dev`](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev).
+
+Modern MoE target models are well suited to expert offloading on memory-constrained systems and large-scale expert parallelism at datacenter scale. Applying the same sparse structure to speculative drafters is therefore a natural direction. Strong precedents include [`DeepSeek-V3`](https://huggingface.co/deepseek-ai/DeepSeek-V3) MTP and [`DeepSeek-V4-Flash-0731`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) DSpark.
+
+This release explores the same direction with a DFlash2 drafter: approximately 2.6B total parameters with approximately 0.3B activated per token.
+
+## Results
+
+| Setting               | Value                                                               |
+| --------------------- | ------------------------------------------------------------------- |
+| Engine                | vLLM                                                                |
+| Hardware              | NVIDIA H200, TP1                                                    |
+| Concurrency           | 1                                                                   |
+| Target                | `Kwaipilot/KAT-Coder-V2.5-Dev` BF16                                 |
+| Thinking sampling     | `temperature=1.0`, `top_p=0.95`, `top_k=20`, `presence_penalty=1.5` |
+| Thinking mode         | `enable_thinking=true`                                              |
+| Non-thinking sampling | `temperature=0.7`, `top_p=0.8`, `top_k=20`, `presence_penalty=1.5`  |
+| Non-thinking mode     | `enable_thinking=false`                                             |
+| Max new tokens        | 4,096                                                               |
+| Context               | native 262,144                                                      |
+| Draft tokens          | 7 per verification step                                             |
+
+| Benchmark                  | Samples | Thinking AL | Non-thinking AL |
+| -------------------------- | ------: | ----------: | --------------: |
+| AA-LCR                     |     100 |       *TBD* |           *TBD* |
+| AIME26                     |      30 |       *TBD* |           *TBD* |
+| GSM8K                      |   1,319 |       *TBD* |           *TBD* |
+| HumanEval                  |     164 |       *TBD* |           *TBD* |
+| LiveCodeBench              |   1,055 |       *TBD* |           *TBD* |
+| LongBench-v2               |     402 |       *TBD* |           *TBD* |
+| MATH500                    |     500 |       *TBD* |           *TBD* |
+| MBPP                       |     257 |       *TBD* |           *TBD* |
+| MT-Bench                   |      80 |       *TBD* |           *TBD* |
+| SWE-bench Pro              |     731 |       *TBD* |           *TBD* |
+| SpeedBench / coding        |      80 |       *TBD* |           *TBD* |
+| SpeedBench / humanities    |      80 |       *TBD* |           *TBD* |
+| SpeedBench / math          |      80 |       *TBD* |           *TBD* |
+| SpeedBench / multilingual  |      80 |       *TBD* |           *TBD* |
+| SpeedBench / qa            |      80 |       *TBD* |           *TBD* |
+| SpeedBench / rag           |      80 |       *TBD* |           *TBD* |
+| SpeedBench / reasoning     |      80 |       *TBD* |           *TBD* |
+| SpeedBench / roleplay      |      80 |       *TBD* |           *TBD* |
+| SpeedBench / stem          |      80 |       *TBD* |           *TBD* |
+| SpeedBench / summarization |      80 |       *TBD* |           *TBD* |
+| SpeedBench / writing       |      80 |       *TBD* |           *TBD* |
+
+Accepted length is calculated from the raw server counters:
+
+```text
+accepted_length = 1 + accepted_tokens / draft_calls
+```
+
+## Architecture
+
+| Setting | Value |
+| --- | --- |
+| Draft layers | 3 |
+| Auxiliary hidden-state IDs | `[3, 12, 21, 30, 39]` |
+| Hidden size | 2,048 |
+| MoE FFN | 256 routed experts, top-8 routing |
+| Routed expert intermediate | 512 |
+| Shared expert | 1 expert, intermediate size 512 |
+| Router | softmax, top-k normalization |
+| Shared-expert gate | sigmoid |
+| Attention | 32 Q heads / 4 KV heads |
+| Head dimension | 128 |
+| Attention window | sliding window 2,048 |
+| Draft attention | non-causal inside the draft block |
+| RoPE theta | 1e7 |
+| Norm / activation | RMSNorm eps 1e-6 / SiLU |
+| Dynamic convolution | kernel 2, group 16 |
+| Candidate selector | rank 256, top-k 16 |
+| Block | 7 draft tokens, `sample_from_anchor=true` |
+| Draft vocabulary | 248,320 |
+| Parameters | 2,638,551,808 total / 298,130,176 activated per token |
+
+## Example: Run with vLLM
+
+This is a drafter checkpoint, not a standalone language model. Pair it with KAT-Coder or a compatible quantized version:
+
+```bash
+vllm serve Kwaipilot/KAT-Coder-V2.5-Dev \
+  --tensor-parallel-size 1 \
+  --reasoning-parser qwen3 \
+  --enable-auto-tool-choice \
+  --tool-call-parser qwen3_coder \
+  --language-model-only \
+  --speculative-config '{
+    "method": "dflash",
+    "model": "DaoCloud/KAT-Coder-V2.5-Dev-DFlash2-2.6B-A0.3B",
+    "num_speculative_tokens": 7
+  }'
+```
+
+This checkpoint currently requires a vLLM build containing both [vLLM PR #54154](https://github.com/vllm-project/vllm/pull/54154), which adds the `sample_from_anchor=true` DFlash query layout, and [vLLM PR #54475](https://github.com/vllm-project/vllm/pull/54475), which adds MoE DFlash2.
+
+## Checkpoint
+
+The BF16 checkpoint contains 63 tensors and 2,638,551,808 parameters.
+
+| File | SHA256 |
+| --- | --- |
+| `model.safetensors` | _TBD_ |
+
+## Future work
+
+- Adaptive verification: explore confidence-head-driven adaptive draft budgets for high-concurrency serving.
+
+## Acknowledgements
+
+This work builds on [`Kwaipilot/KAT-Coder-V2.5-Dev`](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev), [`vllm-project/speculators`](https://github.com/vllm-project/speculators), and [`vllm-project/vllm`](https://github.com/vllm-project/vllm).
+
+## License
+
+Apache-2.0.

config.json

@@ -0,0 +1,102 @@
+{
+  "architectures": [
+    "DFlash2DraftModel"
+  ],
+
+  "aux_hidden_state_layer_ids": [
+    3,
+    12,
+    21,
+    30,
+    39
+  ],
+
+  "block_size": 7,
+  "sample_from_anchor": true,
+
+  "draft_vocab_size": 248320,
+  "mask_token_id": 248077,
+
+  "conv_kernel_size": 2,
+  "conv_group_size": 16,
+
+  "selector_rank": 256,
+  "selector_top_k": 16,
+
+  "draft_ffn_type": "moe",
+  "num_experts": 256,
+  "num_experts_per_tok": 8,
+  "moe_intermediate_size": 512,
+  "shared_expert_intermediate_size": 512,
+  "norm_topk_prob": true,
+
+  "sliding_window_non_causal": true,
+
+  "dtype": "bfloat16",
+  "tie_word_embeddings": false,
+
+  "speculators_model_type": "dflash2",
+
+  "speculators_config": {
+    "algorithm": "dflash2",
+    "default_proposal_method": "greedy",
+    "proposal_methods": [
+      {
+        "accept_tolerance": 0.0,
+        "proposal_type": "greedy",
+        "speculative_tokens": 7,
+        "verifier_accept_k": 1
+      }
+    ],
+    "verifier": {
+      "architectures": [
+        "Qwen3_5MoeForConditionalGeneration"
+      ],
+      "name_or_path": "Kwaipilot/KAT-Coder-V2.5-Dev"
+    }
+  },
+
+  "transformer_layer_config": {
+    "model_type": "qwen3",
+
+    "vocab_size": 248320,
+    "hidden_size": 2048,
+    "intermediate_size": 512,
+    "num_hidden_layers": 3,
+
+    "num_attention_heads": 32,
+    "num_key_value_heads": 4,
+    "head_dim": 128,
+
+    "hidden_act": "silu",
+    "rms_norm_eps": 1e-06,
+    "initializer_range": 0.02,
+
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+
+    "use_sliding_window": true,
+    "sliding_window": 2048,
+    "max_window_layers": 0,
+
+    "layer_types": [
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention"
+    ],
+
+    "max_position_embeddings": 262144,
+
+    "rope_parameters": {
+      "rope_theta": 10000000.0,
+      "rope_type": "default"
+    },
+
+    "use_cache": true,
+    "tie_word_embeddings": false,
+
+    "bos_token_id": null,
+    "eos_token_id": null,
+    "pad_token_id": null
+  }
+}

config.py

@@ -0,0 +1,42 @@
+from typing import Literal
+
+from pydantic import Field, model_validator
+
+from speculators import SpeculatorModelConfig
+from speculators.models.dflash.config import DFlashSpeculatorConfig
+
+
+@SpeculatorModelConfig.register("dflash2")
+class DFlash2SpeculatorConfig(DFlashSpeculatorConfig):
+    """DFlash2 draft-model configuration."""
+
+    speculators_model_type: Literal["dflash2"] = "dflash2"  # type: ignore[assignment]
+
+    architectures: list[str] = Field(
+        default_factory=lambda: ["DFlash2DraftModel"],
+    )
+
+    sliding_window_non_causal: bool = True
+
+    conv_kernel_size: int = Field(default=2, ge=1)
+    conv_group_size: int = Field(default=16, ge=1)
+    selector_rank: int = Field(default=256, ge=1)
+    selector_top_k: int = Field(default=16, ge=1)
+
+    draft_ffn_type: Literal["dense", "moe"] = "dense"
+    num_experts: int = Field(default=256, ge=1)
+    num_experts_per_tok: int = Field(default=8, ge=1)
+    moe_intermediate_size: int = Field(default=512, ge=1)
+    shared_expert_intermediate_size: int = Field(default=512, ge=1)
+
+    @model_validator(mode="after")
+    def validate_moe_routing(self) -> "DFlash2SpeculatorConfig":
+        if (
+            self.draft_ffn_type == "moe"
+            and self.num_experts_per_tok > self.num_experts
+        ):
+            raise ValueError(
+                "num_experts_per_tok cannot exceed num_experts: "
+                f"{self.num_experts_per_tok} > {self.num_experts}."
+            )
+        return self

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8e96fdfa924405443b566be3a961eb087cd6991665040b05d7c3ec739d5469ee
+size 5277110680