--- license: other license_name: kimi-k3 library_name: vllm base_model: moonshotai/Kimi-K3 pipeline_tag: text-generation tags: - dspark - speculative-decoding - draft-model - mla - vllm - torchspec - kimi-k3 --- ## Model Overview **Inferact/Kimi-K3-DSpark** is an **MLA-native DSpark** draft model that accelerates [Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3) on **[vLLM](https://github.com/vllm-project/vllm)**, served through vLLM's native `dspark` speculative method. The draft trains on **target hidden states extracted from vLLM itself** — the same engine that serves it, so the numerics it learns from are the numerics it meets at inference. [TorchSpec](https://github.com/lightseekorg/TorchSpec) provides the loop, streaming those hidden states from live target inference into concurrent FSDP draft training. **DSpark** = a block-diffusion backbone of 5 dense layers with non-causal attention, drafting 7 tokens in a single parallel pass, a low-rank sequential **Markov head** supplying the intra-block dependency, and a confidence head for resource-aware scheduling. Mirroring Kimi-K3's own MLA attention means draft and target share one KV layout (a compact 576-element latent per token), so the draft's pages unify with the target's cache — KV offloading and P/D disaggregation work with no separate page format. --- ## Performance ### Peak bs=1 decode: 464 tok/s Under low-entropy real reasoning workload — Kimi-K3 + DSpark on vLLM can achieve **464 tok/s** using the public `vllm/vllm-openai:kimi-k3` image on 4 × GB300 at `bs=1 & tensor-parallel-size=16`. ### Speculator acceptance Speculators do best on predictable, low-entropy work like the reasoning workload behind the 464 tok/s above, and worst on open-ended, high-entropy generation. So we measured **14 benchmarks** spanning math, code synthesis, real-world software engineering, multi-turn chat, RAG and QA, multilingual text, creative writing and long-context generation — all with the **Kimi-K3 chat template enabled** and production sampling parameters, with `temperature=0` reported alongside for reproducibility. Acceptance stays strong at long context too, verified on AA-LCR's ~95k-token multi-document prompts. That is the most comprehensive view we can give of how the speculator performs on real-world workloads. Acceptance length, with 7 speculative tokens: | benchmark | `temperature=0` | `temperature=1.0`, `top_p=0.95` | prompts | |---|---|---|---| | GSM8K | 5.64 | 5.44 | 1319 | | HumanEval | 5.34 | 5.07 | 164 | | MBPP | 4.44 | 4.31 | 256 | | SPEED-Bench · coding | 4.38 | 4.22 | 80 | | SPEED-Bench · multilingual | 4.21 | 4.10 | 80 | | SPEED-Bench · RAG | 4.11 | 3.97 | 80 | | MATH-500 | 3.82 | 3.77 | 500 | | SPEED-Bench · low-entropy, 10k input | 3.72 | 3.66 | 512 | | SWE-bench Pro | 3.35 | 3.11 | 128 | | AA-LCR · ~95k input | 3.19 | 3.23 | 100 | | MT-Bench | 3.14 | 3.06 | 80 | | SPEED-Bench · QA | 3.07 | 2.98 | 80 | | SPEED-Bench · writing | 2.79 | 2.69 | 80 | | AIME 2026 | 2.72 | 2.64 | 30 | | **mean** | **3.85** | **3.73** | | **Benchmarks:** [GSM8K](https://huggingface.co/datasets/openai/gsm8k), [MATH-500](https://huggingface.co/datasets/HuggingFaceH4/MATH-500), [AIME 2026](https://huggingface.co/datasets/math-ai/aime26), [HumanEval](https://huggingface.co/datasets/openai/openai_humaneval), [MBPP](https://huggingface.co/datasets/Muennighoff/mbpp), [SWE-bench Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro), [MT-Bench](https://huggingface.co/datasets/philschmid/mt-bench), [AA-LCR](https://huggingface.co/datasets/ArtificialAnalysis/AA-LCR) — 100 multi-document prompts of 71k–115k tokens — and six splits of [NVIDIA SPEED-Bench](https://huggingface.co/datasets/nvidia/SPEED-Bench): its `throughput_16k` low-entropy split at 10k-token input, plus five qualitative categories. --- ## Training **Data — all responses regenerated on-policy by Kimi-K3 itself**, so the draft learns the target's own reasoning traces and chat formatting. Prompts come from public datasets: - [`lightseekorg/kimi-mtp-dataset`](https://huggingface.co/datasets/lightseekorg/kimi-mtp-dataset) — general instruction prompts - [`nvidia/OpenCodeInstruct`](https://huggingface.co/datasets/nvidia/OpenCodeInstruct) — single-turn coding - A nine-category prompt pool from the NVIDIA **Nemotron** SFT/RL collections and [`CohereLabs/aya_dataset`](https://huggingface.co/datasets/CohereLabs/aya_dataset), spanning chat, code, multilingual, RAG/QA, math, structured output and safety. Evaluation-set prompts are excluded. **Method:** DSpark with block_size=7, trained on a combined CE + L1 distribution-distillation objective against the target's post-final-norm hidden state, bf16. The draft consumes Kimi-K3 auxiliary hidden states from target layers (2, 23, 47, 71, 89) of 93. Roughly **two epochs** in total, on GB300 nodes. Draft architecture, block size, sequence length, and loss weights are YAML-configurable — see the [TorchSpec repo](https://github.com/lightseekorg/TorchSpec). --- ## Quick Start ### Requirements For serving Kimi-K3 itself — hardware, parallelism and engine flags — follow the official vLLM recipe: **[recipes.vllm.ai/moonshotai/Kimi-K3](https://recipes.vllm.ai/moonshotai/Kimi-K3)**. ### Enable the draft Add to your Kimi-K3 `vllm serve` command: ```bash --speculative-config '{"method": "dspark", "model": "Inferact/Kimi-K3-DSpark", "num_speculative_tokens": 7, "attention_backend": "FLASHINFER_MLA", "draft_sample_method": "probabilistic", "rejection_sample_method": "block"}' ``` ### Sampling options vLLM offers two knobs on top of the defaults, both used in the `temperature=1.0` column above: - **`draft_sample_method`** — `probabilistic` samples the draft from its own distribution instead of taking its argmax. Pair it with a sampling client; use `greedy` when serving at `temperature=0` so the draft matches the client. - **`rejection_sample_method`** — `block` verifies the drafted block as a unit rather than token by token. It is a no-op under greedy decoding (at `temperature=0` verification reduces to a deterministic argmax match), so it only applies to the sampling configuration. The `temperature=0` column was produced with `{"draft_sample_method": "greedy"}` and no `rejection_sample_method`.