| --- |
| pipeline_tag: text-generation |
| library_name: transformers |
| base_model: |
| - Qwen/Qwen3.5-397B-A17B |
| license: apache-2.0 |
| inference: false |
| tags: |
| - dflash |
| - speculative-decoding |
| - speculative-decoding-draft |
| - block-diffusion |
| - draft-model |
| - diffusion-language-model |
| - efficiency |
| - qwen |
| - qwen3 |
| - qwen3.5 |
| - sglang |
| --- |
| |
| # Qwen3.5-397B-A17B-DFlash |
|
|
| [Paper](https://arxiv.org/abs/2602.06036) | [Github](https://github.com/z-lab/dflash) | [Blog](https://z-lab.ai/projects/dflash) |
|
|
| This is a joint release collaboration between [Z-Lab](https://z-lab.ai), [Modal](https://modal.com), and [SGLang](https://github.com/sgl-project/sglang). The draft model is mirrored across the following Hugging Face repositories: |
|
|
| - [`z-lab/Qwen3.5-397B-A17B-DFlash`](https://huggingface.co/z-lab/Qwen3.5-397B-A17B-DFlash) |
| - [`modal-labs/Qwen3.5-397B-A17B-DFlash`](https://huggingface.co/modal-labs/Qwen3.5-397B-A17B-DFlash) |
| - [`lmsys/Qwen3.5-397B-A17B-DFlash`](https://huggingface.co/lmsys/Qwen3.5-397B-A17B-DFlash) |
|
|
| This repository contains a DFlash draft model for `Qwen/Qwen3.5-397B-A17B`. It is not a standalone language model. It is intended to be paired with the target model in a speculative decoding server. |
|
|
| DFlash uses a lightweight block diffusion draft model to propose multiple tokens in parallel. The target model verifies those proposals, improving serving throughput while preserving the target model's output distribution. |
|
|
| ## Quick Start |
|
|
| ### Installation |
|
|
| DFlash serving requires a recent SGLang build. We recommend installing SGLang from a commit at or after `ec36dde58083aca8f26c3740332498a11a06debf`, which includes DFlash v2 overlap scheduling, DFlash sliding-window draft attention, and the required FlashInfer integration. |
|
|
| ```bash |
| pip install -U \ |
| "git+https://github.com/sgl-project/sglang.git@ec36dde58083aca8f26c3740332498a11a06debf#subdirectory=python" \ |
| transformers hf_xet |
| ``` |
|
|
| For production serving, use a CUDA image with Blackwell support, such as `lmsysorg/sglang:v0.5.13-cu130`, and install the pinned SGLang commit inside the image. |
|
|
| ### Launch Server |
|
|
| This model should be used with an inference server that supports DFlash speculative decoding. An example SGLang deployment is: |
|
|
| ```bash |
| export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 |
| |
| python -m sglang.launch_server \ |
| --model-path Qwen/Qwen3.5-397B-A17B \ |
| --trust-remote-code \ |
| --speculative-algorithm DFLASH \ |
| --speculative-draft-model-path z-lab/Qwen3.5-397B-A17B-DFlash \ |
| --speculative-dflash-block-size 8 \ |
| --speculative-draft-attention-backend fa4 \ |
| --attention-backend trtllm_mha \ |
| --linear-attn-prefill-backend triton \ |
| --linear-attn-decode-backend flashinfer \ |
| --mamba-scheduler-strategy extra_buffer \ |
| --tp-size 8 \ |
| --max-running-requests 32 \ |
| --cuda-graph-max-bs-decode 32 \ |
| --cuda-graph-backend-prefill tc_piecewise \ |
| --enable-flashinfer-allreduce-fusion \ |
| --mem-fraction-static 0.8 \ |
| --host 0.0.0.0 \ |
| --port 30000 |
| ``` |
|
|
| Block size `8` is the recommended default for higher-concurrency serving. Block size `16` gives longer accept lengths and the best concurrency-1 throughput in most workloads. |
|
|
| ## Benchmark Results |
|
|
| We benchmarked DFlash against the autoregressive baseline and Qwen3.5's built-in MTP draft path. DFlash reaches up to `4.31x` speedup at concurrency 1 and `2.77x` at concurrency 32. Across the benchmark suite, DFlash delivers higher throughput than MTP at every tested setting. |
|
|
| ### Setup |
|
|
| - Runtime: SGLang on 8x NVIDIA B200 GPUs, tensor parallel size 8, `bfloat16` |
| - Backends: `trtllm_mha` target attention, `fa4` DFlash draft attention, `flashinfer` linear attention |
| - Workloads: GSM8K, MATH500, HumanEval, MBPP, and MT-Bench with the Qwen chat template |
| - Decoding: greedy, thinking enabled, max output length 4096 tokens |
| - Measurement: 5 independent runs per configuration at concurrency 1 and 32 with continuous batching |
| - Throughput: generated output tokens / wall-clock benchmark time, including prefill and scheduling |
| - Accept length: `completion_tokens / spec_verify_ct` per generation turn, averaged across generation turns |
|
|
| For reproduction, the evaluation scripts, Modal runner, runtime patches, and raw benchmark outputs are included under [`benchmark/`](https://huggingface.co/modal-labs/Qwen3.5-397B-A17B-DFlash/tree/main/benchmark). |
|
|
| ### Throughput and Speedup |
|
|
| Each cell is `output tok/s (speedup)`. Bold marks the fastest speculative configuration in each row. |
|
|
| #### Concurrency 1 |
|
|
| | Workload | Baseline | MTP steps=3 | DFlash block=4 | MTP steps=7 | DFlash block=8 | MTP steps=15 | DFlash block=16 | |
| | --- | --- | --- | --- | --- | --- | --- | --- | |
| | gsm8k | 204.6 (1.00x) | 484.5 (2.37x) | 530.0 (2.59x) | 536.7 (2.62x) | 689.0 (3.37x) | 439.2 (2.15x) | **711.3 (3.48x)** | |
| | math500 | 204.4 (1.00x) | 505.2 (2.47x) | 551.3 (2.70x) | 589.4 (2.88x) | 762.6 (3.73x) | 499.8 (2.44x) | **831.9 (4.07x)** | |
| | humaneval | 202.9 (1.00x) | 483.0 (2.38x) | 543.6 (2.68x) | 557.9 (2.75x) | 752.5 (3.71x) | 480.5 (2.37x) | **874.6 (4.31x)** | |
| | mbpp | 204.6 (1.00x) | 487.4 (2.38x) | 550.3 (2.69x) | 543.7 (2.66x) | 751.2 (3.67x) | 445.6 (2.18x) | **807.8 (3.95x)** | |
| | mt-bench | 202.6 (1.00x) | 442.9 (2.19x) | 473.6 (2.34x) | 441.5 (2.18x) | **545.9 (2.69x)** | 338.0 (1.67x) | 515.3 (2.54x) | |
|
|
| #### Concurrency 32 |
|
|
| | Workload | Baseline | MTP steps=3 | DFlash block=4 | MTP steps=7 | DFlash block=8 | MTP steps=15 | DFlash block=16 | |
| | --- | --- | --- | --- | --- | --- | --- | --- | |
| | gsm8k | 2597.7 (1.00x) | 4462.3 (1.72x) | 5078.0 (1.95x) | 4902.9 (1.89x) | **6158.9 (2.37x)** | 4079.5 (1.57x) | 5648.8 (2.17x) | |
| | math500 | 2615.5 (1.00x) | 4838.6 (1.85x) | 5318.6 (2.03x) | 5601.6 (2.14x) | **6910.5 (2.64x)** | 4855.0 (1.86x) | 6857.8 (2.62x) | |
| | humaneval | 2452.7 (1.00x) | 3838.1 (1.56x) | 5081.4 (2.07x) | 4347.4 (1.77x) | 6666.0 (2.72x) | 3917.2 (1.60x) | **6783.7 (2.77x)** | |
| | mbpp | 2582.3 (1.00x) | 4129.8 (1.60x) | 5265.7 (2.04x) | 4461.0 (1.73x) | **6681.4 (2.59x)** | 3783.7 (1.47x) | 6344.1 (2.46x) | |
| | mt-bench | 2537.3 (1.00x) | 4024.6 (1.59x) | 4425.3 (1.74x) | 3991.7 (1.57x) | **4763.2 (1.88x)** | 3100.3 (1.22x) | 4016.0 (1.58x) | |
|
|
| ### Accept Length |
|
|
| Mean accept length at concurrency 1. Bold marks the higher value in each matched MTP/DFlash pair. |
|
|
| | Workload | MTP steps=3 | DFlash block=4 | MTP steps=7 | DFlash block=8 | MTP steps=15 | DFlash block=16 | |
| | --- | --- | --- | --- | --- | --- | --- | |
| | gsm8k | **3.578** | 3.548 | 5.612 | **5.738** | 7.006 | **7.696** | |
| | math500 | **3.636** | 3.622 | 5.810 | **6.003** | 7.294 | **8.212** | |
| | humaneval | 3.643 | **3.697** | 5.893 | **6.292** | 7.541 | **9.344** | |
| | mbpp | 3.532 | **3.588** | 5.357 | **5.860** | 6.398 | **7.801** | |
| | mt-bench | **3.244** | 3.188 | 4.567 | **4.584** | 5.256 | **5.523** | |
|
|
| ## Citation |
|
|
| If you find DFlash useful, please cite the original paper: |
|
|
| ```bibtex |
| @article{chen2026dflash, |
| title = {{DFlash: Block Diffusion for Flash Speculative Decoding}}, |
| author = {Chen, Jian and Liang, Yesheng and Liu, Zhijian}, |
| journal = {arXiv preprint arXiv:2602.06036}, |
| year = {2026} |
| } |
| ``` |
|
|
