--- license: apache-2.0 library_name: llama.cpp pipeline_tag: text-generation base_model: - incoai/Qwen3.8-27B-DFlash2 inference: false tags: - gguf - dflash2 - speculative-decoding - draft-model - llama.cpp - mixed-precision - low-bit - memory-efficient --- # Qwen3.8-27B-DFlash2 — Q2_K_S-MIX draft (half the reference size) A **mixed-precision 2–3-bit re-quant** of the DFlash 2 draft model for [`Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B), built to be **50% the size** of the reference `Q4_K_M` checkpoint while keeping **~98% of its throughput**. It is not a standalone language model: it runs as the speculative draft alongside the target model, exactly like the upstream release (see below). `Q2_K_S-MIX` is a mixed-precision quant that compresses the large feed-forward blocks hard while keeping the small, high-impact tensors (the path selector and feature projection) more precise, so it lands at half the reference size with only a small acceptance loss. ## Measured performance (vs the reference `Q4_K_M`) Both drafts served with `llama-server` (DFlash 2, PR #27342) against the `Qwen3.8-27B` target on a 24 GB NVIDIA GPU, one fixed conversational prompt with reasoning, temperature 1.0, concurrency 1, 3 replicates. | `n_max` | Metric | `Q4_K_M` (1,090 MiB) | `Q2_K` (673 MiB) | **Q2_K_S-MIX** (545 MiB) | Ratio (vs Q4) | | :-- | :--- | ---: | ---: | ---: | :-- | | 3 | Mean draft length | 2.80 | 2.65 | 2.68 | 0.96 | | 4 | Mean draft length | 3.01 | 2.86 | 2.89 | 0.96 | | 2 | Throughput (tok/s) | 94.3 | 94.0 | 93.8 | 0.99 | | 3 | Throughput (tok/s) | 106.8 | 102.6 | 104.0 | 0.97 | | 4 | Throughput (tok/s) | 110.0 | 105.5 | 107.7 | 0.98 | | — | Size | 1,090 MiB (4.76 bpw) | 673 MiB (2.93 bpw) | 545 MiB (2.37 bpw) | **0.50** | The 2.37-bit draft accepts slightly fewer tokens per step than the 4.76-bit reference (e.g. 2.68 vs 2.80 at `n_max=3`), which shows up as ~3% less throughput. Because DFlash 2 is lossless, this costs **speed, not quality** — for the same prompt the output is accepted by the same target at the same quality; the smaller draft just needs marginally more verification steps. ## How it was built (changes vs the reference) Built clean from the upstream **BF16** draft (`incoai/Qwen3.8-27B-DFlash2` GGUF) with [`llama-quantize`](https://github.com/ggml-org/llama.cpp) on a build with DFlash 2 support ([PR #27342](https://github.com/ggml-org/llama.cpp/pull/27342)). No dequant-from-quant: the source is the full-precision checkpoint. It is a `Q2_K_S` base with per-tensor `--tensor-type` overrides: | Component | Tensors | Quant | | :--- | :--- | :--- | | Feed-forward (SwiGLU gate/up/down) | ~69% of params | `iq2_xxs` | | Token-path selector (hidden / predecessor / successor) | ~7% | `iq3_s` | | Feature projection `fc` | `5120 × 25600` | `iq3_xxs` | | Two-tap dynamic-conv projections (attn + ffn) | 2 × 5 blocks | `iq2_xxs` | | Attention (q / k / output, value) | per block | `Q2_K_S` default (`attn_v` promoted to `q4_k`) | | Layer norms + conv bases | 32 tensors | `f32` (held, not quantized) | The feed-forward block is 69% of the parameters, so it carries the size savings; the selector and `fc` are kept at ~3-bit because they drive which tokens the draft proposes (acceptance), and the norms/conv-bases stay full precision. ## Usage Build llama.cpp with DFlash 2 support (see the upstream Quick Start below), then serve with this checkpoint as the draft: ```bash llama-server \ -hf /Qwen3.8-27B-GGUF: \ -hfd HermiHg/Qwen3.8-27B-DFlash2-Q2_K_S-MIX-GGUF:Q2_K_S \ --spec-type draft-dflash \ --spec-draft-n-max 3 ``` --- # Qwen3.8-27B-DFlash2-GGUF [Blog](https://inco.ai/blog/dflash2/) | [GitHub](https://github.com/z-lab/dflash) This repository contains GGUF conversions of [`incoai/Qwen3.8-27B-DFlash2`](https://huggingface.co/incoai/Qwen3.8-27B-DFlash2), the DFlash 2 draft model for [`Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B). It is not a standalone language model: it runs inside a speculative decoding server and drafts tokens for the target model to verify. The checkpoints are also mirrored at [`z-lab/Qwen3.8-27B-DFlash2-GGUF`](https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF). DFlash 2 is a block-diffusion drafter for speculative decoding. It predicts a whole block of tokens in a single pass and keeps the top candidates at every position. A lightweight selector then traces one coherent path through them. Two-tap dynamic convolutions in the backbone keep the draft from decaying toward the end of the block. Decoding is lossless: greedy output matches the target model exactly, and sampling preserves its distribution.
DFlash 2: parallel block drafting with a candidate path selector
| File | Size | | :--- | ---: | | `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` | 1.1 GB | | `Qwen3.8-27B-DFlash2-Q8_0.gguf` | 2.0 GB | | `Qwen3.8-27B-DFlash2-BF16.gguf` | 3.8 GB | ## Quick Start Build [llama.cpp](https://github.com/ggml-org/llama.cpp) with DFlash 2 support ([PR #27342](https://github.com/ggml-org/llama.cpp/pull/27342)): ```bash git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp git fetch origin pull/27342/head:pr-27342 git switch pr-27342 # NVIDIA CUDA cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON cmake --build build -j # Apple Silicon cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON cmake --build build -j ``` Then serve: ```bash ./build/bin/llama-server \ -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M \ -hfd incoai/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M \ --spec-type draft-dflash \ --spec-draft-n-max 7 ``` See the [blog post](https://inco.ai/blog/dflash2/) for other engines and more details. ## Evaluation - Target: [`ggml-org/Qwen3.8-27B-GGUF`](https://huggingface.co/ggml-org/Qwen3.8-27B-GGUF), `Q4_K_M` - Sampling: Qwen3.8's officially recommended parameters (temperature 1.0, top-p 0.95, top-k 20), with `xhigh` reasoning effort - Maximum new tokens: 2048 - Prompts: the first eight GSM8K test examples ### Acceptance Length Acceptance length is the per-request mean of completion tokens divided by verification steps. Higher is better. | Draft GGUF | Acceptance Length | | :--- | ---: | | BF16 | 5.28 | | Q8_0 | 5.13 | | Q4_K_M | 5.39 | Full evaluations of the base checkpoint are on the [main model card](https://huggingface.co/incoai/Qwen3.8-27B-DFlash2). ## Citation If you find DFlash 2 useful, please cite: ```bibtex @misc{inco2026dflash2, title = {{DFlash 2: Keep Drafting Parallel}}, author = {{Inco AI}}, year = {2026}, month = {August}, url = {https://inco.ai/blog/dflash2/} } ``` Please also cite the original DFlash paper: ```bibtex @inproceedings{chen2026dflash, title = {{DFlash: Block Diffusion for Flash Speculative Decoding}}, author = {Chen, Jian and Liang, Yesheng and Liu, Zhijian}, booktitle = {International Conference on Machine Learning (ICML)}, year = {2026} } ```