HermiHg/Qwen3.8-27B-DFlash2-Q2_K_S-MIX-GGUF

Qwen3.8-27B-DFlash2 — Q2_K_S-MIX draft (half the reference size)

A mixed-precision 2–3-bit re-quant of the DFlash 2 draft model for Qwen/Qwen3.8-27B, built to be 50% the size of the reference Q4_K_M checkpoint while keeping ~98% of its throughput. It is not a standalone language model: it runs as the speculative draft alongside the target model, exactly like the upstream release (see below).

This repo ships one file:

File	Size	bpw
Qwen3.8-27B-DFlash2-Q2_K_S-MIX.gguf	545 MiB	2.37

What this file is

Q2_K_S-MIX is a mixed-precision quant that compresses the large feed-forward blocks hard while keeping the small, high-impact tensors (the path selector and feature projection) more precise, so it lands at half the reference size with only a small acceptance loss.

Measured performance (vs the reference Q4_K_M)

Both drafts served with llama-server (DFlash 2, PR #27342) against the Qwen3.8-27B target on a 24 GB NVIDIA GPU, one fixed conversational prompt with reasoning, temperature 1.0, concurrency 1, 3 replicates.

n_max	Metric	Q4_K_M (1,090 MiB)	Q2_K (673 MiB)	Q2_K_S-MIX (545 MiB)	Ratio (vs Q4)
3	Mean draft length	2.80	2.65	2.68	0.96
4	Mean draft length	3.01	2.86	2.89	0.96
2	Throughput (tok/s)	94.3	94.0	93.8	0.99
3	Throughput (tok/s)	106.8	102.6	104.0	0.97
4	Throughput (tok/s)	110.0	105.5	107.7	0.98
—	Size	1,090 MiB (4.76 bpw)	673 MiB (2.93 bpw)	545 MiB (2.37 bpw)	0.50

The 2.37-bit draft accepts slightly fewer tokens per step than the 4.76-bit reference (e.g. 2.68 vs 2.80 at n_max=3), which shows up as ~3% less throughput. Because DFlash 2 is lossless, this costs speed, not quality — for the same prompt the output is accepted by the same target at the same quality; the smaller draft just needs marginally more verification steps.

How it was built (changes vs the reference)

Built clean from the upstream BF16 draft (incoai/Qwen3.8-27B-DFlash2 GGUF) with llama-quantize on a build with DFlash 2 support (PR #27342). No dequant-from-quant: the source is the full-precision checkpoint. It is a Q2_K_S base with per-tensor --tensor-type overrides:

Component	Tensors	Quant
Feed-forward (SwiGLU gate/up/down)	~69% of params	iq2_xxs
Token-path selector (hidden / predecessor / successor)	~7%	iq3_s
Feature projection fc	5120 × 25600	iq3_xxs
Two-tap dynamic-conv projections (attn + ffn)	2 × 5 blocks	iq2_xxs
Attention (q / k / output, value)	per block	Q2_K_S default (attn_v promoted to q4_k)
Layer norms + conv bases	32 tensors	f32 (held, not quantized)

The feed-forward block is 69% of the parameters, so it carries the size savings; the selector and fc are kept at ~3-bit because they drive which tokens the draft proposes (acceptance), and the norms/conv-bases stay full precision.

Usage

Build llama.cpp with DFlash 2 support (see the upstream Quick Start below), then serve with this checkpoint as the draft:

llama-server \
  -hf <your-target-repo>/Qwen3.8-27B-GGUF:<target-file> \
  -hfd HermiHg/Qwen3.8-27B-DFlash2-Q2_K_S-MIX-GGUF:Q2_K_S \
  --spec-type draft-dflash \
  --spec-draft-n-max 3

---

Qwen3.8-27B-DFlash2-GGUF

Blog | GitHub

This repository contains GGUF conversions of incoai/Qwen3.8-27B-DFlash2, the DFlash 2 draft model for Qwen/Qwen3.8-27B. It is not a standalone language model: it runs inside a speculative decoding server and drafts tokens for the target model to verify. The checkpoints are also mirrored at z-lab/Qwen3.8-27B-DFlash2-GGUF.

DFlash 2 is a block-diffusion drafter for speculative decoding. It predicts a whole block of tokens in a single pass and keeps the top candidates at every position. A lightweight selector then traces one coherent path through them. Two-tap dynamic convolutions in the backbone keep the draft from decaying toward the end of the block. Decoding is lossless: greedy output matches the target model exactly, and sampling preserves its distribution.

File	Size
Qwen3.8-27B-DFlash2-Q4_K_M.gguf	1.1 GB
Qwen3.8-27B-DFlash2-Q8_0.gguf	2.0 GB
Qwen3.8-27B-DFlash2-BF16.gguf	3.8 GB

Quick Start

Build llama.cpp with DFlash 2 support (PR #27342):

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git fetch origin pull/27342/head:pr-27342
git switch pr-27342

# NVIDIA CUDA
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
cmake --build build -j

# Apple Silicon
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON
cmake --build build -j

Then serve:

./build/bin/llama-server \
  -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M \
  -hfd incoai/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M \
  --spec-type draft-dflash \
  --spec-draft-n-max 7

See the blog post for other engines and more details.

Evaluation

• Target: ggml-org/Qwen3.8-27B-GGUF, Q4_K_M
• Sampling: Qwen3.8's officially recommended parameters (temperature 1.0, top-p 0.95, top-k 20), with xhigh reasoning effort
• Maximum new tokens: 2048
• Prompts: the first eight GSM8K test examples

Acceptance Length

Acceptance length is the per-request mean of completion tokens divided by verification steps. Higher is better.

Draft GGUF	Acceptance Length
BF16	5.28
Q8_0	5.13
Q4_K_M	5.39

Full evaluations of the base checkpoint are on the main model card.

Citation

If you find DFlash 2 useful, please cite:

@misc{inco2026dflash2,
  title  = {{DFlash 2: Keep Drafting Parallel}},
  author = {{Inco AI}},
  year   = {2026},
  month  = {August},
  url    = {https://inco.ai/blog/dflash2/}
}

Please also cite the original DFlash paper:

@inproceedings{chen2026dflash,
  title     = {{DFlash: Block Diffusion for Flash Speculative Decoding}},
  author    = {Chen, Jian and Liang, Yesheng and Liu, Zhijian},
  booktitle = {International Conference on Machine Learning (ICML)},
  year      = {2026}
}