---
license: mit
library_name: gguf
base_model: z-lab/Qwen3.5-27B-DFlash
tags:
  - gguf
  - speculative-decoding
  - dflash
  - drafter
  - llama.cpp
  - quantized
---

# Qwen3.5-27B-DFlash — GGUF (Q4_K_M)

**Q4_K_M llama.cpp quantization** of [`z-lab/Qwen3.5-27B-DFlash`](https://huggingface.co/z-lab/Qwen3.5-27B-DFlash), the block-diffusion drafter for DFlash speculative decoding. Pair it with any Qwen3.5-27B target model (e.g. `Qwen/Qwen3.5-27B`).

Only **Q4_K_M** is published here — on our RTX 3090 benchmarks Q4_K_M was the optimal operating point for this drafter; Q8_0 and F16 did not produce better end-to-end decode throughput, so they're intentionally omitted to keep the pairing simple.

## Requirements

DFlash speculative decoding is **not yet in upstream `llama.cpp`**. You need the fork that carries the tape-replay rollback, hidden-state capture, and tree-aware SSM kernels:

- Fork: [`spiritbuun/buun-llama-cpp`](https://github.com/spiritbuun/buun-llama-cpp) (branch `master`)
- Built with: `cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=ON -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON`

## Usage

### `llama-server`
```bash
./build/bin/llama-server \
    -m   /path/to/Qwen3.5-27B-target.Q4_K_M.gguf \
    -md  /path/to/dflash-draft-q4_k_m.gguf \
    --spec-type dflash \
    -ngl 99 -ngld 99 \
    -np 1 -c 6048 -cd 256 \
    -fa on -b 256 -ub 64 \
    --host 0.0.0.0 --port 8080 --jinja
```

### `llama-speculative-simple`
```bash
./build/bin/llama-speculative-simple \
    -m   /path/to/Qwen3.5-27B-target.Q4_K_M.gguf \
    -md  /path/to/dflash-draft-q4_k_m.gguf \
    --spec-type dflash \
    -ngl 99 -ngld 99 \
    -c 4096 --draft-max 16 --draft-min 1 \
    -p "Write a Python mergesort."
```

## Observed performance (RTX 3090, Qwen3.5-27B-heretic target Q4_K_M)

| Workload                  | Draft ON (t/s) | Draft OFF (t/s) | Acceptance |
|---------------------------|---------------:|----------------:|-----------:|
| Code (raw prompt)         | 140            | 40              | 69 %       |
| Code (chat template)      | 99             | 40              | 37 %       |
| Prose (raw prompt)        | 60             | 40              | 77 %       |

**Heads-up on chat templates.** The drafter was trained on raw continuations; wrapping the prompt in Qwen chat-template tokens (`<|im_start|>` etc.) shifts the hidden-state distribution the drafter's cross-attention expects. Acceptance on code drops from **69 % → 37 %** when the template is applied, and end-to-end throughput drops correspondingly. Raw-mode generation is still a strong speedup; chat-mode is a more modest one. A drafter retrained on chat-formatted data would be needed to recover the full speedup under a chat template.

## Quantization details

- Source: `z-lab/Qwen3.5-27B-DFlash` (BF16 safetensors, 2 B parameters)
- Converter: `convert_hf_to_gguf.py` from [`spiritbuun/buun-llama-cpp`](https://github.com/spiritbuun/buun-llama-cpp) (adds `DFlashDraftModel` arch)
- Quant: `llama-quantize` → `Q4_K_M`
- File size: ~1.0 GB
- Tensors: drafter transformer + projection heads + cross-attention layers targeting Qwen3.5-27B layer ids `[1, 16, 31, 46, 61]`

## Reproducing the conversion

> **Heads-up (tokenizer error):** the upstream `z-lab/Qwen3.5-27B-DFlash` repo ships only `config.json`, `model.safetensors`, `dflash.py`, and a README — **no tokenizer files**. The drafter shares the target model's tokenizer. If you run `convert_hf_to_gguf.py` directly against the z-lab repo you'll get a vocab / tokenizer error. Copy the Qwen3.5 tokenizer files into the drafter directory first.

```bash
# 1. Pull the DFlash drafter weights
hf download z-lab/Qwen3.5-27B-DFlash --local-dir ./dflash-drafter

# 2. Pull tokenizer files from the target model into the same directory
hf download Qwen/Qwen3.5-27B \
    tokenizer.json tokenizer_config.json vocab.json merges.txt \
    special_tokens_map.json \
    --local-dir ./dflash-drafter

# 3. Convert to GGUF (BF16/F16 first, then quantize)
python convert_hf_to_gguf.py ./dflash-drafter \
    --outtype f16 \
    --outfile dflash-draft-f16.gguf

# 4. Quantize to Q4_K_M
./build/bin/llama-quantize dflash-draft-f16.gguf dflash-draft-q4_k_m.gguf Q4_K_M
```

Required files in `./dflash-drafter/` before step 3:

| File | Source |
|---|---|
| `config.json` | `z-lab/Qwen3.5-27B-DFlash` (has `architectures: ["DFlashDraftModel"]`) |
| `model.safetensors` | `z-lab/Qwen3.5-27B-DFlash` |
| `tokenizer.json`, `tokenizer_config.json`, `vocab.json`, `merges.txt`, `special_tokens_map.json` | `Qwen/Qwen3.5-27B` |

No special converter flag is needed — `convert_hf_to_gguf.py` auto-detects the `DFlashDraftModel` architecture from `config.json` and registers the correct tensor mappings + GGUF hparams (`dflash.block_size`, `dflash.mask_token_id`, `dflash.target_layer_ids`, `dflash.n_target_features`).

---

# Original model card — `z-lab/Qwen3.5-27B-DFlash`

*Reproduced from the upstream model page. License: MIT.*

## Overview
**Qwen3.5-27B-DFlash** is a lightweight **drafter** component for DFlash speculative decoding. It must be used with the target model `Qwen/Qwen3.5-27B`.

- **Paper**: https://arxiv.org/abs/2602.06036
- **GitHub**: https://github.com/z-lab/dflash
- **Blog**: https://z-lab.ai/projects/dflash/
- **Model Size**: 2B parameters (BF16)
- **Context Length**: 4096 tokens

## What is DFlash?
DFlash is a novel speculative decoding method using a lightweight block diffusion model for drafting, enabling efficient, high-quality parallel drafting that significantly speeds up inference.

## Upstream Quick Start (vLLM / SGLang)

### vLLM
```bash
uv pip install -U vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly

vllm serve Qwen/Qwen3.5-27B \
  --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.5-27B-DFlash", "num_speculative_tokens": 15}' \
  --attention-backend flash_attn \
  --max-num-batched-tokens 32768
```

### SGLang
```bash
uv pip install "git+https://github.com/sgl-project/sglang.git@refs/pull/20547/head#subdirectory=python"

python -m sglang.launch_server \
    --model-path Qwen/Qwen3.5-27B \
    --speculative-algorithm DFLASH \
    --speculative-draft-model-path z-lab/Qwen3.5-27B-DFlash \
    --speculative-num-draft-tokens 16 \
    --tp-size 1 \
    --attention-backend fa3 \
    --mem-fraction-static 0.75 \
    --trust-remote-code
```

## Upstream Benchmarks (NVIDIA B200, BF16)

**Up to 5.2× speedup** over autoregressive baseline at concurrency 1 on HumanEval with block size 16.

| Task      | Concurrency | AR  | DFlash (B16) | Speedup |
|-----------|-------------|-----|--------------|---------|
| Math500   | 1           | 84  | **397**      | **4.7×** |
| HumanEval | 1           | 83  | **427**      | **5.2×** |
| GSM8K     | 1           | 83  | **330**      | **4.0×** |

## Citation
```bibtex
@article{chen2026dflash,
  title   = {{DFlash: Block Diffusion for Flash Speculative Decoding}},
  author  = {Chen, Jian and Liang, Yesheng and Liu, Zhijian},
  journal = {arXiv preprint arXiv:2602.06036},
  year    = {2026}
}
```

## License
**MIT** — inherited from the upstream model. This repository redistributes a quantized derivative under the same terms.