---
license: apache-2.0
base_model: z-lab/Qwen3.8-27B-DFlash2
tags:
  - llama.cpp
  - gguf
  - speculative-decoding
  - dflash2
  - qwen
  - qwen3.8
  - 2-bit
pipeline_tag: text-generation
---

# Qwen3.8-27B-DFlash2-Q2_K (2-Bit Zero-Loss Speculative Drafter)

This repository provides the **`Q2_K` (2-bit)** quantization of [z-lab/Qwen3.8-27B-DFlash2](https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2) (~700 MB on disk). 

It is engineered specifically for **24GB consumer GPUs (NVIDIA GeForce RTX 4090 / RTX 3090)** to run speculative decoding alongside `Qwen3.8-27B` dense models without spilling into system RAM.

---

## 🚀 Benchmark Discovery: Zero Quality Loss at 2-Bit

By quantizing the DFlash 2 block-diffusion drafter down from the official `Q4_K_M` (~1.1 GB) to `Q2_K` (~700 MB), we save **~400–450 MB of VRAM** with **zero degradation in speculative draft accuracy**.

### Strict A/B Performance on Single RTX 4090 (24GB VRAM):
*Evaluated on `Qwen3.8-27B-UD-Q4_K_XL` using `llama-server` (PR #27342) with `--spec-draft-n-max 3` across a 28k prompt baseline.*

| Drafter Quant | File Size | Mean Accepted Length | Acceptance Rate | Decode Speed | Max Usable Context (24GB VRAM) |
|---|---|---|---|---|---|
| **Official Q4_K_M** | ~1.10 GB | 2.80 / 3.0 tokens | 60.00% | 75.89 t/s | **150,000 tokens** (Q4 KV) |
| **Custom Q2_K (This Repo)** | **~700 MB** | **2.81 / 3.0 tokens** | **60.22%** | **75.93 t/s** | **170,000 tokens (+20k)** |

### Key Takeaways:
1. **Identical Acceptance Rate:** The target 27B model accepts **2.81 out of 3.0 drafted tokens on average** $(93.7\% \text{ theoretical efficiency})$, matching the 4-bit drafter identically.
2. **Context Expansion:** The ~400 MB saved VRAM directly unlocks:
   * **170,000 Context** on `Q4_0 KV` (~76 t/s @ 23.89 GB)
   * **100,000 Context** on high-precision `Q8_0 KV` (~75 t/s @ 23.88 GB)

---

## ⚙️ Why Q2_K is the True Physical Floor for DFlash 2
DFlash 2 is a cross-layer block-diffusion drafter that extracts hidden states from layers `[6, 20, 34, 48, 62]` of the base model. Because it is not a standalone causal LM, standalone `llama-imatrix` calibration fails (`dflash requires ctx_other to be set`). 

This means importance-matrix quants (`IQ1_S`, `IQ2_XXS`) cannot be computed, making **`Q2_K` the absolute lowest non-degrading quantization floor** for DFlash 2.

---

## 🛠️ Quickstart with `llama.cpp`

Requires `llama.cpp` built from [PR #27342](https://github.com/ggml-org/llama.cpp/pull/27342).

### 1. Build `llama.cpp`:
```bash
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git fetch origin pull/27342/head:pr-27342
git switch pr-27342

# Build with CUDA
cmake -B build -DGGML_CUDA=ON && cmake --build build -j