Qwen3.5-4B-DFlash-GPTQ-W4A16

Introduction

We reduce Qwen3.5-4B inference latency on a single A10G by quantizing both the target model and its speculative-decoding drafter to INT4, while maintaining quality across all three gates (MMLU-Pro, IFEval, GPQA-Diamond). This repository reproduces our 3rd-place Efficient Qwen Competition (AdaptFM @ ICML 2026) entry end to end. It combines three techniques:

• Quantization-Aware Distillation (QAD): trains the INT4 target model to follow the original BF16 model's distribution, recovering the accuracy that post-training quantization (PTQ) loses.
• Two-Stage Training for Block-Diffusion Drafter: pretrains the DFlash drafter with the BF16 target, then finetunes it on the QAD-applied INT4 target so it aligns with the quantized target distribution better than direct QAD-target training does.
• Drafter Optimization: applies INT4 GPTQ and Sliding-Window Attention (SWA) to reduce drafter overhead while maintaining acceptance length.

Hardware: Final serving and evaluation run on a single NVIDIA A10G 24 GB GPU. Retraining the QAD target and drafter requires an 80 GB-class multi-GPU machine, with at least 2 GPUs and preferably 8 GPUs. The gated prompt dataset is a one-time ~42 GB download.

Quickstart (vLLM)

pip install vllm==0.22.1
git clone https://github.com/nota-github/adaptfm-quant-dflash
pip install -e adaptfm-quant-dflash/src/vllm_plugin    # auto-loads via entry point

EQC_DFLASH_QUANT_PATCH=1 EQC_DFLASH_SWA_WINDOW=1024 \
vllm serve nota-ai/Qwen3.5-4B-QAD-W4A16 \
  --quantization compressed-tensors --trust-remote-code \
  --speculative-config '{"method": "dflash", "model": "nota-ai/Qwen3.5-4B-DFlash-GPTQ-W4A16",
                         "num_speculative_tokens": 15, "quantization": "compressed-tensors"}' \
  --attention-backend FLASH_ATTN --max-num-batched-tokens 16384

• Required: EQC_DFLASH_QUANT_PATCH=1 loads this W4A16 draft on vLLM 0.22.1.

• EQC_DFLASH_SWA_WINDOW=1024 enables sliding-window attention over the drafter, which improves long-context decoding latency. Drop it for full drafter attention.

Citation

@article{kim2026quantize,
  title   = {Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B},
  author  = {Jaeyeon Kim and Jewon Lee and Bo-Kyeong Kim},
  journal = {arXiv preprint arXiv:2607.04244},
  year    = {2026}
}

License & provenance

Apache-2.0. Trained with the MIT-licensed SpecForge framework on data derived from Nemotron-Post-Training-Dataset-v2 (gated; see its terms); drafts for Qwen3.5-4B (Apache-2.0).