Qwen3.5-4B-QAD-W4A16

Introduction

We reduce Qwen3.5-4B inference latency on a single A10G by quantizing both the target model and its speculative-decoding drafter to INT4, while maintaining quality across all three gates (MMLU-Pro, IFEval, GPQA-Diamond). This repository reproduces our 3rd-place Efficient Qwen Competition (AdaptFM @ ICML 2026) entry end to end. It combines three techniques:

• Quantization-Aware Distillation (QAD): trains the INT4 target model to follow the original BF16 model's distribution, recovering the accuracy that post-training quantization (PTQ) loses.
• Two-Stage Training for Block-Diffusion Drafter: pretrains the DFlash drafter with the BF16 target, then finetunes it on the QAD-applied INT4 target so it aligns with the quantized target distribution better than direct QAD-target training does.
• Drafter Optimization: applies INT4 GPTQ and Sliding-Window Attention (SWA) to reduce drafter overhead while maintaining acceptance length.

Hardware: Final serving and evaluation run on a single NVIDIA A10G 24 GB GPU. Retraining the QAD target and drafter requires an 80 GB-class multi-GPU machine, with at least 2 GPUs and preferably 8 GPUs. The gated prompt dataset is a one-time ~42 GB download.

Quickstart (vLLM)

vllm serve nota-ai/Qwen3.5-4B-QAD-W4A16 \
    --quantization compressed-tensors \
    --trust-remote-code

• For speculative decoding with the companion W4A16 DFlash drafter, see the drafter card.

Citation

@article{kim2026quantize,
  title   = {Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B},
  author  = {Jaeyeon Kim and Jewon Lee and Bo-Kyeong Kim},
  journal = {arXiv preprint arXiv:2607.04244},
  year    = {2026}
}

License & provenance

Apache-2.0, derived from Qwen/Qwen3.5-4B (Apache-2.0) with INT4 initialization from cyankiwi/Qwen3.5-4B-AWQ-4bit. Distillation data derives from NVIDIA's Nemotron-Post-Training-Dataset-v2 (gated; see its terms).