README.md

---
license: apache-2.0
base_model: Qwen/Qwen3-ASR-0.6B
tags:
  - asr
  - speech
  - quantization
  - knowledge-distillation
  - int4
  - awq
  - multilingual
language:
  - en
  - vi
  - zh
  - ar
  - de
  - fr
  - es
  - ja
  - ko
  - pt
  - ru
  - nl
  - it
  - pl
  - tr
  - sv
  - cs
  - fi
  - hu
  - ro
  - uk
  - he
  - id
  - ms
  - th
  - hi
  - bn
  - ta
  - te
  - mr
pipeline_tag: automatic-speech-recognition
---

# Qwen3-ASR-0.6B — INT4 AWQ + Quantization-Aware Distillation

This is a compressed and distillation-refined version of [Qwen/Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B).  
The model applies **INT4 AWQ (Activation-aware Weight Quantization)** post-training quantization followed by **Quantization-Aware Distillation (QAD)** to recover accuracy lost during aggressive 4-bit compression — while preserving low-latency, low-memory inference.

---

## Method Overview

### Stage 1 — INT4 AWQ (PTQ)
The base FP16 model is quantized to INT4 using the **AWQ (`awq_full`)** algorithm via [NVIDIA ModelOpt](https://github.com/NVIDIA/TensorRT-Model-Optimizer). To preserve acoustic feature extraction and final token prediction integrity, the `audio_tower` and `lm_head` are explicitly excluded from quantization and kept in FP16. Calibration was performed dynamically across English, Chinese, and 28 other languages to ensure a balanced quantization scale mapping.

### Stage 2 — Quantization-Aware Distillation (QAD)
To close the accuracy gap introduced by heavy 4-bit quantization, we apply a **knowledge distillation** fine-tuning stage where:

- **Teacher**: [Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B) FP16 model (frozen)
- **Student**: the INT4-quantized 0.6B model (trainable QKV/MLP quantized weights — audio encoder and LM head are frozen)
- **Data**: unlabeled speech data spanning **30 languages**, with pseudo-labels generated by the 1.7B teacher model
- **Loss**: a combination of KL-divergence distillation loss (`alpha_kd = 0.5`) and cross-entropy loss
- **Optimizer**: AdamW with cosine decay learning rate schedule

---

## Benchmark Results (Trilingual Evaluation)

| Model | [VIVOS](https://ailab.hcmus.edu.vn/vivos) (Viet) WER ↓ | [LibriSpeech](https://www.openslr.org/12) (Eng) WER ↓ | [Chinese Fleurs](https://huggingface.co/datasets/google/fleurs) CER ↓ |
|---|---|---|---|
| Teacher 1.7B (FP16 base) | 7.24% | 2.32% | 7.12% |
| INT4 AWQ (pre-QAD) | 14.34% | 3.47% | 8.16% |
| **INT4 + QAD (this model)** | **12.81%** | **3.41%** | **8.10%** |

> QAD successfully recovers **~1.53% absolute WER** in Vietnamese and stabilizes English/Chinese performance against aggressive 4-bit compression degradation, with no additional labeled data required.

---

## Usage

Install dependencies:

```bash
pip install qwen-asr nvidia-modelopt soundfile numpy
```

Run inference:

```python
import soundfile as sf
import numpy as np
import torch
import modelopt.torch.opt as mto
from qwen_asr import Qwen3ASRModel

# Enable ModelOpt quantization state restore
mto.enable_huggingface_checkpointing()

model = Qwen3ASRModel.from_pretrained(
    "vrfai/Qwen3-ASR-0.6B-int4-QAD",
    dtype=torch.float16,
    device_map="cuda:0",
    max_new_tokens=256,
)

audio, sr = sf.read("your_audio.wav")
if audio.ndim > 1:
    audio = audio.mean(axis=1)
audio = audio.astype(np.float32)

results = model.transcribe(audio=(audio, sr), language=None)
print(results[0].text)
```

---

## Model Details

| Property | Value |
|---|---|
| Base model | [Qwen/Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B) |
| Parameters | ~0.6B |
| Quantization | INT4 AWQ (`awq_full` via [NVIDIA ModelOpt](https://github.com/NVIDIA/TensorRT-Model-Optimizer)) |
| Audio encoder | Frozen (FP16, not quantized) |
| LM head | Frozen (FP16, not quantized) |
| Quantized scope | Transformer decoder layers |
| Distillation data | Multilingual unlabeled speech (30 languages) |
| License | Apache 2.0 |

---

## Citation

If you use this model, please also cite the original Qwen3-ASR work:

```bibtex
@misc{qwen3asr2025,
  title  = {Qwen3-ASR},
  author = {Qwen Team},
  year   = {2025},
  url    = {[https://huggingface.co/Qwen/Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B)}
}
```

---

## Acknowledgements

- [Qwen Team](https://huggingface.co/Qwen) for the base ASR model
- [NVIDIA ModelOpt](https://github.com/NVIDIA/TensorRT-Model-Optimizer) for quantization tooling

## Quantization Script

The recipes and scripts used to quantize this model can be found in the following repository:
- [VinRobotics/model-quantization-recipes](https://github.com/VinRobotics/model-quantization-recipes/tree/main/recipes/qwen3-asr/llm-qad)