---
license: apache-2.0
base_model: Qwen/Qwen3-ASR-0.6B
tags:
  - asr
  - speech
  - quantization
  - knowledge-distillation
  - int8
  - multilingual
language:
  - en
  - vi
  - zh
  - ar
  - de
  - fr
  - es
  - ja
  - ko
  - pt
  - ru
  - nl
  - it
  - pl
  - tr
  - sv
  - cs
  - fi
  - hu
  - ro
  - uk
  - he
  - id
  - ms
  - th
  - hi
  - bn
  - ta
  - te
  - mr
pipeline_tag: automatic-speech-recognition
---

# Qwen3-ASR-0.6B — INT8 + Quantization-Aware Distillation

This is a compressed and distillation-refined version of [Qwen/Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B).  
The model applies **INT8 SmoothQuant** post-training quantization followed by **Quantization-Aware Distillation (QAD)** to recover accuracy lost during compression — while preserving low-latency, low-memory inference.

---

## Method Overview

### Stage 1 — INT8 SmoothQuant (PTQ)
The base FP16 model is quantized to INT8 using **SmoothQuant**, a channel-wise activation smoothing technique that migrates quantization difficulty from activations to weights. This reduces model memory footprint and accelerates inference on hardware with INT8 tensor cores.

### Stage 2 — Quantization-Aware Distillation (QAD)
To close the accuracy gap introduced by quantization, we apply a **knowledge distillation** fine-tuning stage where:

- **Teacher**: original FP16 base model (frozen)
- **Student**: the INT8-quantized model (trainable weights only — audio encoder and LM head are frozen)
- **Data**: unlabeled speech data spanning **30 languages**, with pseudo-labels generated by the teacher model
- **Loss**: a combination of KL-divergence distillation loss and cross-entropy loss, computed exclusively on response tokens (post audio-end token positions)
- **Optimizer**: AdamW with cosine decay learning rate schedule and linear warmup

The QAD stage teaches the quantized student to match the teacher's output distribution on diverse real-world speech, without requiring any manual transcription labels.

---

## Benchmark Results

### English — LibriSpeech dev-clean-2

| Model | WER ↓ |
|---|---|
| Qwen3-ASR-0.6B (FP16 base) | 2.66% |
| INT8 SmoothQuant (pre-QAD) | 2.81% |
| **INT8 + QAD (this model)** | **2.76%** |

### Vietnamese — VIVOS test set

| Model | WER ↓ |
|---|---|
| Qwen3-ASR-0.6B (FP16 base) | 10.53% |
| INT8 SmoothQuant (pre-QAD) | 11.75% |
| **INT8 + QAD (this model)** | **11.55%** |

> QAD recovers a meaningful portion of the WER degradation introduced by INT8 quantization, with no additional labeled data required.

---

## Usage

Install dependencies:

```bash
pip install qwen-asr nvidia-modelopt soundfile numpy
```

Run inference:

```python
import soundfile as sf
import numpy as np
import torch
import modelopt.torch.opt as mto
from qwen_asr import Qwen3ASRModel

# Enable ModelOpt quantization state restore
mto.enable_huggingface_checkpointing()

model = Qwen3ASRModel.from_pretrained(
    "vrfai/Qwen3-ASR-0.6B-int8-QAD",
    dtype=torch.float16,
    device_map="cuda:0",
    max_new_tokens=256,
)

audio, sr = sf.read("your_audio.wav")
if audio.ndim > 1:
    audio = audio.mean(axis=1)
audio = audio.astype(np.float32)

results = model.transcribe(audio=(audio, sr), language=None)
print(results[0].text)
```

---

## Model Details

| Property | Value |
|---|---|
| Base model | Qwen/Qwen3-ASR-0.6B |
| Parameters | ~0.6B |
| Quantization | INT8 SmoothQuant (via NVIDIA ModelOpt) |
| Audio encoder | Frozen (FP16, not quantized) |
| LM head | Frozen |
| Quantized scope | Transformer decoder layers |
| Distillation data | Multilingual unlabeled speech (30 languages) |
| License | Apache 2.0 |

---

## Citation

If you use this model, please also cite the original Qwen3-ASR work:

```bibtex
@misc{qwen3asr2025,
  title  = {Qwen3-ASR},
  author = {Qwen Team},
  year   = {2025},
  url    = {https://huggingface.co/Qwen/Qwen3-ASR-0.6B}
}
```

---

## Acknowledgements

- [Qwen Team](https://huggingface.co/Qwen) for the base ASR model
- [NVIDIA ModelOpt](https://github.com/NVIDIA/TensorRT-Model-Optimizer) for quantization tooling

## Quantization Script

The recipes and scripts used to quantize this model can be found in the following repository:
- [VinRobotics/model-quantization-recipes](https://github.com/VinRobotics/model-quantization-recipes/tree/main/recipes/qwen3-asr/llm-qad)