File size: 4,344 Bytes
f967a49 1002c44 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 | ---
license: apache-2.0
base_model: Qwen/Qwen3-ASR-0.6B
tags:
- asr
- speech
- quantization
- knowledge-distillation
- int8
- multilingual
language:
- en
- vi
- zh
- ar
- de
- fr
- es
- ja
- ko
- pt
- ru
- nl
- it
- pl
- tr
- sv
- cs
- fi
- hu
- ro
- uk
- he
- id
- ms
- th
- hi
- bn
- ta
- te
- mr
pipeline_tag: automatic-speech-recognition
---
# Qwen3-ASR-0.6B — INT8 + Quantization-Aware Distillation
This is a compressed and distillation-refined version of [Qwen/Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B).
The model applies **INT8 SmoothQuant** post-training quantization followed by **Quantization-Aware Distillation (QAD)** to recover accuracy lost during compression — while preserving low-latency, low-memory inference.
---
## Method Overview
### Stage 1 — INT8 SmoothQuant (PTQ)
The base FP16 model is quantized to INT8 using **SmoothQuant**, a channel-wise activation smoothing technique that migrates quantization difficulty from activations to weights. This reduces model memory footprint and accelerates inference on hardware with INT8 tensor cores.
### Stage 2 — Quantization-Aware Distillation (QAD)
To close the accuracy gap introduced by quantization, we apply a **knowledge distillation** fine-tuning stage where:
- **Teacher**: original FP16 base model (frozen)
- **Student**: the INT8-quantized model (trainable weights only — audio encoder and LM head are frozen)
- **Data**: unlabeled speech data spanning **30 languages**, with pseudo-labels generated by the teacher model
- **Loss**: a combination of KL-divergence distillation loss and cross-entropy loss, computed exclusively on response tokens (post audio-end token positions)
- **Optimizer**: AdamW with cosine decay learning rate schedule and linear warmup
The QAD stage teaches the quantized student to match the teacher's output distribution on diverse real-world speech, without requiring any manual transcription labels.
---
## Benchmark Results
### English — LibriSpeech dev-clean-2
| Model | WER ↓ |
|---|---|
| Qwen3-ASR-0.6B (FP16 base) | 2.66% |
| INT8 SmoothQuant (pre-QAD) | 2.81% |
| **INT8 + QAD (this model)** | **2.76%** |
### Vietnamese — VIVOS test set
| Model | WER ↓ |
|---|---|
| Qwen3-ASR-0.6B (FP16 base) | 10.53% |
| INT8 SmoothQuant (pre-QAD) | 11.75% |
| **INT8 + QAD (this model)** | **11.55%** |
> QAD recovers a meaningful portion of the WER degradation introduced by INT8 quantization, with no additional labeled data required.
---
## Usage
Install dependencies:
```bash
pip install qwen-asr nvidia-modelopt soundfile numpy
```
Run inference:
```python
import soundfile as sf
import numpy as np
import torch
import modelopt.torch.opt as mto
from qwen_asr import Qwen3ASRModel
# Enable ModelOpt quantization state restore
mto.enable_huggingface_checkpointing()
model = Qwen3ASRModel.from_pretrained(
"vrfai/Qwen3-ASR-0.6B-int8-QAD",
dtype=torch.float16,
device_map="cuda:0",
max_new_tokens=256,
)
audio, sr = sf.read("your_audio.wav")
if audio.ndim > 1:
audio = audio.mean(axis=1)
audio = audio.astype(np.float32)
results = model.transcribe(audio=(audio, sr), language=None)
print(results[0].text)
```
---
## Model Details
| Property | Value |
|---|---|
| Base model | Qwen/Qwen3-ASR-0.6B |
| Parameters | ~0.6B |
| Quantization | INT8 SmoothQuant (via NVIDIA ModelOpt) |
| Audio encoder | Frozen (FP16, not quantized) |
| LM head | Frozen |
| Quantized scope | Transformer decoder layers |
| Distillation data | Multilingual unlabeled speech (30 languages) |
| License | Apache 2.0 |
---
## Citation
If you use this model, please also cite the original Qwen3-ASR work:
```bibtex
@misc{qwen3asr2025,
title = {Qwen3-ASR},
author = {Qwen Team},
year = {2025},
url = {https://huggingface.co/Qwen/Qwen3-ASR-0.6B}
}
```
---
## Acknowledgements
- [Qwen Team](https://huggingface.co/Qwen) for the base ASR model
- [NVIDIA ModelOpt](https://github.com/NVIDIA/TensorRT-Model-Optimizer) for quantization tooling
## Quantization Script
The recipes and scripts used to quantize this model can be found in the following repository:
- [VinRobotics/model-quantization-recipes](https://github.com/VinRobotics/model-quantization-recipes/tree/main/recipes/qwen3-asr/llm-qad) |