File size: 4,690 Bytes
228884a 32688d0 228884a 32688d0 228884a 32688d0 228884a 32688d0 228884a 32688d0 228884a 32688d0 228884a 32688d0 228884a 32688d0 228884a 32688d0 228884a 32688d0 228884a 32688d0 228884a 32688d0 69690e2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 | ---
license: apache-2.0
base_model: Qwen/Qwen3-ASR-0.6B
tags:
- asr
- speech
- quantization
- knowledge-distillation
- int4
- awq
- multilingual
language:
- en
- vi
- zh
- ar
- de
- fr
- es
- ja
- ko
- pt
- ru
- nl
- it
- pl
- tr
- sv
- cs
- fi
- hu
- ro
- uk
- he
- id
- ms
- th
- hi
- bn
- ta
- te
- mr
pipeline_tag: automatic-speech-recognition
---
# Qwen3-ASR-0.6B — INT4 AWQ + Quantization-Aware Distillation
This is a compressed and distillation-refined version of [Qwen/Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B).
The model applies **INT4 AWQ (Activation-aware Weight Quantization)** post-training quantization followed by **Quantization-Aware Distillation (QAD)** to recover accuracy lost during aggressive 4-bit compression — while preserving low-latency, low-memory inference.
---
## Method Overview
### Stage 1 — INT4 AWQ (PTQ)
The base FP16 model is quantized to INT4 using the **AWQ (`awq_full`)** algorithm via [NVIDIA ModelOpt](https://github.com/NVIDIA/TensorRT-Model-Optimizer). To preserve acoustic feature extraction and final token prediction integrity, the `audio_tower` and `lm_head` are explicitly excluded from quantization and kept in FP16. Calibration was performed dynamically across English, Chinese, and 28 other languages to ensure a balanced quantization scale mapping.
### Stage 2 — Quantization-Aware Distillation (QAD)
To close the accuracy gap introduced by heavy 4-bit quantization, we apply a **knowledge distillation** fine-tuning stage where:
- **Teacher**: [Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B) FP16 model (frozen)
- **Student**: the INT4-quantized 0.6B model (trainable QKV/MLP quantized weights — audio encoder and LM head are frozen)
- **Data**: unlabeled speech data spanning **30 languages**, with pseudo-labels generated by the 1.7B teacher model
- **Loss**: a combination of KL-divergence distillation loss (`alpha_kd = 0.5`) and cross-entropy loss
- **Optimizer**: AdamW with cosine decay learning rate schedule
---
## Benchmark Results (Trilingual Evaluation)
| Model | [VIVOS](https://ailab.hcmus.edu.vn/vivos) (Viet) WER ↓ | [LibriSpeech](https://www.openslr.org/12) (Eng) WER ↓ | [Chinese Fleurs](https://huggingface.co/datasets/google/fleurs) CER ↓ |
|---|---|---|---|
| Teacher 1.7B (FP16 base) | 7.24% | 2.32% | 7.12% |
| INT4 AWQ (pre-QAD) | 14.34% | 3.47% | 8.16% |
| **INT4 + QAD (this model)** | **12.81%** | **3.41%** | **8.10%** |
> QAD successfully recovers **~1.53% absolute WER** in Vietnamese and stabilizes English/Chinese performance against aggressive 4-bit compression degradation, with no additional labeled data required.
---
## Usage
Install dependencies:
```bash
pip install qwen-asr nvidia-modelopt soundfile numpy
```
Run inference:
```python
import soundfile as sf
import numpy as np
import torch
import modelopt.torch.opt as mto
from qwen_asr import Qwen3ASRModel
# Enable ModelOpt quantization state restore
mto.enable_huggingface_checkpointing()
model = Qwen3ASRModel.from_pretrained(
"vrfai/Qwen3-ASR-0.6B-int4-QAD",
dtype=torch.float16,
device_map="cuda:0",
max_new_tokens=256,
)
audio, sr = sf.read("your_audio.wav")
if audio.ndim > 1:
audio = audio.mean(axis=1)
audio = audio.astype(np.float32)
results = model.transcribe(audio=(audio, sr), language=None)
print(results[0].text)
```
---
## Model Details
| Property | Value |
|---|---|
| Base model | [Qwen/Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B) |
| Parameters | ~0.6B |
| Quantization | INT4 AWQ (`awq_full` via [NVIDIA ModelOpt](https://github.com/NVIDIA/TensorRT-Model-Optimizer)) |
| Audio encoder | Frozen (FP16, not quantized) |
| LM head | Frozen (FP16, not quantized) |
| Quantized scope | Transformer decoder layers |
| Distillation data | Multilingual unlabeled speech (30 languages) |
| License | Apache 2.0 |
---
## Citation
If you use this model, please also cite the original Qwen3-ASR work:
```bibtex
@misc{qwen3asr2025,
title = {Qwen3-ASR},
author = {Qwen Team},
year = {2025},
url = {[https://huggingface.co/Qwen/Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B)}
}
```
---
## Acknowledgements
- [Qwen Team](https://huggingface.co/Qwen) for the base ASR model
- [NVIDIA ModelOpt](https://github.com/NVIDIA/TensorRT-Model-Optimizer) for quantization tooling
## Quantization Script
The recipes and scripts used to quantize this model can be found in the following repository:
- [VinRobotics/model-quantization-recipes](https://github.com/VinRobotics/model-quantization-recipes/tree/main/recipes/qwen3-asr/llm-qad) |