Update README.md
Browse files
README.md
CHANGED
|
@@ -46,35 +46,33 @@ pipeline_tag: automatic-speech-recognition
|
|
| 46 |
# Qwen3-ASR-0.6B β INT4 AWQ + Quantization-Aware Distillation
|
| 47 |
|
| 48 |
This is a compressed and distillation-refined version of [Qwen/Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B).
|
| 49 |
-
The model applies **INT4 AWQ Activation-aware Weight Quantization** post-training quantization followed by **Quantization-Aware Distillation QAD** to recover accuracy lost during aggressive 4-bit compression β while preserving low-latency, low-memory inference.
|
| 50 |
|
| 51 |
---
|
| 52 |
|
| 53 |
## Method Overview
|
| 54 |
|
| 55 |
-
### Stage 1 β INT4 AWQ PTQ
|
| 56 |
-
The base FP16 model is quantized to INT4 using the **AWQ awq_full** algorithm via NVIDIA ModelOpt. To preserve acoustic feature extraction and final token prediction integrity, the audio_tower and lm_head are explicitly excluded from quantization and kept in FP16. Calibration was performed dynamically across English, Chinese, and 28 other languages to ensure a balanced quantization scale mapping.
|
| 57 |
|
| 58 |
-
### Stage 2 β Quantization-Aware Distillation QAD
|
| 59 |
To close the accuracy gap introduced by heavy 4-bit quantization, we apply a **knowledge distillation** fine-tuning stage where:
|
| 60 |
|
| 61 |
-
- **Teacher**: Qwen3-ASR-1.7B FP16 model frozen
|
| 62 |
-
- **Student**: the INT4-quantized 0.6B model trainable QKV/MLP quantized weights β audio encoder and LM head are frozen
|
| 63 |
- **Data**: unlabeled speech data spanning **30 languages**, with pseudo-labels generated by the 1.7B teacher model
|
| 64 |
-
- **Loss**: a combination of KL-divergence distillation loss alpha_kd = 0.5 and cross-entropy loss
|
| 65 |
- **Optimizer**: AdamW with cosine decay learning rate schedule
|
| 66 |
|
| 67 |
-
The QAD stage teaches the quantized student to match the teacher's output distribution on diverse real-world speech, without requiring any manual transcription labels.
|
| 68 |
-
|
| 69 |
---
|
| 70 |
|
| 71 |
-
## Benchmark Results Trilingual Evaluation
|
| 72 |
|
| 73 |
-
| Model | VIVOS
|
| 74 |
|---|---|---|---|
|
| 75 |
-
| Teacher 1.7B FP16 base | 7.24% | 2.32% | 7.12% |
|
| 76 |
-
| INT4 AWQ pre-QAD | 14.34% | 3.47% | 8.16% |
|
| 77 |
-
| **INT4 + QAD this model** | **12.81%** | **3.41%** | **8.10%** |
|
| 78 |
|
| 79 |
> QAD successfully recovers **~1.53% absolute WER** in Vietnamese and stabilizes English/Chinese performance against aggressive 4-bit compression degradation, with no additional labeled data required.
|
| 80 |
|
|
@@ -122,13 +120,13 @@ print(results[0].text)
|
|
| 122 |
|
| 123 |
| Property | Value |
|
| 124 |
|---|---|
|
| 125 |
-
| Base model | Qwen/Qwen3-ASR-0.6B |
|
| 126 |
| Parameters | ~0.6B |
|
| 127 |
-
| Quantization | INT4 AWQ awq_full via NVIDIA ModelOpt |
|
| 128 |
-
| Audio encoder | Frozen FP16, not quantized |
|
| 129 |
-
| LM head | Frozen FP16, not quantized |
|
| 130 |
| Quantized scope | Transformer decoder layers |
|
| 131 |
-
| Distillation data | Multilingual unlabeled speech 30 languages |
|
| 132 |
| License | Apache 2.0 |
|
| 133 |
|
| 134 |
---
|
|
@@ -150,5 +148,5 @@ If you use this model, please also cite the original Qwen3-ASR work:
|
|
| 150 |
|
| 151 |
## Acknowledgements
|
| 152 |
|
| 153 |
-
- Qwen Team for the base ASR model
|
| 154 |
-
- NVIDIA ModelOpt for quantization tooling
|
|
|
|
| 46 |
# Qwen3-ASR-0.6B β INT4 AWQ + Quantization-Aware Distillation
|
| 47 |
|
| 48 |
This is a compressed and distillation-refined version of [Qwen/Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B).
|
| 49 |
+
The model applies **INT4 AWQ (Activation-aware Weight Quantization)** post-training quantization followed by **Quantization-Aware Distillation (QAD)** to recover accuracy lost during aggressive 4-bit compression β while preserving low-latency, low-memory inference.
|
| 50 |
|
| 51 |
---
|
| 52 |
|
| 53 |
## Method Overview
|
| 54 |
|
| 55 |
+
### Stage 1 β INT4 AWQ (PTQ)
|
| 56 |
+
The base FP16 model is quantized to INT4 using the **AWQ (`awq_full`)** algorithm via [NVIDIA ModelOpt](https://github.com/NVIDIA/TensorRT-Model-Optimizer). To preserve acoustic feature extraction and final token prediction integrity, the `audio_tower` and `lm_head` are explicitly excluded from quantization and kept in FP16. Calibration was performed dynamically across English, Chinese, and 28 other languages to ensure a balanced quantization scale mapping.
|
| 57 |
|
| 58 |
+
### Stage 2 β Quantization-Aware Distillation (QAD)
|
| 59 |
To close the accuracy gap introduced by heavy 4-bit quantization, we apply a **knowledge distillation** fine-tuning stage where:
|
| 60 |
|
| 61 |
+
- **Teacher**: [Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B) FP16 model (frozen)
|
| 62 |
+
- **Student**: the INT4-quantized 0.6B model (trainable QKV/MLP quantized weights β audio encoder and LM head are frozen)
|
| 63 |
- **Data**: unlabeled speech data spanning **30 languages**, with pseudo-labels generated by the 1.7B teacher model
|
| 64 |
+
- **Loss**: a combination of KL-divergence distillation loss (`alpha_kd = 0.5`) and cross-entropy loss
|
| 65 |
- **Optimizer**: AdamW with cosine decay learning rate schedule
|
| 66 |
|
|
|
|
|
|
|
| 67 |
---
|
| 68 |
|
| 69 |
+
## Benchmark Results (Trilingual Evaluation)
|
| 70 |
|
| 71 |
+
| Model | [VIVOS](https://ailab.hcmus.edu.vn/vivos) (Viet) WER β | [LibriSpeech](https://www.openslr.org/12) (Eng) WER β | [Chinese Fleurs](https://huggingface.co/datasets/google/fleurs) CER β |
|
| 72 |
|---|---|---|---|
|
| 73 |
+
| Teacher 1.7B (FP16 base) | 7.24% | 2.32% | 7.12% |
|
| 74 |
+
| INT4 AWQ (pre-QAD) | 14.34% | 3.47% | 8.16% |
|
| 75 |
+
| **INT4 + QAD (this model)** | **12.81%** | **3.41%** | **8.10%** |
|
| 76 |
|
| 77 |
> QAD successfully recovers **~1.53% absolute WER** in Vietnamese and stabilizes English/Chinese performance against aggressive 4-bit compression degradation, with no additional labeled data required.
|
| 78 |
|
|
|
|
| 120 |
|
| 121 |
| Property | Value |
|
| 122 |
|---|---|
|
| 123 |
+
| Base model | [Qwen/Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B) |
|
| 124 |
| Parameters | ~0.6B |
|
| 125 |
+
| Quantization | INT4 AWQ (`awq_full` via [NVIDIA ModelOpt](https://github.com/NVIDIA/TensorRT-Model-Optimizer)) |
|
| 126 |
+
| Audio encoder | Frozen (FP16, not quantized) |
|
| 127 |
+
| LM head | Frozen (FP16, not quantized) |
|
| 128 |
| Quantized scope | Transformer decoder layers |
|
| 129 |
+
| Distillation data | Multilingual unlabeled speech (30 languages) |
|
| 130 |
| License | Apache 2.0 |
|
| 131 |
|
| 132 |
---
|
|
|
|
| 148 |
|
| 149 |
## Acknowledgements
|
| 150 |
|
| 151 |
+
- [Qwen Team](https://huggingface.co/Qwen) for the base ASR model
|
| 152 |
+
- [NVIDIA ModelOpt](https://github.com/NVIDIA/TensorRT-Model-Optimizer) for quantization tooling
|