quangnd58 commited on
Commit
32688d0
Β·
verified Β·
1 Parent(s): 228884a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +19 -21
README.md CHANGED
@@ -46,35 +46,33 @@ pipeline_tag: automatic-speech-recognition
46
  # Qwen3-ASR-0.6B β€” INT4 AWQ + Quantization-Aware Distillation
47
 
48
  This is a compressed and distillation-refined version of [Qwen/Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B).
49
- The model applies **INT4 AWQ Activation-aware Weight Quantization** post-training quantization followed by **Quantization-Aware Distillation QAD** to recover accuracy lost during aggressive 4-bit compression β€” while preserving low-latency, low-memory inference.
50
 
51
  ---
52
 
53
  ## Method Overview
54
 
55
- ### Stage 1 β€” INT4 AWQ PTQ
56
- The base FP16 model is quantized to INT4 using the **AWQ awq_full** algorithm via NVIDIA ModelOpt. To preserve acoustic feature extraction and final token prediction integrity, the audio_tower and lm_head are explicitly excluded from quantization and kept in FP16. Calibration was performed dynamically across English, Chinese, and 28 other languages to ensure a balanced quantization scale mapping.
57
 
58
- ### Stage 2 β€” Quantization-Aware Distillation QAD
59
  To close the accuracy gap introduced by heavy 4-bit quantization, we apply a **knowledge distillation** fine-tuning stage where:
60
 
61
- - **Teacher**: Qwen3-ASR-1.7B FP16 model frozen
62
- - **Student**: the INT4-quantized 0.6B model trainable QKV/MLP quantized weights β€” audio encoder and LM head are frozen
63
  - **Data**: unlabeled speech data spanning **30 languages**, with pseudo-labels generated by the 1.7B teacher model
64
- - **Loss**: a combination of KL-divergence distillation loss alpha_kd = 0.5 and cross-entropy loss
65
  - **Optimizer**: AdamW with cosine decay learning rate schedule
66
 
67
- The QAD stage teaches the quantized student to match the teacher's output distribution on diverse real-world speech, without requiring any manual transcription labels.
68
-
69
  ---
70
 
71
- ## Benchmark Results Trilingual Evaluation
72
 
73
- | Model | VIVOS Vietnamese WER ↓ | LibriSpeech English WER ↓ | Chinese Fleurs CER ↓ |
74
  |---|---|---|---|
75
- | Teacher 1.7B FP16 base | 7.24% | 2.32% | 7.12% |
76
- | INT4 AWQ pre-QAD | 14.34% | 3.47% | 8.16% |
77
- | **INT4 + QAD this model** | **12.81%** | **3.41%** | **8.10%** |
78
 
79
  > QAD successfully recovers **~1.53% absolute WER** in Vietnamese and stabilizes English/Chinese performance against aggressive 4-bit compression degradation, with no additional labeled data required.
80
 
@@ -122,13 +120,13 @@ print(results[0].text)
122
 
123
  | Property | Value |
124
  |---|---|
125
- | Base model | Qwen/Qwen3-ASR-0.6B |
126
  | Parameters | ~0.6B |
127
- | Quantization | INT4 AWQ awq_full via NVIDIA ModelOpt |
128
- | Audio encoder | Frozen FP16, not quantized |
129
- | LM head | Frozen FP16, not quantized |
130
  | Quantized scope | Transformer decoder layers |
131
- | Distillation data | Multilingual unlabeled speech 30 languages |
132
  | License | Apache 2.0 |
133
 
134
  ---
@@ -150,5 +148,5 @@ If you use this model, please also cite the original Qwen3-ASR work:
150
 
151
  ## Acknowledgements
152
 
153
- - Qwen Team for the base ASR model
154
- - NVIDIA ModelOpt for quantization tooling
 
46
  # Qwen3-ASR-0.6B β€” INT4 AWQ + Quantization-Aware Distillation
47
 
48
  This is a compressed and distillation-refined version of [Qwen/Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B).
49
+ The model applies **INT4 AWQ (Activation-aware Weight Quantization)** post-training quantization followed by **Quantization-Aware Distillation (QAD)** to recover accuracy lost during aggressive 4-bit compression β€” while preserving low-latency, low-memory inference.
50
 
51
  ---
52
 
53
  ## Method Overview
54
 
55
+ ### Stage 1 β€” INT4 AWQ (PTQ)
56
+ The base FP16 model is quantized to INT4 using the **AWQ (`awq_full`)** algorithm via [NVIDIA ModelOpt](https://github.com/NVIDIA/TensorRT-Model-Optimizer). To preserve acoustic feature extraction and final token prediction integrity, the `audio_tower` and `lm_head` are explicitly excluded from quantization and kept in FP16. Calibration was performed dynamically across English, Chinese, and 28 other languages to ensure a balanced quantization scale mapping.
57
 
58
+ ### Stage 2 β€” Quantization-Aware Distillation (QAD)
59
  To close the accuracy gap introduced by heavy 4-bit quantization, we apply a **knowledge distillation** fine-tuning stage where:
60
 
61
+ - **Teacher**: [Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B) FP16 model (frozen)
62
+ - **Student**: the INT4-quantized 0.6B model (trainable QKV/MLP quantized weights β€” audio encoder and LM head are frozen)
63
  - **Data**: unlabeled speech data spanning **30 languages**, with pseudo-labels generated by the 1.7B teacher model
64
+ - **Loss**: a combination of KL-divergence distillation loss (`alpha_kd = 0.5`) and cross-entropy loss
65
  - **Optimizer**: AdamW with cosine decay learning rate schedule
66
 
 
 
67
  ---
68
 
69
+ ## Benchmark Results (Trilingual Evaluation)
70
 
71
+ | Model | [VIVOS](https://ailab.hcmus.edu.vn/vivos) (Viet) WER ↓ | [LibriSpeech](https://www.openslr.org/12) (Eng) WER ↓ | [Chinese Fleurs](https://huggingface.co/datasets/google/fleurs) CER ↓ |
72
  |---|---|---|---|
73
+ | Teacher 1.7B (FP16 base) | 7.24% | 2.32% | 7.12% |
74
+ | INT4 AWQ (pre-QAD) | 14.34% | 3.47% | 8.16% |
75
+ | **INT4 + QAD (this model)** | **12.81%** | **3.41%** | **8.10%** |
76
 
77
  > QAD successfully recovers **~1.53% absolute WER** in Vietnamese and stabilizes English/Chinese performance against aggressive 4-bit compression degradation, with no additional labeled data required.
78
 
 
120
 
121
  | Property | Value |
122
  |---|---|
123
+ | Base model | [Qwen/Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B) |
124
  | Parameters | ~0.6B |
125
+ | Quantization | INT4 AWQ (`awq_full` via [NVIDIA ModelOpt](https://github.com/NVIDIA/TensorRT-Model-Optimizer)) |
126
+ | Audio encoder | Frozen (FP16, not quantized) |
127
+ | LM head | Frozen (FP16, not quantized) |
128
  | Quantized scope | Transformer decoder layers |
129
+ | Distillation data | Multilingual unlabeled speech (30 languages) |
130
  | License | Apache 2.0 |
131
 
132
  ---
 
148
 
149
  ## Acknowledgements
150
 
151
+ - [Qwen Team](https://huggingface.co/Qwen) for the base ASR model
152
+ - [NVIDIA ModelOpt](https://github.com/NVIDIA/TensorRT-Model-Optimizer) for quantization tooling