---
license: cc-by-4.0
language:
- vi
library_name: nemo
pipeline_tag: automatic-speech-recognition
base_model: nvidia/stt_en_fastconformer_transducer_large
tags:
- automatic-speech-recognition
- speech
- audio
- fastconformer
- transducer
- vietnamese
- telephony
- callbot
datasets:
- vivos
- mozilla-foundation/common_voice_17_0
- google/fleurs
- linhtran92/viet_bud500
metrics:
- wer
---

# Vietnamese ASR — FastConformer-Transducer 114M

A Vietnamese speech-to-text model built for **conversational / call-center (telephony) audio**.
It is adapted from NVIDIA's English `stt_en_fastconformer_transducer_large` to Vietnamese via
vocabulary adaptation and a 3-stage curriculum over ~14 public Vietnamese datasets, then
hardened for **robust digit-string reading on 8 kHz telephony**.

- **Architecture:** FastConformer-Transducer (RNNT), ~114M parameters
- **Language:** Vietnamese (vi)
- **Base model:** [`nvidia/stt_en_fastconformer_transducer_large`](https://huggingface.co/nvidia/stt_en_fastconformer_transducer_large) (CC-BY-4.0)
- **Framework:** NVIDIA NeMo

---

## Highlights

Two things this model was built to demonstrate, both aimed at **cold-start, noisy, narrow-domain** settings where labeled data is scarce:

### 1. Vocabulary extension without retraining from scratch

The Vietnamese vocabulary/tokenizer was **extended in place** (new tokens grafted onto the existing
model) rather than resetting the decoder and retraining from zero. Only a small number of
vocabulary-dependent tensors need updating, so new symbols (e.g. `f / j / w / z` for loanwords) can be
added cheaply. This makes **continued, incremental training toward mixed-language content** practical
instead of a full re-train each time.

### 2. An STT ↔ TTS mutual-bootstrapping loop

Training data for hard, narrow domains was generated with a **loop between STT and TTS**:

- a human seeds **topics** and a few **sample clips** to clone voices from,
- a voice-cloning TTS synthesizes many in-domain utterances (e.g. connected-digit read-backs),
- the audio is degraded to 8 kHz telephony conditions,
- an STT self-check gate keeps only clips read back correctly,
- the resulting set fine-tunes the STT — which in turn makes a better gate.

This produces controlled, labeled, in-domain data **without needing real production recordings**,
which is exactly the constraint in cold-start and privacy-restricted settings.

---

## Results

### General Vietnamese WER (9 public test sets, `normalize_vi`)

| Test set | Characteristic | WER |
| --- | --- | --- |
| bud500 | 3-region conversational | 6.09% |
| vivos | studio read speech | 7.87% |
| lsvsc | natural, multi-region | 13.03% |
| fleurs | international read speech | 15.92% |
| common_voice | everyday-mic read speech | 16.54% |
| fosd | read speech (FOSD) | 18.53% |
| vietsuperspeech | conversational (closest to callbot) | 21.81% |
| vlsp | formal news | 25.28% |
| vietmed | medical (out-of-domain probe) | 25.99% |
| **average** | | **16.78%** |

### Digit robustness on 8 kHz telephony

The digit-hardening step targets long spoken number strings (phone, ID, card numbers) read back
over 8 kHz telephony — the highest-friction failure in call-center audio.

- **Digit recall** on a controlled 8 kHz digit eval: **0.52 → 0.92**
- **Collapse** (model emitting an unknown token instead of digits): **eliminated**
- Non-digit performance is preserved (no measurable forgetting).

> Numbers on real call recordings are directional only (small labeled sample); controlled synthetic
> eval is used for reliable measurement.

---

## Intended use

- Vietnamese **conversational / call-center** transcription, especially where callers read out
  **numbers and identifiers** over a phone line.
- Offline transcription of 8 kHz / 16 kHz Vietnamese speech.

## Limitations

- Optimized for telephony/conversational Vietnamese; very formal read speech (news) is weaker.
- Domains unseen in training (e.g. medical terminology) degrade.
- Real-world call numbers were measured on a small labeled set; treat production metrics as indicative.

---

## How to use

```python
import nemo.collections.asr as nemo_asr

model = nemo_asr.models.ASRModel.from_pretrained("kyle/vi-asr-fastconformer-114m")
# 16 kHz mono wav; the model also handles 8 kHz telephony audio
hyp = model.transcribe(["sample_vi.wav"])
print(hyp)
```

## Training data

All training data is **public**. Vietnamese sets used across the curriculum include:
VIVOS, Common Voice (vi), FLEURS (vi), VLSP2020, LSVSC, FOSD (FPT Open Speech Dataset),
Bud500, VietSuperSpeech, and VietMed (held-out probe). Digit-robustness data was
synthesized with a voice-cloning TTS as described above.

## License & attribution

- Released under **CC-BY-4.0**, inheriting the license of the base model.
- Derived from **NVIDIA `stt_en_fastconformer_transducer_large`** (CC-BY-4.0); please retain NVIDIA
  attribution.
- Built with **NVIDIA NeMo**.

## Acknowledgements

Thanks to the open Vietnamese ASR dataset community and to NVIDIA for the FastConformer base model
and the NeMo framework.