---
license: apache-2.0
language:
  - zh
  - en
tags:
  - text-to-speech
  - tts
  - onnx
  - on-device
  - jetson
  - telephony
  - mandarin
  - taiwanese-mandarin
base_model: owensong/Inflect-Nano-v1
base_model_relation: finetune
library_name: onnxruntime
pipeline_tag: text-to-speech
---

# PrimeTTS — tiny bilingual zh‑TW + English TTS (8 kHz, CPU)

A **4.63M‑parameter** Mandarin (Taiwan) + English text‑to‑speech model that runs **entirely on CPU**
and emits **8 kHz** audio — sized for **G.711 telephony** and **on‑device (Jetson‑class)** use.
One model, **one young‑female voice**: Chinese, English, and code‑mix through a single frontend (no
language routing).

> 🔊 **Live demo:** https://huggingface.co/spaces/Luigi/PrimeTTS-vs-Inflect-Nano-v1 · 🧩 **Base:** [`owensong/Inflect-Nano-v1`](https://huggingface.co/owensong/Inflect-Nano-v1) (warm‑started fine‑tune, **same frozen architecture**)

| | |
|---|---|
| **Parameters** | 4.63M (3.47M acoustic + 1.17M vocoder) |
| **Sample rate** | 8 kHz (telephony‑band) |
| **Runtime** | `onnxruntime`, **CPU‑only**, torch‑free at inference |
| **Languages** | zh‑TW (Traditional) + English + code‑mix, single voice |
| **Voice** | young female, **Taiwan‑Mandarin accent** |
| **Architecture** | FastSpeech‑style (no attention) + Snake‑HiFiGAN — **frozen**, no NAS |
| **License** | Apache‑2.0 |

### Held‑out quality (eval_big, 36 unseen phone‑attendant sentences)

| metric | this model | prior 8k release |
|---|---|---|
| zh‑CER (Breeze‑ASR‑25) | **0.090** | 0.132 |
| code‑mix CER | **0.178** | 0.254 |
| en‑WER (Whisper) | **0.083** | 0.092 |
| Taiwan‑accent gap¹ | **+0.088** | +0.017 |
| SQUIM PESQ | **3.31** | 2.07 |
| SQUIM MOS | **4.24** | 3.30 |

¹ `CER(generic ASR) − CER(Taiwan‑tuned Breeze‑ASR‑25)` per zh clip; `>0` ⇒ a Taiwan‑tuned recognizer
understands it better ⇒ genuine Taiwan accent present.

---

## Quickstart (inference, CPU)

```bash
pip install onnxruntime numpy soundfile g2pw g2p_en cn2an
huggingface-cli download Luigi/PrimeTTS --local-dir PrimeTTS
```
```python
# from inside the PrimeTTS dir (uses the bundled frontend + scripts)
import sys; sys.path.insert(0, "scripts")
import json, numpy as np, onnxruntime as ort, soundfile as sf
import frontend_bopomofo as F
from synth_from_text import host_regulate          # numpy length‑regulator

meta = json.load(open("meta.json"))
enc = ort.InferenceSession("acoustic_encoder.onnx", providers=["CPUExecutionProvider"])
dec = ort.InferenceSession("acoustic_decoder.onnx", providers=["CPUExecutionProvider"])
voc = ort.InferenceSession("vocoder.onnx",          providers=["CPUExecutionProvider"])

o = F.text_to_ids("您好,歡迎使用 PrimeTTS。Thank you for calling.")   # text -> phone/tone/lang ids
ph, tn, lg = (np.array([o[k]], np.int64) for k in ("phone_ids", "tone_ids", "lang_ids"))
cond, dur, pitch = enc.run(None, {"phone": ph, "tone": tn,
                                  "lang": lg, "speaker": np.zeros(1, np.int64)})
reg = host_regulate(cond, dur, pitch, meta["abs_frame_bins"], meta["max_frames"])
mel = dec.run(None, {k: reg[k] for k in
      ["frames","frame_meta","local_ctx_raw","abs_pos","pitch_frame","frame_mask"]})[0]
wav = voc.run(None, {"mel": mel.astype(np.float32)})[0].reshape(-1)
sf.write("out.wav", wav, meta["sample_rate"])
```
The whole pipeline — `encoder.onnx → numpy length‑regulator → decoder.onnx → vocoder.onnx` — is
torch‑free and runs as‑is on a Jetson Nano CPU. See `scripts/synth_from_text.py` for the full runtime.

---

## Training data

Everything is **distilled from a single teacher voice** so zh / en / code‑mix share one timbre and accent.

- **Reference voice** — a young Taiwan‑female sample synthesized with **Edge‑TTS `zh‑TW‑HsiaoChenNeural`**
  (a genuine Taiwan‑Mandarin voice that also reads English). This is what fixes the accent: Taiwan
  Mandarin can't be prompted out of a generic model, so we *clone a Taiwan reference* instead.
- **Teacher** — **VoxCPM2** (`openbmb/VoxCPM2`) voice‑clones that one reference for every line, giving a
  consistent young‑female voice across all three languages (48 kHz, resampled to 8 kHz for training).
- **Text** — Taiwan office / phone‑attendant register: diverse Mandarin, general + domain English, and
  frame‑bank code‑mix (`gen_codemix.py`) with English in varied positions.
- **ASR quality gate** — every clip is transcribed and kept only if it matches its text, using a
  **Taiwan‑tuned recognizer** so the gate never penalizes the accent we want:
  - zh & code‑mix → **Breeze‑ASR‑25** Han‑level CER (≤ 0.12 zh, ≤ 0.15 mix)
  - English → **Whisper‑medium** WER (≤ 0.20)

| split | clean clips | dropped by gate |
|---|---|---|
| pure Chinese | 2,077 | 16.9% |
| code‑mix (zh+en) | 2,571 | 8.2% |
| pure English | 1,975 | 1.2% |
| **total** | **6,623** | 9.3% |

---

## How it was trained — the levers

Inflect‑Nano‑v1's 4.63M architecture is **not capacity‑limited** for this task. Quality came from four
fixable things, all keeping the **architecture frozen** (no NAS, no param changes):

1. **Phone‑level alignment** (`align_durations_v4.py`) — true per‑phone durations (espeak phoneme‑CTC +
   `torchaudio.forced_align`) instead of crude char/letter CTC. Skipping this is what makes tiny TTS garble.
2. **Vocabulary coverage + diverse code‑mix** — broad character coverage and a code‑mix **frame bank**
   (varied syntax, English in varied positions) so the model isn't overfit to a few templates.
3. **Teacher choice** — the English a tiny model learns is only as native as the teacher's. A Taiwan‑
   biased teacher gave flat, accented English; **VoxCPM2** gives clean, natural zh *and* en in one voice.
4. **Warm‑start from Inflect‑Nano‑v1** — the acoustic model is initialized from the English‑native v1
   checkpoint (**199/199 tensors copied, 0 skipped** — the bilingual symbol table already matches), so
   v1's English transfers directly; the corpus then teaches Taiwan Mandarin on top.

A **2D mel‑GAN** discriminator (training‑only; ONNX is unchanged) sharpens the mel after a 25k pure‑
reconstruction warmup, lifting PESQ/MOS. The shipped checkpoint is the **35k step** — the held‑out sweet
spot, *before* the GAN starts over‑sharpening held‑out intelligibility.

---

## Architecture

- **Acoustic** — `MicroFastSpeech` (~3.47M): depthwise Conv‑FFN, **no attention**, external durations +
  length regulator, frame‑pitch, BiGRU, postnet.
- **Vocoder** — Snake‑HiFiGAN (~1.17M), **8 kHz variant** `snake_8k` (sr 8000, n_fft 512, hop 128, 80 mels),
  retrained on the teacher corpus.
- **Frontend** — `g2pw` (Taiwan bopomofo + polyphone disambiguation) + `g2p_en` (arpabet), merged into
  one phone sequence with per‑phone language ids → handles zh, en, and code‑mix in a single pass.

---

## Reproduce / fine‑tune your own

Pipeline: **teacher corpus → ASR gate → align → train vocoder → warm‑start + train acoustic → export**.
Repo layout:

```
acoustic_encoder.onnx  acoustic_decoder.onnx  vocoder.onnx  meta.json  symbol_table.json   ← deployable weights
acoustic_tw_8k_35k.pt                                                                       ← shipped checkpoint
scripts/        frontend, aligner, corpus‑gen, diverse‑text, train, export, eval
inflect_nano/   the trainer (acoustic.py + vocoder.py), forked from Inflect‑Nano‑v1 (LICENSE included)
```

**Prerequisites:** Python 3.12, a GPU for training; `pip install torch torchaudio transformers
onnxruntime soundfile librosa g2pw g2p_en cn2an opencc faster-whisper edge-tts`.

### 1 · Teacher corpus (one cloned voice)
```bash
# make a Taiwan‑female reference, then VoxCPM2‑clone every line in that voice
edge-tts --voice zh-TW-HsiaoChenNeural --text "<ref sentence>" --write-media ref.mp3
python gen_voxcpm_corpus.py --texts texts.jsonl --ref ref.wav --ref-text ref.txt \
       --out-dir corpus --manifest manifest.jsonl
```

### 2 · ASR quality gate (Taiwan‑tuned)
```bash
python asr_filter.py --manifest manifest.jsonl --out manifest \
       --device cuda            # Breeze‑ASR‑25 (zh/mix) + Whisper‑medium (en) → manifest.clean.jsonl
```

### 3 · Phone‑level alignment  ⭐ *the key step*
```bash
python scripts/align_durations_v4.py --manifest manifest.clean.jsonl --out align.jsonl
```

### 4 · Train the 8 kHz vocoder
```bash
PYTHONPATH=. python -m inflect_nano.vocoder --train-jsonl voc_rows.jsonl \
  --out-dir vocoder_8k --variant snake_8k --steps 40000 --stft-weight 2.0
```

### 5 · Warm‑start + train the acoustic (GAN recipe)
```bash
PYTHONPATH=. python -m inflect_nano.acoustic --durations-jsonl align.jsonl \
  --out-dir acoustic_8k --vocoder-variant snake_8k --sample-rate 8000 \
  --vocoder-checkpoint vocoder_8k/hifigan-snake_8k-final.pt --vocoder-mel-weight 1.0 \
  --init-checkpoint inflect_nano_v1_acoustic.pt \
  --mel-gan-weight 0.1 --gan-2d --gan-fm-auto --gan-r1-gamma 1.0 --gan-crop 128 --gan-warmup-steps 25000 \
  --steps 60000 --batch-size 16 --en-upsample 1
```

### 6 · Export to ONNX + evaluate
```bash
python scripts/export_8k.py --acoustic-ckpt acoustic_8k/…pt --vocoder-ckpt vocoder_8k/…pt --out-dir onnx/
python scripts/synth_from_text.py --onnx-dir onnx --out-dir syn --texts eval.jsonl
python scripts/assess_big.py --synth-dir syn        # offline CER/WER
```
Evaluate on **≥30 held‑out sentences** — small eval sets are too noisy to trust. Sweep checkpoints and
pick the **held‑out** sweet spot (the GAN keeps improving train‑set sharpness past the held‑out optimum).

---

## Credits & licenses

- **Base model / trainer:** [`owensong/Inflect-Nano-v1`](https://huggingface.co/owensong/Inflect-Nano-v1) (Apache‑2.0; see `inflect_nano/LICENSE.inflect-nano`)
- **Teacher TTS:** [`openbmb/VoxCPM2`](https://huggingface.co/openbmb/VoxCPM2) · **Reference voice:** Microsoft **Edge‑TTS** `zh‑TW‑HsiaoChenNeural`
- **Gate ASR:** `Breeze-ASR-25` (MediaTek Research, Taiwan Mandarin + code‑switch) · OpenAI **Whisper‑medium**
- **Aligner:** `facebook/wav2vec2-lv-60-espeak-cv-ft` + `torchaudio.forced_align`
- **Frontend:** `g2pw` (Taiwan readings) + `g2p_en` · **Eval ASR:** sherpa‑onnx X‑ASR (zh‑en Zipformer)

This repository: **Apache‑2.0**.