---
license: apache-2.0
language:
  - zh
  - en
tags:
  - text-to-speech
  - tts
  - onnx
  - gguf
  - on-device
  - jetson
  - telephony
  - vits
  - mb-istft-vits
  - mandarin
  - taiwanese-mandarin
base_model: owensong/Inflect-Nano-v1
base_model_relation: finetune
library_name: onnxruntime
pipeline_tag: text-to-speech
---

# PrimeTTS — on-device zh-TW + English TTS

Taiwan-Mandarin + English text-to-speech built for on-device use (contact-centre, GPS, transit): one
voice across Chinese, English, and code-mix through a single frontend (no language routing), with
**entity correctness** — phone numbers, emails, addresses, prices, dates, temperatures, %, serials.

Two model generations:

- **`v2_mbistft_16k/` — PrimeTTS v2 (34.7M, 16 kHz) — current flagship.** End-to-end
  **MB-iSTFT-VITS** targeting the Jetson Nano **GPU** (and any CPU via ONNX). Best quality and
  intelligibility of the family; female Mandarin voice ("Xinran").
- **`v1b_16k/` / `v1b_8k/` — PrimeTTS v1 (~5.0M / 4.09M).** FastSpeech + Snake-HiFiGAN, pure-**CPU**,
  young-female zh-TW voice; `v1b_8k` reaches **RTF 0.35 on a Jetson Nano CPU** (1 thread). Use v1
  when the deployment budget is CPU-only and tight.

> 🔊 **Live demo (serves v2 + v1):** https://huggingface.co/spaces/Luigi/PrimeTTS-vs-Inflect-Nano-v1

## PrimeTTS v2 (`v2_mbistft_16k/`)

| | PrimeTTS v2 |
|---|---|
| **Architecture** | MB-iSTFT-VITS (end-to-end VAE + flow + adversarial; multi-band iSTFT head; conv-only, no LSTM) |
| **Parameters** | 34.7M (generator) |
| **Sample rate** | 16 kHz |
| **Voice** | female Mandarin, "Xinran" |
| **Eval (36 held-out zh/mix/en sentences)** | X-ASR CER **0.027** overall — zh 0.033 · code-mix 0.039 · en 0.008 (below its 7B teacher's 0.043 on the same eval) |
| **Runtime** | single ONNX (`primetts_v2_xinran.onnx`, ORT-CPU) · `primetts_v2_xinran.gguf` for the ggml-CUDA Jetson-Nano runtime ([RapidSpeech.cpp](https://github.com/vieenrose/RapidSpeech.cpp), `mbistft-vits` arch) |

### On-device deployment (measured on a Jetson Nano gen-1, Tegra X1)

Two runtime tiers, both real-time. RTF = compute-time / audio-time (lower is faster; <1.0 = real-time).

| Tier | Runtime | Precision | RTF | Quality |
|---|---|---|---|---|
| **GPU** | RapidSpeech.cpp ggml-CUDA, 1 CPU thread | fp32 | **0.42** (2.4× RT) | full (parity 0.9998) |
| **CPU** *(default)* | onnxruntime, 2 threads | **fp16** | **0.77** (1.3× RT) | full — voice-cos 0.915, CER 0.032, 55 MB |
| **CPU** | onnxruntime, 4 threads | fp32 | **0.52** (1.9× RT) | full — voice-cos 0.916, CER 0.033 |

Notes: both tiers are full-fidelity, real-time, and need no GPU. The GPU RTF is launch-overhead-bound on Maxwell (sm_53, no CUDA-graph replay) — 0.42 is the practical floor. **fp16** is the shipped CPU default: half the size, lossless, and leaves cores free (on this ARMv8.0 CPU it casts to fp32 so it's not *faster* than fp32 — the size/headroom is the win). **On int8:** static-int8 was fast but shifted the voice (voice-cos 0.748); dynamic-int8 preserves the voice (0.975) but runs *slower* than fp32 on this core (no dot-product / no FP16 arithmetic on the Cortex-A57). So int8 is offered only as a smaller *download* (`quantize_dynamic`, QUInt8) — not a speed tier. The on-device speed lever is a **smaller model**, not quantization.

**Training:** distilled from a **VibeVoice-Large** (MIT) teacher speaking the `zh-Xinran_woman` preset —
29k utterances over the same entity-rich zh-TW corpus as v1, per-utterance speaker-consistency QC
(retry-regenerated until >99% of clips match the target voice), trained from scratch at 16 kHz with a
3-embedding frontend (phone + tone + language, 88 symbols) and deterministic duration predictor.

```python
# v2 quickstart — one session, one call
import numpy as np, onnxruntime as ort, soundfile as sf
import sys; sys.path.insert(0, "PrimeTTS/scripts")
import frontend_bopomofo as F

sess = ort.InferenceSession("PrimeTTS/v2_mbistft_16k/primetts_v2_xinran.onnx",
                            providers=["CPUExecutionProvider"])
o = F.text_to_ids("您好,歡迎使用 PrimeTTS。Thank you for calling.")
blank = lambda s: np.array([[0] + [v for x in s for v in (x, 0)]], np.int64)  # add_blank=true
wav = sess.run(None, {"x": blank(o["phone_ids"]), "tone": blank(o["tone_ids"]),
                      "lang": blank(o["lang_ids"]),
                      "x_lengths": np.array([2*len(o["phone_ids"])+1], np.int64),
                      "noise_scale": np.array([0.667], np.float32),
                      "length_scale": np.array([1.0], np.float32)})[0].reshape(-1)
sf.write("out.wav", wav, 16000)
```

---

# PrimeTTS v1 (legacy CPU family)

| | flagship `v1b_16k/` | on-device `v1b_8k/` |
|---|---|---|
| **Parameters** | **~5.0M** (3.56M acoustic + 1.43M vocoder) | **4.09M** (3.56M + 0.53M) |
| **Sample rate** | **16 kHz** (0–8 kHz) | 8 kHz (0–4 kHz) |
| **SQUIM PESQ** | **2.70** | 2.22 |
| **zh-CER / en-WER** | 0.149 / 0.083 | 0.109 / 0.083 |
| **Jetson Nano RTF** | (heavier) | **0.35** (1 thread) |

*Runtime: `onnxruntime`, CPU-only, torch-free. Voice: young female, Taiwan-Mandarin accent. License: Apache-2.0.*

## Highlights

- **Tiny + CPU-only** — ~4M params, ONNX, torch-free; real-time on a Jetson Nano (**RTF 0.35, single thread**).
- **One voice, three modes** — zh / en / code-mix share one timbre and accent through a single frontend; no
  language tag needed.
- **Mandarin tones via a frame-pitch refiner** — a 97K-param module turns coarse per-phoneme pitch into a
  per-frame F0 contour (the tone carrier). Ablating it costs **+18% relative zh-CER** (zh-only; English is
  unaffected — no lexical tone).
- **Entity-correct** — a normalization layer reads numbers, dates, prices, emails, addresses, serials, and
  spells acronyms/letters (VIP → V-I-P), applied identically in training and at inference.

## Performance — held-out (36 unseen phone-attendant sentences)

Per-checkpoint headline numbers are in the table at the top. Both checkpoints share the same intelligibility
and accent (**en-WER 0.083**; Taiwan-accent gap **+0.033**¹ ⇒ a genuine TW accent); they differ in **clarity**
(16 kHz **PESQ 2.70** vs 8 kHz 2.22) and **footprint** (8 kHz: **RTF 0.347** on a Jetson Nano, 1 thread).
zh-CER sits in the **0.11–0.15** band depending on checkpoint and recognizer, and **~half of the residual is
ASR homophone/variant noise**, not synthesis error (see *Known characteristics*).

¹ `CER(generic ASR) − CER(Taiwan-tuned Breeze-ASR-25)` per zh clip; `>0` ⇒ a Taiwan recognizer understands
it better ⇒ a real Taiwan accent is present.

> **On sample rate & clarity:** 8 kHz caps the band at 4 kHz (Nyquist), discarding the brightness/sibilance
> above it — intelligible but *telephone-band*. The **16 kHz flagship** doubles the band to 0–8 kHz
> (**PESQ 2.70 vs 2.22**, recovering ~22% of the energy 8 kHz throws away) while staying CPU-only. Choose the
> 8 kHz checkpoint only when you need the absolute tightest on-device RTF.

## Architecture

**Acoustic — `MicroFastSpeech` (3.56M).** FastSpeech-style, **no attention**: depthwise gated Conv-FFN,
external durations + length regulator, frame-pitch, BiGRU, postnet — plus the **frame-pitch refiner**
(`Conv1d → SiLU → Conv1d(groups=4) → SiLU → Conv1d`, 97K) that builds the per-frame F0 contour = Mandarin tones.

```json
{ "vocab_size": 256, "tone_size": 16, "lang_size": 4, "n_mels": 80,
  "hidden": 168, "encoder_layers": 5, "decoder_layers": 6, "decoder_ff_mult": 3,
  "sample_rate": 8000, "max_frames": 1000, "use_frame_pitch_refiner": true }
```

**Vocoder — Snake-HiFiGAN.** The on-device model uses the lightweight `snake_8k_lite`. The family — same
architecture, different width / sample rate — is **why a model's headline param count varies**:

| variant | used by | params | rate | band | PESQ² |
|---|---|---|---|---|---|
| **`snake_16k`** | **flagship `v1b_16k/`** | **1.43M** | 16 kHz | 0–8 kHz | **2.70** |
| `snake_8k_lite` | on-device `v1b_8k/` | 0.53M | 8 kHz | 0–4 kHz | 2.22 |
| `snake_8k` | 8 kHz, full width | 1.15M | 8 kHz | 0–4 kHz | 2.60 |
| `snake_v2mid` | 24 kHz (legacy) | 1.17M | 24 kHz | 0–12 kHz | 3.23 |

² SQUIM-PESQ. The dominant lever is the **sample rate** (band), not the vocoder: 16 kHz adds the 4–8 kHz
brightness 8 kHz can't represent. `snake_8k_lite` trades ~0.26 PESQ vs the full `snake_8k` for ~2.2× less
compute — the right call for the 8 kHz on-device build, where RTF matters more than the last bit of fidelity.

**Frontend.** `g2pw` (Taiwan bopomofo + polyphone disambiguation) + `g2p_en` (arpabet), merged into one phone
sequence with per-phone language ids — zh, en, code-mix in a single pass. **88-symbol table.** Entity
normalization (`text_norm.py`) handles numbers / dates / prices / emails / serials, spells ALL-CAPS acronyms
and a small brand lexicon. Text past `max_frames` is auto-chunked at punctuation.

## Model files

```
v2_mbistft_16k/primetts_v2_xinran.onnx        ← PrimeTTS v2 FLAGSHIP fp32 (34.7M, 16 kHz) — full quality, the demo serves this
v2_mbistft_16k/primetts_v2_xinran.gguf        ← fp32 weights for the ggml-CUDA Jetson-Nano runtime (RapidSpeech.cpp)
v2_mbistft_16k/primetts_v2_xinran_fp16.onnx   ← fp16 ONNX (55 MB, half size, lossless) — the shipped CPU default (@2 threads)
v1b_16k/{acoustic_encoder,acoustic_decoder,vocoder}.onnx + meta.json  ← v1 16 kHz (~5.0M, CPU)
v1b_8k/ {acoustic_encoder,acoustic_decoder,vocoder}.onnx + meta.json  ← v1 leanest on-device (4.09M, 8 kHz, Nano CPU)
{acoustic_encoder,…}.onnx + meta.json   ·   v3_4.6M/                  ← legacy 24 kHz variants (6.85M / 4.63M), for record
scripts/        frontend, aligner, corpus-gen, train / export, eval
inflect_nano/   the v1 trainer (acoustic.py + vocoder.py), forked from Inflect-Nano-v1 (LICENSE included)
```

## Quickstart (CPU)

```bash
pip install onnxruntime numpy soundfile g2pw g2p_en cn2an
huggingface-cli download Luigi/PrimeTTS --local-dir PrimeTTS
```
```python
import sys; sys.path.insert(0, "PrimeTTS/scripts")
import json, numpy as np, onnxruntime as ort, soundfile as sf
import frontend_bopomofo as F
from synth_from_text import host_regulate            # numpy length-regulator

D = "PrimeTTS/v1b_16k"                                # the flagship (use v1b_8k for the leanest Nano RTF)
meta = json.load(open(f"{D}/meta.json"))
enc = ort.InferenceSession(f"{D}/acoustic_encoder.onnx", providers=["CPUExecutionProvider"])
dec = ort.InferenceSession(f"{D}/acoustic_decoder.onnx", providers=["CPUExecutionProvider"])
voc = ort.InferenceSession(f"{D}/vocoder.onnx",          providers=["CPUExecutionProvider"])

o = F.text_to_ids("您好,歡迎使用 PrimeTTS。Thank you for calling.")
ph, tn, lg = (np.array([o[k]], np.int64) for k in ("phone_ids", "tone_ids", "lang_ids"))
cond, dur, pitch = enc.run(None, {"phone": ph, "tone": tn, "lang": lg, "speaker": np.zeros(1, np.int64)})
reg = host_regulate(cond, dur, pitch, meta["abs_frame_bins"], meta["max_frames"])
mel = dec.run(None, {k: reg[k] for k in
      ["frames","frame_meta","local_ctx_raw","abs_pos","pitch_frame","frame_mask"]})[0]
wav = voc.run(None, {"mel": mel.astype(np.float32)})[0].reshape(-1)
sf.write("out.wav", wav, meta["sample_rate"])
```
The pipeline — `encoder → numpy length-regulator → decoder → vocoder` — is torch-free and runs as-is on a
Jetson Nano CPU. (`scripts/synth_long.py` adds the punctuation auto-chunking for long text.)

## Training

**Distilled from a single teacher voice** so zh / en / code-mix share one timbre and accent:

- **Reference voice** — a young Taiwan-female speaker from **Mozilla Common Voice zh-TW** (**CC0 / public
  domain**, commercial-clear). ~13 s of the cleanest clips fix the accent (it comes from the *reference*, not
  prompting) and keep the model shippable.
- **Teacher** — **VoxCPM2** (`openbmb/VoxCPM2`) voice-clones that reference for every line (48 kHz → resampled).
- **Corpus** — Taiwan office / phone / GPS / transit register: diverse Mandarin, general + domain English,
  code-mix in varied positions, a large **named-entity bank** (TW + world places / roads / transit / companies
  / people / products), plus a **rare-character + brand + email booster** (the latest data lever).
- **ASR gate** — Breeze-ASR-25 (zh / mix CER) + Whisper-medium (en WER) keep only clips matching their text;
  proper-noun coverage clips are trusted unfiltered.

The three levers that matter most for a tiny model: **phone-level alignment** (espeak phoneme-CTC +
`torchaudio.forced_align` — sub-syllable boundaries are what separate intelligible speech from fluent babble),
**broad coverage + diverse code-mix**, and the **teacher** (a student's English is only as native as its
teacher's). Pipeline: `teacher corpus → ASR gate → align → train vocoder → warm-start + train acoustic →
export`. The 8 kHz on-device model warm-starts the 24 kHz acoustic and adapts to 8 kHz — the trainer resamples
audio and rescales durations internally. Full commands and a one-shot `scripts/rebuild_voice.sh` (swap in your
own ~10 s reference clip) are in the repo.

## Known characteristics & limitations

- **8 kHz checkpoint is telephone-band** (4 kHz ceiling) — use the **16 kHz flagship** for full brightness.
- **Empty-rime syllables** (是 / 十 / 日, the syllabic ㄭ) and **isolated spelled letters** (the leading "A" of a
  serial) are the fragile cases at this size: the frontend emits the right phones, but a ~4M acoustic renders
  them weakly. Cross-checking a robust (Breeze) vs strict (X-ASR) recognizer exposes this where a single CER
  number hides it.
- **Phrase-initial bare vowels** in *ultra-short isolated* inputs ("二月" alone) can garble; fine in normal
  sentences.

## Credits & licenses

- **v2 architecture:** [MB-iSTFT-VITS](https://github.com/MasayaKawamura/MB-iSTFT-VITS) (Kawamura et al., Apache-2.0)
  · **v2 teacher:** VibeVoice-Large (Microsoft, **MIT**) speaking its `zh-Xinran_woman` preset
  (via the MIT [community repo](https://github.com/vibevoice-community/VibeVoice)); synthesized speech,
  AI-generated voice — mark it as such in products
- **v1 base / trainer:** [`owensong/Inflect-Nano-v1`](https://huggingface.co/owensong/Inflect-Nano-v1) (Apache-2.0)
- **v1 teacher:** [`openbmb/VoxCPM2`](https://huggingface.co/openbmb/VoxCPM2) · **v1 reference voice:**
  [Mozilla Common Voice zh-TW](https://commonvoice.mozilla.org/datasets) (**CC0 / public domain**)
- **Gate ASR:** Breeze-ASR-25 (MediaTek Research) · Whisper-medium · **Aligner:**
  `facebook/wav2vec2-lv-60-espeak-cv-ft` + `torchaudio.forced_align` · **Eval:** sherpa-onnx X-ASR

This repository: **Apache-2.0**.