README: document VoxCPM2-TW teacher, v1 warm-start, ASR gate, GAN recipe + real metrics/data

README.md

@@ -22,19 +22,35 @@ pipeline_tag: text-to-speech
 
 A **4.63M‑parameter** Mandarin (Taiwan) + English text‑to‑speech model that runs **entirely on CPU**
 and emits **8 kHz** audio — sized for **G.711 telephony** and **on‑device (Jetson‑class)** use.
-One model, one voice: Chinese, English, and code‑mix through a single frontend (no language routing).
+One model, **one young‑female voice**: Chinese, English, and code‑mix through a single frontend (no
+language routing).
 
-> 🔊 **Live demo:** https://huggingface.co/spaces/Luigi/PrimeTTS-vs-Inflect-Nano-v1 · 🧩 **Base:** [`owensong/Inflect-Nano-v1`](https://huggingface.co/owensong/Inflect-Nano-v1) (fine‑tune, **same frozen architecture**)
+> 🔊 **Live demo:** https://huggingface.co/spaces/Luigi/PrimeTTS-vs-Inflect-Nano-v1 · 🧩 **Base:** [`owensong/Inflect-Nano-v1`](https://huggingface.co/owensong/Inflect-Nano-v1) (warm‑started fine‑tune, **same frozen architecture**)
 
 | | |
 |---|---|
 | **Parameters** | 4.63M (3.47M acoustic + 1.17M vocoder) |
 | **Sample rate** | 8 kHz (telephony‑band) |
 | **Runtime** | `onnxruntime`, **CPU‑only**, torch‑free at inference |
-| **Languages** | zh‑TW (Traditional) + English + code‑mix |
+| **Languages** | zh‑TW (Traditional) + English + code‑mix, single voice |
+| **Voice** | young female, **Taiwan‑Mandarin accent** |
 | **Architecture** | FastSpeech‑style (no attention) + Snake‑HiFiGAN — **frozen**, no NAS |
 | **License** | Apache‑2.0 |
 
+### Held‑out quality (eval_big, 36 unseen phone‑attendant sentences)
+
+| metric | this model | prior 8k release |
+|---|---|---|
+| zh‑CER (Breeze‑ASR‑25) | **0.090** | 0.132 |
+| code‑mix CER | **0.178** | 0.254 |
+| en‑WER (Whisper) | **0.083** | 0.092 |
+| Taiwan‑accent gap¹ | **+0.088** | +0.017 |
+| SQUIM PESQ | **3.31** | 2.07 |
+| SQUIM MOS | **4.24** | 3.30 |
+
+¹ `CER(generic ASR) − CER(Taiwan‑tuned Breeze‑ASR‑25)` per zh clip; `>0` ⇒ a Taiwan‑tuned recognizer
+understands it better ⇒ genuine Taiwan accent present.
+
 ---
 
 ## Quickstart (inference, CPU)
@@ -70,22 +86,49 @@ torch‑free and runs as‑is on a Jetson Nano CPU. See `scripts/synth_from_text
 
 ---
 
-## Why it works — the two levers
+## Training data
+
+Everything is **distilled from a single teacher voice** so zh / en / code‑mix share one timbre and accent.
 
-Inflect‑Nano‑v1's 4.63M architecture is **not capacity‑limited** for this task (the original English
-checkpoint already scores ~0.05 CER). Our first retrains were still unintelligible — **not** because of
-size, but because of two fixable things. Both fixes keep the **architecture frozen**:
+- **Reference voice** — a young Taiwan‑female sample synthesized with **Edge‑TTS `zh‑TW‑HsiaoChenNeural`**
+  (a genuine Taiwan‑Mandarin voice that also reads English). This is what fixes the accent: Taiwan
+  Mandarin can't be prompted out of a generic model, so we *clone a Taiwan reference* instead.
+- **Teacher** — **VoxCPM2** (`openbmb/VoxCPM2`) voice‑clones that one reference for every line, giving a
+  consistent young‑female voice across all three languages (48 kHz, resampled to 8 kHz for training).
+- **Text** — Taiwan office / phone‑attendant register: diverse Mandarin, general + domain English, and
+  frame‑bank code‑mix (`gen_codemix.py`) with English in varied positions.
+- **ASR quality gate** — every clip is transcribed and kept only if it matches its text, using a
+  **Taiwan‑tuned recognizer** so the gate never penalizes the accent we want:
+  - zh & code‑mix → **Breeze‑ASR‑25** Han‑level CER (≤ 0.12 zh, ≤ 0.15 mix)
+  - English → **Whisper‑medium** WER (≤ 0.20)
 
-| Lever | What went wrong / the fix | Held‑out Mandarin CER |
+| split | clean clips | dropped by gate |
 |---|---|---|
-| **1. Phone‑level alignment** | Crude char/letter‑CTC → wrong per‑phone durations → over‑smoothed, garbled mel. Replace with true phone‑level forced alignment (`align_durations_v4.py`). | 0.88 → **0.40** |
-| **2. Diverse training text** | A narrow corpus (~234 Han chars) leaves ~39% of held‑out characters unseen — the model can't pronounce what it never saw. Expand coverage (`select_diverse_text.py`). | 0.40 → **0.06** |
+| pure Chinese | 2,077 | 16.9% |
+| code‑mix (zh+en) | 2,571 | 8.2% |
+| pure English | 1,975 | 1.2% |
+| **total** | **6,623** | 9.3% |
 
-Applied to **both** languages in one single‑voice corpus, the same recipe yields a genuinely bilingual
-model (zh ≈ 0.07, English ≈ 0.04–0.12 WER) — no routing. Everything else is Inflect‑Nano‑v1's defaults.
+---
+
+## How it was trained — the levers
+
+Inflect‑Nano‑v1's 4.63M architecture is **not capacity‑limited** for this task. Quality came from four
+fixable things, all keeping the **architecture frozen** (no NAS, no param changes):
 
-**Takeaway for your own fine‑tune:** *alignment quality* and *vocabulary coverage* dominate. Get those
-two right and a tiny frozen model is enough.
+1. **Phone‑level alignment** (`align_durations_v4.py`) — true per‑phone durations (espeak phoneme‑CTC +
+   `torchaudio.forced_align`) instead of crude char/letter CTC. Skipping this is what makes tiny TTS garble.
+2. **Vocabulary coverage + diverse code‑mix** — broad character coverage and a code‑mix **frame bank**
+   (varied syntax, English in varied positions) so the model isn't overfit to a few templates.
+3. **Teacher choice** — the English a tiny model learns is only as native as the teacher's. A Taiwan‑
+   biased teacher gave flat, accented English; **VoxCPM2** gives clean, natural zh *and* en in one voice.
+4. **Warm‑start from Inflect‑Nano‑v1** — the acoustic model is initialized from the English‑native v1
+   checkpoint (**199/199 tensors copied, 0 skipped** — the bilingual symbol table already matches), so
+   v1's English transfers directly; the corpus then teaches Taiwan Mandarin on top.
+
+A **2D mel‑GAN** discriminator (training‑only; ONNX is unchanged) sharpens the mel after a 25k pure‑
+reconstruction warmup, lifting PESQ/MOS. The shipped checkpoint is the **35k step** — the held‑out sweet
+spot, *before* the GAN starts over‑sharpening held‑out intelligibility.
 
 ---
 
@@ -93,89 +136,80 @@ two right and a tiny frozen model is enough.
 
 - **Acoustic** — `MicroFastSpeech` (~3.47M): depthwise Conv‑FFN, **no attention**, external durations +
   length regulator, frame‑pitch, BiGRU, postnet.
-- **Vocoder** — Snake‑HiFiGAN (~1.17M), **8 kHz variant** `snake_8k` (sr 8000, n_fft 512, hop 128, 80 mels).
+- **Vocoder** — Snake‑HiFiGAN (~1.17M), **8 kHz variant** `snake_8k` (sr 8000, n_fft 512, hop 128, 80 mels),
+  retrained on the teacher corpus.
 - **Frontend** — `g2pw` (Taiwan bopomofo + polyphone disambiguation) + `g2p_en` (arpabet), merged into
   one phone sequence with per‑phone language ids → handles zh, en, and code‑mix in a single pass.
 
 ---
 
-## Train / fine‑tune your own (voice or language)
+## Reproduce / fine‑tune your own
 
-The pipeline is 5 steps: **data → align → train acoustic → train vocoder → export**. Repo layout:
+Pipeline: **teacher corpus → ASR gate → align → train vocoder → warm‑start + train acoustic → export**.
+Repo layout:
 
 ```
 acoustic_encoder.onnx  acoustic_decoder.onnx  vocoder.onnx  meta.json  symbol_table.json   ← deployable weights
-acoustic_zh_v2_35k.pt                                                                       ← checkpoint (resume/fine‑tune)
+acoustic_tw_8k_35k.pt                                                                       ← shipped checkpoint
 scripts/        frontend, aligner, corpus‑gen, diverse‑text, train, export, eval
 inflect_nano/   the trainer (acoustic.py + vocoder.py), forked from Inflect‑Nano‑v1 (LICENSE included)
 ```
 
 **Prerequisites:** Python 3.12, a GPU for training; `pip install torch torchaudio transformers
-onnxruntime soundfile librosa g2pw g2p_en cn2an opencc`. A single‑speaker teacher TTS (or clean
-recordings) for the audio. Put the trainer on your path: `PYTHONPATH=. python -m inflect_nano.acoustic …`.
+onnxruntime soundfile librosa g2pw g2p_en cn2an opencc faster-whisper edge-tts`.
 
-### 1 · Diverse text  →  `train_zh.tsv`, `train_en.tsv`
+### 1 · Teacher corpus (one cloned voice)
 ```bash
-python scripts/select_diverse_text.py --lang zh --n 6000 --out train_zh.tsv
-python scripts/select_diverse_text.py --lang en --n 6000 --out train_en.tsv
+# make a Taiwan‑female reference, then VoxCPM2‑clone every line in that voice
+edge-tts --voice zh-TW-HsiaoChenNeural --text "<ref sentence>" --write-media ref.mp3
+python gen_voxcpm_corpus.py --texts texts.jsonl --ref ref.wav --ref-text ref.txt \
+       --out-dir corpus --manifest manifest.jsonl
 ```
-Tatoeba → OpenCC `s2twp` (zh) → greedy char/word‑coverage selection. **Coverage is the #1 driver of
-held‑out quality** — don't skimp here.
 
-### 2 · Teacher corpus  →  `corpus/{*.wav, manifest.jsonl}`
+### 2 · ASR quality gate (Taiwan‑tuned)
 ```bash
-python scripts/gen_breezy_corpus.py --corpus train_zh.tsv --out-dir corpus --cer-thresh 0.30
+python asr_filter.py --manifest manifest.jsonl --out manifest \
+       --device cuda            # Breeze‑ASR‑25 (zh/mix) + Whisper‑medium (en) → manifest.clean.jsonl
 ```
-Synthesizes each line in **one voice**, keeping a clip only if an ASR transcript matches the text
-(here BreezyVoice + Breeze‑ASR‑25, t2s‑normalized). Any clean single‑speaker source works — for native
-code‑mix, use the **same voice** for zh and en.
 
-### 3 · Phone‑level alignment  →  `align.jsonl`  ⭐ *the key step*
+### 3 · Phone‑level alignment  ⭐ *the key step*
 ```bash
-python scripts/align_durations_v4.py --manifest corpus/manifest.jsonl --out align.jsonl
+python scripts/align_durations_v4.py --manifest manifest.clean.jsonl --out align.jsonl
 ```
-Per‑phone durations from the real audio via espeak phoneme‑CTC + `torchaudio.forced_align`.
-**Skipping or approximating this is what makes tiny TTS sound garbled.**
 
-### 4 · Train the acoustic model  →  `acoustic_8k/…pt`
-```bash
-PYTHONPATH=. python -m inflect_nano.acoustic --durations-jsonl align.jsonl \
-  --out-dir acoustic_8k --vocoder-variant snake_8k --sample-rate 8000 \
-  --steps 60000 --batch-size 16 --vocoder-checkpoint <vocoder.pt> --vocoder-mel-weight 1.0
-```
-Mix languages in one corpus, single speaker (see `scripts/run_bilingual.sh`).
-
-### 5 · Train the 8 kHz vocoder  →  `vocoder_8k/…pt`
+### 4 · Train the 8 kHz vocoder
 ```bash
 PYTHONPATH=. python -m inflect_nano.vocoder --train-jsonl voc_rows.jsonl \
-  --out-dir vocoder_8k --variant snake_8k --steps 40000 --stft-weight 2.5
+  --out-dir vocoder_8k --variant snake_8k --steps 40000 --stft-weight 2.0
 ```
-Train on the same diverse audio. Higher `--stft-weight` → crisper waveform (see `scripts/run_voc_retrain.sh`).
 
-### 6 · Export to ONNX  →  the deployable weights
+### 5 · Warm‑start + train the acoustic (GAN recipe)
 ```bash
-python scripts/export_8k.py --acoustic-ckpt acoustic_8k/…pt --vocoder-ckpt vocoder_8k/…pt --out-dir onnx/
+PYTHONPATH=. python -m inflect_nano.acoustic --durations-jsonl align.jsonl \
+  --out-dir acoustic_8k --vocoder-variant snake_8k --sample-rate 8000 \
+  --vocoder-checkpoint vocoder_8k/hifigan-snake_8k-final.pt --vocoder-mel-weight 1.0 \
+  --init-checkpoint inflect_nano_v1_acoustic.pt \
+  --mel-gan-weight 0.1 --gan-2d --gan-fm-auto --gan-r1-gamma 1.0 --gan-crop 128 --gan-warmup-steps 25000 \
+  --steps 60000 --batch-size 16 --en-upsample 1
 ```
 
-### 7 · Evaluate
+### 6 · Export to ONNX + evaluate
 ```bash
+python scripts/export_8k.py --acoustic-ckpt acoustic_8k/…pt --vocoder-ckpt vocoder_8k/…pt --out-dir onnx/
 python scripts/synth_from_text.py --onnx-dir onnx --out-dir syn --texts eval.jsonl
-python scripts/assess_big.py --synth-dir syn        # offline X‑ASR CER/WER
+python scripts/assess_big.py --synth-dir syn        # offline CER/WER
 ```
-Use **≥30 held‑out sentences** — small eval sets are too noisy to trust.
-
-> **Note on the published weights:** these are the **zh‑focused** checkpoint (strong Mandarin). The
-> more balanced **bilingual** checkpoint (zh ≈ 0.07 / English ≈ 0.04 WER) replaces them as training
-> completes; the demo always loads whatever is here.
+Evaluate on **≥30 held‑out sentences** — small eval sets are too noisy to trust. Sweep checkpoints and
+pick the **held‑out** sweet spot (the GAN keeps improving train‑set sharpness past the held‑out optimum).
 
 ---
 
 ## Credits & licenses
 
 - **Base model / trainer:** [`owensong/Inflect-Nano-v1`](https://huggingface.co/owensong/Inflect-Nano-v1) (Apache‑2.0; see `inflect_nano/LICENSE.inflect-nano`)
-- **Teacher / gate ASR:** BreezyVoice · `Breeze-ASR-25` (MediaTek Research)
+- **Teacher TTS:** [`openbmb/VoxCPM2`](https://huggingface.co/openbmb/VoxCPM2) · **Reference voice:** Microsoft **Edge‑TTS** `zh‑TW‑HsiaoChenNeural`
+- **Gate ASR:** `Breeze-ASR-25` (MediaTek Research, Taiwan Mandarin + code‑switch) · OpenAI **Whisper‑medium**
 - **Aligner:** `facebook/wav2vec2-lv-60-espeak-cv-ft` + `torchaudio.forced_align`
 - **Frontend:** `g2pw` (Taiwan readings) + `g2p_en` · **Eval ASR:** sherpa‑onnx X‑ASR (zh‑en Zipformer)
-- **Text:** Tatoeba (CC‑BY 2.0 FR)
 
 This repository: **Apache‑2.0**.