--- license: cc-by-nc-4.0 language: - tw - en pipeline_tag: text-to-speech tags: - twi - akan - ghana - tts - piper - vits - ipa - code-switching --- # Twi / Ghanaian English TTS (IPA-driven) A Piper VITS voice for **Twi** that also reads the **English words** Ghanaian text is full of. It is driven by **IPA phonemes** rather than orthography, which is what lets one model serve both languages from a shared inventory. Warm-started from `en_US/libritts_r/medium` and finetuned on 327 h (164 h Twi + 164 h Ghanaian English), 22.05 kHz, 1,555 speaker embeddings. ## Use it ```bash pip install stable-twi-tts && pip install "stable-twi-tts[twi]" && apt install espeak-ng stable-twi-tts --model . --voice twi-6 --text "Akwaaba, wo ho te sen?" --out hello.wav stable-twi-tts --model . --language mixed --voice twi-1 \ --text "Mepe se mesua [computer science] wo [University of Ghana]." --out mixed.wav ``` Code: https://github.com/GhanaNLP/stable-twi-tts ## Voices are ranked by measurement, not by training hours 12 voices are exposed of 1,555 speaker embeddings. Each synthesised the same held-out text, which was re-recognised and scored for phoneme error -- Twi with `ghana-speech-phoneme-asr`, the English spans with `KoelLabs/xlsr-english-01`. | pick | for | best | |---|---|---| | `tiers.codeswitch` | text mixing English into Twi | `twi-1` (59.8%) | | `tiers.twi_only` | pure Twi | `twi-6` (26.8%; floor 25.9%) | **The two rankings disagree sharply.** `twi-1` is the best code-switch voice yet 21st of 30 on pure Twi. Two of the three best Twi voices have under 3.3 h of audio each and would never have survived an hours-based shortlist. `voices.json` records every measurement. ## Quality, honestly Round-trip phoneme error against a real-audio floor, on held-out text: | | floor | model | gap | |---|---|---|---| | Twi | 25.9% | 33.5% | +7.6 | | English | 32.2% | 59.5% | +27.3 | **Twi is good; English is markedly weaker.** English had a third of the utterances (43k vs 151k) at 3.5x the length, and VITS learns alignment per utterance, so English got far less practice on harder material. English audio was also 16 kHz upsampled to 22.05 kHz, so English voices are band-limited to 8 kHz and sound duller. Twi was 24 kHz and is unaffected. ## Training Warm-started from `en_US/libritts_r/medium` and finetuned for **10 epochs / 52,039 steps**, about **21.5 hours** on a single H200. Batch 32, `bf16-mixed`, single device, 178,833 training utterances across 1,555 speakers. The shipped weights are **epoch 7**. ### Per-validation metrics Validation ran twice per epoch. `val_mos` is UTMOS, a learned naturalness predictor — higher is better. Everything else is a loss — lower is better. | step | epoch | val_loss | val_mel | val_dur | val_kl | val_mos | |---|---|---|---|---|---|---| | 1,999 | 0 | 38.08 | 0.4792 | 1.6759 | 2.9333 | 2.563 | | 3,999 | 0 | 39.06 | 0.4859 | 1.6750 | 2.9237 | 2.647 | | 7,559 | 1 | 38.62 | 0.4851 | 1.6687 | 2.9323 | 2.658 | | 9,559 | 1 | 40.12 | 0.4813 | 1.6738 | 2.8489 | 2.697 | | 13,119 | 2 | 38.60 | 0.4738 | 1.6664 | 2.9387 | 2.746 | | 15,119 | 2 | 38.05 | 0.4743 | 1.6708 | 2.9791 | 2.993 | | 18,679 | 3 | 42.69 | 0.4821 | 1.6709 | 2.9007 | 2.695 | | 20,679 | 3 | 37.51 | 0.4734 | 1.6643 | 2.9741 | 2.853 | | 24,239 | 4 | 38.05 | 0.4774 | 1.6642 | 2.9261 | 2.809 | | 26,239 | 4 | 38.12 | 0.4734 | 1.6645 | 2.9265 | 2.873 | | 29,799 | 5 | 38.26 | 0.4815 | 1.6677 | 2.8775 | 2.977 | | 31,799 | 5 | 38.50 | 0.4727 | 1.6695 | 2.9064 | 2.772 | | 35,359 | 6 | 38.34 | 0.4743 | 1.6652 | 2.9190 | 2.795 | | 37,359 | 6 | 38.85 | **0.4688** | 1.6620 | 2.9368 | 2.945 | | 40,919 | **7** | 38.17 | 0.4741 | 1.6644 | 2.9444 | **3.018** | | 42,919 | 7 | 38.04 | 0.4795 | 1.6656 | 2.9047 | 2.896 | | 46,479 | 8 | 38.22 | **0.4686** | 1.6646 | 2.9374 | 2.761 | | 48,479 | 8 | 37.96 | 0.4776 | 1.6653 | 2.8999 | 2.749 | | 52,039 | 9 | 38.04 | 0.4732 | 1.6628 | 2.8492 | 2.879 | ### Training losses | | start | best | end | |---|---|---|---| | `loss_g` (generator, total) | 41.05 | 33.50 | 37.86 | | `loss_d` (discriminator) | 2.00 | 1.24 | 2.15 | | `train_mel` | 0.5211 | 0.4047 | 0.4741 | ### Reading these numbers **`val_loss` is not a quality signal here, and treating it as one would have been a mistake.** It starts at 38.08 and ends at 38.04 — flat across the entire run, with a 42.69 spike at epoch 3 that means nothing. That is expected for a GAN: the generator's total loss is measured against a discriminator that is improving alongside it, so the number tracks the *balance* between the two, not how good the audio is. A run that looked converged at step 2,000 by `val_loss` in fact kept improving for another 40,000 steps. The two signals that did carry information disagreed with each other: - **`val_mel`** (mel-spectrogram L1) bottomed at **0.4686 at epoch 8**, having drifted down from 0.4792. A 2.2% total improvement — real, but small. - **`val_mos`** peaked at **3.018 at epoch 7** and never recovered: 2.90, 2.76, 2.75, 2.88 after. It is also noisy — epoch 3 swings 2.695 → 2.853 within one epoch — so no single validation is trustworthy on its own. **Epoch 7 was shipped on `val_mos` plus listening**, not on `val_mel`. Epoch 8's marginally better mel did not sound better. Where a spectral distance and a naturalness predictor disagree by this little, the honest reading is that the model plateaued somewhere around epoch 6–7 and the rest is noise. The decisive check was neither: a **round-trip evaluation** — synthesise held-out text, re-recognise it with a phoneme ASR, score the phoneme error against the error the ASR makes on *real* audio of the same text. That is the "Quality, honestly" table above, and it is the number to trust, because it measures whether the audio says the right thing rather than whether it resembles a reference spectrogram. | checkpoint | Twi UER | English UER | |---|---|---| | epoch 0 | 32.1% | 64.9% | | epoch 1 | 31.3% | 63.7% | | epoch 7 | 33.5% | 59.5% | | *real-audio floor* | *25.9%* | *32.2%* | Twi was essentially converged by epoch 1; the remaining 8 epochs bought **English** (−4.2 points), which is the harder half. The Twi number moving *up* 2.2 points while listening quality clearly improved is a reminder that a 2-point UER difference is within this instrument's noise. **Training was stopped mid-epoch 9**, at step 3,459 of 5,560. Nothing was still improving: `val_mos` had been below its epoch-7 peak for four consecutive validations, and `val_mel`'s best was a checkpoint that sounded no better. Further epochs would have risked overfitting 1,555 pseudo-speakers without a metric able to tell us it was happening. ## Finetuning `finetune/` has the Lightning checkpoint, the phoneme id map, the training command and dataset stats. **The phoneme id map is required** -- the weights encode "id 26 means /n/", so loading them against a different map yields nonsense. See `finetune/FINETUNING.md`, which also documents two traps: resize the speaker table before loading, and clear the phoneme cache or stale targets are silently reused. ## Limits - **Speakers are derived, not real identities** -- pseudo-speakers from clustering unlabelled broadcast audio. One person may appear as two voices, and no voice is a consented identity. - **Trained on broadcast speech**, so it suits news-register text better than conversation. - **Training targets came from an ASR**, not human annotation, at roughly 17% phoneme error. - **cc-by-nc-4.0**: non-commercial, inherited from the Twi source audio.