--- language: - ne - en license: cc-by-4.0 pipeline_tag: text-to-speech library_name: matcha-tts tags: - matcha-tts - text-to-speech - tts - nepali - multispeaker - multi-speaker - speaker-conditioning - flow-matching - hifi-gan - speech-synthesis - pitch-filtered --- # Matcha-TTS Nepali — v3, multi-speaker (pitch-filtered male) Self-contained release of the 2-speaker Nepali Matcha-TTS model: acoustic checkpoint, vocoder, exact training configs, the `matcha` package, a seeded synthesis script and reference audio. Everything needed to reproduce the shipped samples bit-for-bit is in this directory. - **Model:** - **Source / training pipeline:** (branch `feature/finetune-english-male-voice`, bundle under `release/version3-multispeaker-pitchfiltered/`) The GitHub copy gitignores the checkpoint and vocoder, so it is code and samples only — this Hugging Face repo is the runnable one. Previous versions: [v2](https://huggingface.co/Imbatmann/matcha-tts-nepali-multispeaker-v2), [v1](https://huggingface.co/Imbatmann/matcha-tts-nepali-single-v1). **What changed vs v2:** v2 trained on the raw `chunks/001` corpus whose audio is ~1/3 female-range pitch; both speakers converged to a compromise ~185 Hz voice that flipped registers on words absent from the male data. For v3 the male corpus was **filtered by median F0 (≤155 Hz)** — 479 of 732 chunks kept — and training warm-started from the pure female model instead of the male-biased single-speaker one, so no contradictory pitch enters the shared weights. Result: a stable male register (~100–115 Hz) including on out-of- vocabulary words, and an unchanged clean female voice. | | | | --- | --- | | Run | `logs/train/male_female_2spk_v2/runs/2026-08-21_17-10-56` | | Branch | `feature/finetune-english-male-voice` | | Trained | 2026-08-21, 17:10 → 19:00 (~1 h 50 min), single NVIDIA GB10 | | Sample rate | 22 050 Hz, 80-band mel, hop 256 | | Speakers | 2 | | Vocabulary | `union` symbol set, `n_vocab = 264` | | Params | 20.9 M | ## Speaker ids | spk | Voice | Training data | | --- | --- | --- | | `0` | male (pitch-filtered podcast chunks) | 455 train / 24 val utterances | | `1` | female (chatterbox corpus) | 1441 train / 75 val utterances | Keep this ordering everywhere — the API `voice` field maps `"male" -> 0`, `"female" -> 1`. ## Contents ``` version3-multispeaker-pitchfiltered/ ├── README.md this file ├── synthesize.py seeded, standalone synthesis (v3 recipe) ├── synth_samples.sh regenerates every wav in samples/ ├── requirements.txt exact pinned runtime ├── MANIFEST.sha256 checksums for every shipped file ├── checkpoints/ │ └── matcha_male_female_2spk_v3.ckpt epoch 199 / step 5999 (final = last.ckpt) ├── vocoder/ │ └── g_02500000 HiFi-GAN universal v1 generator ├── configs/ │ ├── experiment_male_female_2spk_v2.yaml hydra experiment config │ ├── data_male_female_2spk.yaml hydra data config │ ├── train_config_resolved.yaml fully-resolved config as trained │ ├── train_overrides.yaml CLI overrides for the run │ └── train.log training log for the run ├── matcha/ the model/text/vocoder package └── samples/ ├── texts.txt 6 source sentences ├── texts_medium.txt 6 two-clause sentences ├── texts_long.txt 4 paragraph-length passages ├── phonemes.txt their G2P output + token counts ├── epoch199/{spk0_male,spk1_female}/ 000–005.wav ├── tests_medium/{spk0_male,spk1_female}/ 000–005.wav (4–9 s) └── tests_long/{spk0_male,spk1_female}/ 000–003.wav (14–25 s) ``` > **Fixed 2026-09-02 — mel-stats correction was silently disabled.** > `synthesize.py` guarded the correction with `isinstance(stats, dict)`, but > `model.hparams.data_statistics` is an omegaconf `DictConfig`, so the test > was always false and the correction the recipe below depends on never ran. > Effect: ~15–20 dB of extra broadband hiss (noise floor −63 dB instead of > −87 dB), audible as a metallic background, worst on the female voice and on > long utterances. Only this bundle was affected — `scripts/synthesize.py` > and `deploy/server.py` index the stats directly and were always correct. > Everything under `samples/tests_*` is rendered with the fixed script; the > `samples/epoch199/` wavs predate the bug and were always correct. ## Checkpoint `matcha_male_female_2spk_v3.ckpt` is the final epoch (199). Validation loss: best 3.1268 @ step ~1500, final 3.204 — flat-ish across the run like v2, but unlike v2 **late epochs are what you want here**: speaker separation and the male register keep sharpening to the end (ep49 male median was 142 Hz with register flips; ep199 is 111 Hz stable). Use it as shipped. ## Synthesis ```bash pip install -r requirements.txt # see the note about `real_nepali` # single sentence python synthesize.py --spk 0 --text "नमस्ते! हाम्रो सेवा केन्द्रमा स्वागत छ।" --out male.wav python synthesize.py --spk 1 --text "नमस्ते! हाम्रो सेवा केन्द्रमा स्वागत छ।" --out female.wav # a file of sentences, one per line python synthesize.py --spk 0 --file samples/texts.txt --out out_dir/ ``` Defaults ARE the verified v3 recipe: `--steps 50 --temperature 0 --denoiser_strength 0.02`, mel-stats correction applied automatically, HiFi-GAN universal v1, seed 1234. This combination is what fixed the metallic "pause-and-play" artefact of earlier attempts (denoiser 0.001 + mel smoothing thinned the signal; temperature 0.667 added register randomness). Useful extras: `--speaking_rate 1.1` (>1 is slower), `--lowpass_khz 9`, `--print-phonemes`. ### Text frontend Devanagari is phonemized by the hand-written `real_nepali` G2P (profile `real_nepali_v0.2`), **not** espeak-ng — from the `nepali_frontend` repo pinned at commit `b7395f28`, installed editable. Latin runs inside mixed text go through espeak-ng `en-us`, so `espeak-ng` must be on `PATH` for code-switched input. `samples/phonemes.txt` records the token string and id count per sample sentence (`--print-phonemes` to compare). > **Note on `spk0_male/000.wav`:** this is the human-approved master take > (pinned byte-for-byte, also shipped as `000_master.wav` and restored by > `synth_samples.sh` after a re-render). The seeded re-render of the same > sentence is acoustically equivalent (same duration, ~95–100 Hz male) but > not byte-identical — at temperature 0 the flow-matching prior still draws > its initial latent from the RNG, so the seed shifts micro-timing only. ## Reproducibility `synthesize.py` seeds the RNG per utterance (`--seed`, default 1234); with temperature 0 output is deterministic anyway, but seeding keeps batch order independent too. ```bash PYTHON=python bash synth_samples.sh # re-render everything under samples/ sha256sum -c MANIFEST.sha256 # confirm byte-for-byte match ``` Bit-exact only on the same GPU/library stack (NVIDIA GB10, CUDA 13.0, torch 2.13.0+cu130); elsewhere perceptually identical but different checksums. ## Training recipe Warm-started from the **pure female model** (`checkpoints/matcha_nepali_final.ckpt`) widened to 2 speakers by `scripts/adapt_1spk_to_2spk.py` (old weights embedded top-left, new speaker channels zeroed, fresh random `spk_emb`). The male voice is therefore learned *only* from pitch-consistent data — no female-range segments pull the shared weights around. Lineage: English LJSpeech base → Nepali (union symbols) → **2-speaker male(pitch-filtered)+female** (this release). Data pipeline (reproduce before training): ```bash .venv/bin/python scripts/filter_chunks_by_pitch.py \ --chunks .../data/chunks/001 --out .../data/chunks/001_male --max-f0 155 .venv/bin/python scripts/prepare_chunks.py \ --chunks .../data/chunks/001_male --name male_001m --spk-id 0 bash scripts/compute_stats.sh male_001m data/filelists_male_001m .venv/bin/python scripts/build_multispk_filelists.py bash scripts/compute_stats.sh male_female_2spk data/filelists_multispk .venv/bin/python scripts/adapt_1spk_to_2spk.py \ --src checkpoints/matcha_nepali_final.ckpt \ --out checkpoints/male_female_2spk_adapted_v2.ckpt bash scripts/train.sh male_female_2spk_v2 ``` | | | | --- | --- | | Init | `checkpoints/male_female_2spk_adapted_v2.ckpt` | | Optimizer | Adam, lr `1e-4`, no scheduler, weight decay 0 | | Epochs | 200 (ran to completion), ~30 steps/epoch | | Batch | 32, `accumulate_grad_batches: 2` | | Precision | `16-mixed`, `gradient_clip_val: 5.0` | | Encoder | RoPE, 6 layers, 192 ch, 2 heads, `spk_emb_dim: 64` | | Decoder | CFM/euler, `sigma_min 1e-4`, channels [256, 256], snakebeta | | Mel stats | `mel_mean -5.840723`, `mel_std 2.485387` (merged filelist) | | Seed | 1234 | Full settings in `configs/train_config_resolved.yaml`. ## Deploying `deploy/server.py` in the main repo already serves this model: `CHECKPOINT` → `matcha_nepali_v3.ckpt`, `DEFAULT_STEPS=50`, `DEFAULT_TEMPERATURE=0.0`, mel-stats correction on by default, `voice` mapped via `VOICE_TO_SPK` (`male/am_male -> 0`, `female/af_kore -> 1`). ```bash curl -s http://localhost:8002/v1/audio/speech \ -H 'Content-Type: application/json' \ -d '{"input": "नमस्ते", "voice": "male", "response_format": "wav"}' -o /tmp/male.wav curl -s http://localhost:8002/v1/audio/speech \ -H 'Content-Type: application/json' \ -d '{"input": "नमस्ते", "voice": "female", "response_format": "wav"}' -o /tmp/female.wav ``` ## Known limitations - Male corpus is 479 utterances after filtering (was 732); rare words still have thin coverage — pronunciation is inherited from the shared network and is good, but prosody on unseen words is generic. - Opening phrase of an utterance can sit slightly high (~165 Hz) before settling into the male register within ~1 s. - Occasional whole-sentence register flip on loanword-dense male input: `samples/tests_medium/spk0_male/002.wav` ("इन्टरनेट राउटर पुनः सुरु") renders at ~159 Hz median rather than the usual ~105 Hz. The same content in `tests_long/spk0_male/001.wav` is correct, so it is sentence-specific, not a length effect. - No code-switched utterances in either corpus; mixed Nepali+English in one sentence garbles at the language transition.