MERaLiON/MERaLiON-OmniVoice-Hokkien-TTS

🔥 MERaLiON-TTS 🔥

🚀 MERaLiON-OmniVoice-Hokkien-TTS

MERaLiON-OmniVoice-Hokkien-TTS is a fine-tuned text-to-speech model that generates natural-sounding Singapore Hokkien (Southern Min / 闽南语) speech from Chinese character input. It is built on OmniVoice and fine-tuned on Hokkien conversational data using SFT (Supervised Finetuning) and GRPO (Group Relative Policy Optimization).

Hokkien is an under-resourced language in the TTS space. Many Chinese TTS systems produce Mandarin pronunciation even when given Hokkien text written in Chinese characters. This model addresses that gap by generating speech with authentic Hokkien pronunciation, tones, and prosody.

Model Details

Attributes	Details
Base model	OmniVoice
Fine-tuning method	SFT + GRPO
Language	Singapore Hokkien (Southern Min / 闽南语)
Input	Chinese characters (汉字), with support for Malay/English loanwords
Output	24kHz mono WAV audio
Voice cloning	Supported via reference audio prompt

Demo Samples

Mandarin (华语翻译)	Hokkien Text	Audio
各位来宾大家好，大家今天心情怎么样？	各位来宾大家好，大家今日心情如何？
今天的天气好吗？	今日个天气好无？
今天天气很热，想买一碗清汤来喝。	今日天氣真熱，想買一碗清湯來啉。
今天外面一直下雨，大家有带雨伞吗？	今日外口一直落雨，大家有带雨傘无？
如果下雨记得收衣服。	若落雨记得收衫。
大家来拍照！	大家來翕相！
不好意思，应该搭哪一辆车去牛车水？	歹势，应该搭哪一辆车去牛车水？
他突然肚子痛得很厉害，我们赶紧送他去看医生。	伊突然腹肚痛真厉害，咱紧送佗去老君处。
我喜欢吃红豆冰。	我 suka 食红豆冰。
大家好，吃饱了吗？今天心情好吗？	大家好，食饱未？今仔日心情好无
谁比新加坡人更怕输更怕死	誰比新加坡人更驚輸更驚死
感谢大家，谢谢大家，你喜欢我说的福建话吗？	感謝大家,多谢大家，汝suka我讲的福建话无？

Inference

Requirements

pip install torch transformers soundfile
pip install omnivoice  # or clone https://huggingface.co/k2-fsa/OmniVoice

Basic Usage

import torch
import soundfile as sf
from omnivoice.models.omnivoice import OmniVoice

model = OmniVoice.from_pretrained(
    "MERaLiON/MERaLiON-OmniVoice-Hokkien-TTS",
    device_map="cuda:0",
    dtype=torch.float16,
)

text = "你阿母煮的菜真好食"
audios = model.generate(text=text, language="nan")
sf.write("output.wav", audios[0], model.sampling_rate)

Voice Cloning

For best results, provide a short reference audio clip (3-8 seconds) of the target speaker along with its transcript. The model will clone the speaker's voice characteristics.

import torch
import soundfile as sf
from omnivoice.models.omnivoice import OmniVoice

model = OmniVoice.from_pretrained(
    "MERaLiON/MERaLiON-OmniVoice-Hokkien-TTS",
    device_map="cuda:0",
    dtype=torch.float16,
)

# Create a reusable voice clone prompt (process once, use many times)
voice_prompt = model.create_voice_clone_prompt(
    ref_audio="path/to/reference_speaker.wav",
    ref_text="Reference transcript in Hokkien",
)

# Generate with the cloned voice
text = "你阿母煮的菜真好食"
audios = model.generate(
    text=text,
    language="nan",
    voice_clone_prompt=voice_prompt,
)
sf.write("output.wav", audios[0], model.sampling_rate)

Tips:

• Use a clean, single-speaker reference clip at 16kHz mono WAV.
• The reference text should match what is spoken in the reference audio.
• language="nan" specifies Southern Min (ISO 639-3 code for Hokkien).
• create_voice_clone_prompt() encodes the reference once; reuse the returned prompt across multiple generations to avoid redundant processing.

Evaluation

We conducted a systematic evaluation comparing MERaLiON-OmniVoice-Hokkien-TTS against six other TTS systems on a benchmark of 25 Hokkien sentences. The evaluation covers four complementary dimensions: language correctness, transcription accuracy, perceived naturalness, and signal quality.

Test Set

The test set consists of 25 Singapore Hokkien sentences written in Chinese characters, ranging from short phrases (3-4 characters) to complex sentences (20+ characters). They cover everyday topics — greetings, directions, medical, food — and include Malay loanwords (e.g., "suka") that are characteristic of Singapore Hokkien.

We plan to expand evaluation with a lager phonetically-balanced test set and human MOS in future updates.

Full test set (25 sentences)

ID	Text
001	你阿母煮的菜真好食
002	伊走了真紧
003	这件代志真要紧
004	我明仔载要去做工
005	你为啥物毋来？
006	按呢袂使
007	囡仔佇外口咧耍
008	怹已经食过饭矣
009	我毋知影
010	你佇佗位？
011	找个时间共去呷杯咖啡讲古
012	週末去超市買點物件，順便去公園
013	毋通只顧做工，多飲水，較少熬夜
014	你出门能顺便帮我把快递投进柜子里吗？
015	我最近胃口袂好，想食點清淡个
016	我个手机快毋着电矣，汝带充电宝无？
017	下晡通常會落陣雨，出門記得帶雨傘，免著濕
018	汝看那间店排队排恁长，味道一定好，咱袂是也去凑热闹？
019	伊刚才突然肚痛真厉害，咱紧送佗去老君处看老君
020	我头痛，老君几点来？
021	我 suka 食炒粿条！
022	阿婆，起来食药了！
023	我打电话给你囡仔好无？
024	你佗位有痛无？
025	汝食飽未？

Compared Systems

System	Type	Description
MERaLiON-OmniVoice-Hokkien-TTS	Local	Our fine-tuned model (this card)
OmniVoice Original	Local	Base OmniVoice without Hokkien fine-tuning
CosyVoice3 0.5B	Local	Alibaba's CosyVoice3 with zero-shot voice cloning
Qwen3 TTS	Local	Alibaba's Qwen3-based TTS
Qwen3 TTS Flash	Cloud API	Alibaba Cloud's Qwen3 TTS Flash endpoint
CosyVoice	Cloud API	Alibaba Cloud's CosyVoice endpoint

All local models used the same Hokkien reference audio for voice cloning where supported.

Metrics

1. Language Identification (LangID)

What: An audio-based classifier determines whether each generated utterance sounds like Hokkien or Mandarin.

Why: The core challenge for Hokkien TTS is that most systems default to Mandarin pronunciation when reading Chinese characters. A model that produces Mandarin speech — regardless of how accurate or natural it sounds — has fundamentally failed at the task. LangID is therefore a gate metric: it tells us whether the model is even attempting the right language.

How: Qwen 3.5 Omni Plus listens to each audio clip and classifies it as Hokkien or Mandarin based on pronunciation and tonal patterns (not text content). The prompt explicitly instructs the model to focus on phonetic features and to treat Singapore Hokkien loanwords as Hokkien.

We chose Qwen 3.5 Omni Plus over Gemini 3.1 Pro because the former performed better on Hokkien ASR based on our tests. This also influenced our choice of Qwen 3.5 Omni Plus for WER and Naturalness.

2. Word Error Rate (WER)

What: Character-level WER between the original text and an ASR transcription of the generated audio.

Why: WER measures whether the model is saying the right words. A low WER means the generated speech faithfully reproduces the input text. We compute character-level WER (each Chinese character is a token) because Hokkien text has no standard word segmentation.

How: We use two independent ASR systems for robustness:

• Qwen 3.5 Omni Plus — prompted as a Hokkien transcription expert. Samples classified as Mandarin by LangID are penalized to WER=1.0 (since producing the wrong language is a total failure regardless of word accuracy).
• MERaLiON-3-3B-ASR — a local multilingual speech model. We report both raw scores and scores with the same LangID penalty applied.

3. Naturalness (MOS-like)

What: A 1–10 rating of how natural and human-like each utterance sounds.

Why: A model can pronounce words correctly but still sound robotic, choppy, or unnatural. Naturalness captures the perceptual quality — prosody, rhythm, fluency, voice quality, and absence of artifacts — that determines whether the speech is pleasant to listen to.

How: Qwen 3.5 Omni Plus rates each clip on a 1–10 scale, evaluating prosody, fluency, voice quality, pronunciation correctness, and artifacts. The evaluator is prompted with expertise in Hokkien and Chinese dialects.

4. DNSMOS P.835

What: An objective signal-quality score (1–5 scale) predicted by Microsoft's DNSMOS model.

Why: DNSMOS provides a language-agnostic signal quality assessment. It measures speech distortion (SIG), background noise (BAK), and overall quality (OVRL) without any dependence on language understanding. This complements the LLM-based metrics by catching low-level audio quality issues (clipping, noise, codec artifacts) that a language model might overlook.

How: Microsoft's pre-trained ONNX model (P.835 framework) runs locally on each audio file. No API calls or language-specific configuration needed.

Results

Summary Table

System	LangID (↑)	WER-Qwen (↓)	WER-MERaLiON (↓)	Naturalness (↑)	DNSMOS (↑)
MERaLiON-OmniVoice-Hokkien-TTS	24/25	0.33	0.31	8.40	3.13
OmniVoice Original	17/25	0.61	0.64	3.00	3.28
CosyVoice3 0.5B	18/25	0.87	0.86	1.76	2.40
Qwen3 TTS	7/25	0.82	0.81	2.40	3.11
Qwen3 TTS Flash	25/25	0.40	0.44	8.12	3.07
CosyVoice (Cloud)	15/25	0.70	0.60	2.08	3.29

WER-Qwen and WER-MERaLiON both include the LangID penalty (Mandarin → WER=1.0). DNSMOS is the Overall (OVRL) score on the P.835 scale (1–5).

Key Findings

MERaLiON-OmniVoice-Hokkien-TTS achieves the best overall balance across all metrics. It is a local model with near-perfect language identification (24/25 Hokkien), the lowest WER (0.33/0.31), the highest naturalness (8.40), and competitive DNSMOS (3.13). It performs well across both language-aware and language-agnostic evaluation dimensions.

Most general-purpose TTS systems fail at Hokkien. Qwen3 TTS produces Mandarin 72% of the time (7/25 Hokkien). CosyVoice3 and CosyVoice Cloud also frequently default to Mandarin pronunciation. These models achieve high DNSMOS scores because the audio is clean — but they are speaking the wrong language.

Qwen3 TTS Flash is a strong cloud baseline. It achieves 25/25 Hokkien identification, 0.40 WER, 8.12 naturalness, and 3.07 DNSMOS. However, it requires API access and is not available for local deployment.

The Qwen naturalness evaluator is language-aware. Unlike Gemini Flash (which rated all models 8+), Qwen 3.5 Omni Plus assigns low naturalness scores (1.76–3.00) to models that produce Mandarin instead of Hokkien. This suggests it evaluates pronunciation correctness as part of naturalness, making it a more discriminating evaluator for dialect TTS.

Two independent ASR systems agree on rankings. The Qwen-based and MERaLiON-based WER scores produce consistent rankings when the LangID penalty is applied, lending confidence to the results despite the absence of a dedicated Hokkien ASR ground truth.

Detailed DNSMOS Breakdown

System	Overall (↑)	Signal (↑)	Background (↑)
MERaLiON-OmniVoice-Hokkien-TTS	3.13	3.39	4.04
OmniVoice Original	3.28	3.51	4.13
CosyVoice3 0.5B	2.40	2.96	3.23
Qwen3 TTS	3.11	3.43	4.01
Qwen3 TTS Flash	3.07	3.37	3.97
CosyVoice (Cloud)	3.29	3.53	4.09

Evaluation Limitations

• No human evaluation. All subjective metrics (LangID, WER, naturalness) rely on LLM judges or ASR models, which may have systematic biases.
• Small test set. 25 sentences provide directional signal but are not sufficient for statistical significance claims.
• No dedicated Hokkien ASR. Both WER evaluators (Qwen, MERaLiON) were not specifically trained for Hokkien transcription, so absolute WER values should be interpreted as relative comparisons rather than ground truth error rates.
• Single-speaker evaluation. All voice-cloned models use the same reference speaker. Results may vary with different reference voices.

Citation

If you use this model or evaluation framework, please cite and attribute this work by linking back to this repository:

@misc{omnivoice-hokkien-2025,
  title={MERaLiON-OmniVoice-Hokkien-TTS: Fine-tuned Text-to-Speech for Singapore Hokkien},
  author={Lewis Won, MERaLiON Team},
  year={2026},
  url={https://huggingface.co/MERaLiON/MERaLiON-OmniVoice-Hokkien-TTS}
}

Model Files

This repository contains the complete model — no additional downloads required:

├── config.json                      # Model architecture config
├── model.safetensors                # Fine-tuned LLM weights (1.9 GB)
├── tokenizer.json                   # Text tokenizer
├── tokenizer_config.json
├── chat_template.jinja
└── audio_tokenizer/                 # Bundled HiGGS Audio v2 tokenizer (769 MB)
    ├── config.json
    ├── model.safetensors
    └── preprocessor_config.json

Total size: ~2.7 GB.

License

Please refer to the MERaLiON-3 Public Licence for model terms.