Hugging Face's logo

formospeech
/
omnivoice-hakka-community-1

Text-to-Speech
Safetensors
Hakka Chinese
omnivoice
zero-shot
multilingual
voice-cloning
voice-design
taiwanese-hakka
hakka
Eval Results (legacy)
Model card Files
xet
 Community

You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

This model is released under the Creative Commons Attribution-NonCommercial 4.0 International License, also known as CC BY-NC 4.0.

Access is limited to non-commercial use. Users must provide appropriate attribution when sharing or adapting the model where required by the license. Commercial use, including use in paid products, paid services, or commercial deployment, is not permitted under this license.

Log in or Sign Up to review the conditions and access this model content.

OmniVoice Hakka Community 1

formospeech/omnivoice-hakka-community-1 is a Taiwanese Hakka fine-tuned checkpoint based on k2-fsa/OmniVoice.

Summary

  • Base model: k2-fsa/OmniVoice
  • Model type: OmniVoice
  • Primary use case: Taiwanese Hakka text-to-speech
  • Released checkpoint: trained for 16000 steps
  • Supported generation modes:
    • zero-shot voice cloning with ref_audio + ref_text
    • voice design with instruct

Usage

To get started, install the Hakka-enabled OmniVoice fork:

  • FormoSpeech/OmniVoice-hakka

This fork is required because the upstream k2-fsa/OmniVoice runtime does not support Hakka dialect labels such as ๅฎข่ชžๅ››็ธฃ่…” in instruct.

For this Hakka checkpoint, the target dialect must be specified through instruct if you want to control the generated Hakka dialect. This differs from the base OmniVoice usage because Hakka dialect prompting is part of the intended inference setup for this release.

Access and Authentication

This model is hosted as a gated Hugging Face repository. Before using it:

  1. Visit the model page and request access.
  2. Log in with the same Hugging Face account that has been granted access.
  3. Authenticate your local environment with a Hugging Face access token.

A read token is sufficient for inference.

pip install -U huggingface_hub
hf auth login

Alternatively, you can provide the token through the HF_TOKEN environment variable:

export HF_TOKEN=hf_xxx

Do not hard-code your Hugging Face token in scripts, notebooks, or public repositories.

If you see an error such as Cannot access gated repo, make sure that:

  • your Hugging Face account has been granted access to this model;
  • hf auth whoami shows the expected account;
  • HF_HUB_DISABLE_IMPLICIT_TOKEN is not set.

Installation

Step 1: Install PyTorch

NVIDIA GPU 
# Install pytorch with your CUDA version, e.g.
pip install torch==2.8.0+cu128 torchaudio==2.8.0+cu128 --extra-index-url https://download.pytorch.org/whl/cu128

See PyTorch official site for other versions installation.

Apple Silicon 
pip install torch==2.8.0 torchaudio==2.8.0

Step 2: Install OmniVoice

pip install git+https://github.com/FormoSpeech/OmniVoice-hakka.git

Python API 

import soundfile as sf
import torch
from omnivoice import OmniVoice

model = OmniVoice.from_pretrained(
    "formospeech/omnivoice-hakka-community-1",
    device_map="cuda:0",
    dtype=torch.float16,
)

audio = model.generate(
    text="ๅฎข่ชž่ชž้Ÿณๅˆๆˆๆธฌ่ฉฆใ€‚",
    ref_audio="ref.wav",
    ref_text="้€™ไฟ‚ๅƒ่€ƒ่ชž้Ÿณใ€‚",
    instruct="ๅฎข่ชžๅ››็ธฃ่…”",
)

sf.write("out.wav", audio[0], 24000)

Supported Hakka Dialect Labels

Supported Hakka dialect labels in this project are:

  • ๅฎข่ชžๅ››็ธฃ่…”
  • ๅฎข่ชžๆตท้™ธ่…”
  • ๅฎข่ชžๅคงๅŸ”่…”
  • ๅฎข่ชž้ฅ’ๅนณ่…”
  • ๅฎข่ชž่ฉ”ๅฎ‰่…”
  • ๅฎข่ชžๅ—ๅ››็ธฃ่…”

Training Data

Training uses a mixed FormoSpeech Hakka setup.

Original datasets:

  • formospeech/hakkaradio_news_clean
  • formospeech/hat_tts_hailu_clean
  • formospeech/hat_tts_sixian_clean
  • formospeech/hakka_elearning_example_clean

Denoised -R datasets:

  • formospeech/hat_asr_sixian_reading_clean_r
  • formospeech/hat_asr_hailu_reading_clean_r
  • formospeech/hat_asr_nansixian_reading_clean_r
  • formospeech/hat_asr_sixian_broadcast_clean_r

Text Processing

Training uses cleaned hanzi_cln / pinyin_cln text with a restored-punctuation pipeline.

The restored pipeline applies the following steps:

  1. Samples containing <UNK> or <spn> are filtered out.
  2. <SIL> / <sil> markers are removed from the cleaned text.
  3. The raw text is split into text units and the punctuation attached to each unit is recorded.
    • For Hanzi, the units are individual Chinese characters plus inline ASCII word spans.
    • For pinyin, the units are whitespace-delimited pinyin tokens.
  4. The cleaned text is converted into the same kind of units after removing silence markers.
  5. The raw-unit sequence and cleaned-unit sequence are aligned with partial matching.
  6. Punctuation is restored for the parts of the cleaned text that still align with the raw text.
    • Small insertions, deletions, or replacements do not prevent punctuation recovery for surrounding matched regions.
    • Punctuation is copied back only for aligned regions and is not forced into mismatched regions.
  7. After local punctuation transfer, the text is normalized:
    • leading punctuation is removed
    • missing or weak sentence-final punctuation is normalized
    • in text_pinyin, full-width punctuation is kept and extra spaces around punctuation are removed

In practice, this means the pipeline is conservative for local mismatches while still recovering punctuation around them. Short or fragmentary utterances can still be normalized into sentence-final ใ€‚.

Training Config

This release was trained with:

  • cleaned cln restored manifests
  • instruct_ratio = 1.0
  • only_instruct_ratio = 0.3
  • use_pinyin_ratio = 0.3
  • total training steps: 16000

Evaluation

Evaluation uses a custom Taiwanese Hakka test list built per-dialect from the test split of formospeech/hakkaradio_news_clean (commit e9e8762), configs Hakka_Sixian and Hakka_Hailu.

Text cleaning (hanzi_cln):

  • rows containing <UNK> or <spn> are dropped
  • <SIL> / <sil> markers are removed
  • remaining words are joined with no inter-word space

Construction rules:

  • target utterances: duration in 3-30 seconds
  • reference utterances: duration in 3-15 seconds
  • ref_audio comes from the same speaker as the target, ref_id != id
  • references are assigned round-robin per speaker to maximize diversity

Evaluation metrics and models:

  • CER: formospeech/whisper-large-v2-taiwanese-hakka-v1, served via a vLLM OpenAI-compatible /v1/audio/transcriptions endpoint
  • SIM-o: WavLM-based speaker verification model from k2-fsa/TTS_eval_models
  • UTMOS: UTMOS predictor from k2-fsa/TTS_eval_models

CER is computed on NFKC-normalized, punctuation-stripped, whitespace-stripped text, matching between the ASR hypothesis and the ground-truth text field.

Results (per dialect)

Hakka Sixian (ๅฎข่ชžๅ››็ธฃ่…”)

Model CER avg CER weighted SIM-o UTMOS
Original target audio (ground truth) 5.24% 4.44% 0.771 3.12
OmniVoice Taiwanese Hakka (this release) 2.03% 1.85% 0.795 3.38
F5-TTS g2p 11.08% 11.89% 0.807 2.96
F5-TTS gt_pinyin 10.08% 11.31% 0.808 2.95

Hakka Hailu (ๅฎข่ชžๆตท้™ธ่…”)

Model CER avg CER weighted SIM-o UTMOS
Original target audio (ground truth) 3.03% 2.83% 0.786 3.67
OmniVoice Taiwanese Hakka (this release) 2.44% 2.35% 0.807 3.77
F5-TTS g2p 6.42% 6.26% 0.791 3.36
F5-TTS gt_pinyin 5.91% 5.76% 0.792 3.35

F5-TTS numbers are from formospeech/f5-tts-hita-finetune-v1 on the same per-dialect test lists:

  • g2p: inference converts Hanzi to pinyin with formog2p.hakka.g2p
  • gt_pinyin: inference uses the ground-truth pinyin annotation from the source dataset

The "Original target audio" rows evaluate the real human recordings themselves (CER against the ASR model, SIM-o against ref_audio, UTMOS on the recording) as a sanity-check reference point, not a system being benchmarked.

Two observations worth calling out:

  • OmniVoice's CER is lower than the ground-truth recordings' own CER on both dialects. This does not mean OmniVoice is more intelligible than real speech โ€” it means the ASR model finds OmniVoice's cleaner, less variable output easier to transcribe than real recordings, which carry background noise and natural speech variation the ASR model wasn't perfectly tuned for.
  • OmniVoice is noticeably stronger on Sixian than Hailu on CER, while F5-TTS shows the opposite pattern (much worse on Sixian than Hailu). The two systems do not degrade the same way across dialects.

Reproducing this evaluation

The per-dialect test lists (id, text, text_pinyin, ref_id, ref_audio, ref_text, ref_text_pinyin, instruct, language_id, language_name, duration) are published alongside this model under eval/:

ref_audio paths in these files are relative (e.g. ref_audio/Hakka_Sixian/<id>.wav) and are not bundled with this repo. To reproduce the exact same test lists, including the reference audio, run the build script published in the FormoSpeech/OmniVoice-hakka repo:

python tools/build_omnivoice_hakka_community1_eval_list.py --out-root <your-output-dir>

The script pins formospeech/hakkaradio_news_clean to commit e9e8762e4200c11781e9108924f832c4d46477aa and uses a fixed seed (42), so re-running it reproduces byte-identical test lists (both jsonl and audio) independent of later changes to the source dataset.

Notes

  • This release is optimized for Taiwanese Hakka TTS and keeps the general OmniVoice interface from the base model.

  • The original OmniVoice features for Voice Design and Fine-grained Control may be weaker in this checkpoint. The finetuning data used for this release does not provide the full annotation coverage needed to preserve those capabilities. This is especially true for Voice Design, because finetuning actively uses instruct, so the Hakka-dialect prompting setup can interfere more strongly with the broader attribute-control behavior from the base model.

  • Evaluation numbers reported here come from the project-side Hakka benchmark setup, not from a standardized public benchmark leaderboard.

Downloads last month
93
Safetensors
Model size
0.6B params
Tensor type
I64
ยท
F32
ยท
Inference Providers NEW
Text-to-Speech
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for formospeech/omnivoice-hakka-community-1

Base model

Qwen/Qwen3-0.6B-Base
Finetuned
Qwen/Qwen3-0.6B
Finetuned
k2-fsa/OmniVoice
Finetuned
(58)
this model

Spaces using formospeech/omnivoice-hakka-community-1 2

Evaluation results