You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

This model performs zero-shot voice cloning from a 3-5 second sample, so it can imitate a real person's voice from a short recording. Access is gated for that reason. By requesting access you confirm that you will not use it to impersonate any person, to create audio presented as a genuine recording of someone, or to clone a voice without that person's informed consent.

Nepali Pocket-TTS — 6-layer distilled student

A Nepali text-to-speech model with zero-shot voice cloning that runs ~5x faster than real time on CPU (4 threads, no GPU). It is a 6-layer depth-distilled student of a 24-layer Nepali teacher, and it outperforms that teacher on intelligibility while being 3.3x faster and 3x smaller.

	student (this model)	teacher (24L)
backbone layers	6	24
parameters	109.5M (89.4M LM + 20.1M Mimi)	336.1M
weights on disk	438 MB	1.34 GB
CPU speed	5.04x real-time	1.53x real-time
WER / CER (held-out, Nepali ASR)	0.322 / 0.139	0.402 / 0.177

Quickstart

pip install pocket-tts
huggingface-cli login     # required: this repo is gated, so hf:// downloads need auth

Access must be granted before the code below can fetch anything — request it with the form at the top of this page, then log in with a token from the same account. Without that, the hf:// paths return HTTP 401.

from pocket_tts.models.tts_model import TTSModel

# config.yaml self-resolves: it fetches this repo's weights and tokenizer via hf://
model = TTSModel.load_model(config="hf://himalaya-ai/pocket-tts-nepali-6l/config.yaml")
model.to("cpu")

# Any 3-5 s mono clip of the voice you want. You must supply this yourself --
# no reference audio ships with this repo (see "Voice prompts" below).
state = model.get_state_for_audio_prompt("my_voice.wav")

audio = model.generate_audio(state, "नेपाल हिमाल, पहाड र तराई गरी तीन भौगोलिक क्षेत्रमा विभाजित छ।")

import scipy.io.wavfile as wav
wav.write("out.wav", model.config.mimi.sample_rate, audio.detach().cpu().numpy().squeeze())

A runnable version with CPU thread pinning and timing is in inference.py.

Pin your thread count. torch.set_num_threads(4) is in the example on purpose. Oversubscribing threads on a loaded machine erased the entire 3.3x speed difference between this model and its teacher in our first benchmark — both measured ~0.16x real-time until the count was pinned. If you measure this model as slower than real time, check thread contention before anything else.

Model details

Architecture — Kyutai Pocket-TTS: a flow-matching LM over Mimi latents. 6-layer transformer backbone (d_model 1024, 16 heads), 6-layer/512-dim flow head, Mimi codec at 12.5 Hz frame rate / 24 kHz output. Single-step LSD flow decode.
Text frontend — Nepali SentencePiece BPE, 4000 tokens, trained on this corpus. The text embedding was reinitialized from scratch for Nepali; the English tokenizer's vocabulary contains no Devanagari.
Languages — Nepali (Devanagari) only. It has no English capability: the text embedding was reset and it saw no English during training.
Sample rate — 24 kHz mono.
Precision — float32.

How this was trained

Two stages, 200k steps each, batch_size 64 on one H100.

Teacher (24L) — kyutai/pocket-tts English 24-layer checkpoint, fine-tuned on Nepali with a fresh text embedding and the Nepali BPE-4000 tokenizer. lr 2e-4 cosine.
Student (6L, this model) — trained from scratch (no pretrained init) to match the teacher's backbone activations. Same d_model, so the flow head and all non-backbone weights are copied from the teacher and the flow head stays frozen. lr 4e-4 cosine, 1000 warmup steps, EMA 0.9999. Final distillation MSE 0.0118 on the validation split.

Why the student beats its teacher. Distillation targets were computed with classifier-free guidance at coefficient 2.0, so the student learned the teacher's guided output distribution. Guidance is not implemented anywhere in the Pocket-TTS inference path, so the teacher can only ever be sampled unguided — the student has guidance baked into its weights and needs no second forward pass to get it. The teacher is now only a distillation source; there is no reason to deploy it.

Training data

Roughly 2,000 hours of Nepali speech after quality gating (about 750,000 utterances). The bulk is YouTube-derived audio; the remainder comes from open-source Nepali speech datasets.

Quality gating rather than raw duration decided what was kept: clips were retained only where three independent ASR systems agreed on the transcript, clips with more than one talker were removed, and noise, SNR and DNSMOS thresholds were applied. Every kept utterance is word-aligned.

The distribution is dominated by spontaneous, in-the-wild speech rather than studio read speech, which is why it clones conversational voices more convincingly than polished narration.

The training corpus is not available for download

Most of it is not ours to redistribute, and the openly-licensed portion is already available from its original sources — so there is nothing to gain from us re-hosting it. Please do not request the corpus.

Evaluation

100 held-out utterances, 77 distinct speaker identities, spanning all six kinds of speech in the corpus. The target utterance is strictly held out; the voice prompt is a different clip of the same speaker taken from the train split, since a prompt is conditioning rather than a target.

Speaker identity is keyed on (directory, diarization label) — a bare SPEAKER_01 label is per-video, so two videos' SPEAKER_01 are different people, and keying on the label alone silently pairs unrelated voices.

real_human is the genuine recording of the same utterance, scored through the identical pipeline. It is a ceiling and a sanity check on the metric, not a competitor.

Intelligibility

Scored with a Nepali-fine-tuned Whisper large-v3 (WER 10.98% on its own clean test split), text NFC-normalized and punctuation-stripped:

system	n	WER	CER
real human (ceiling)	100	0.343	0.122
teacher 24L	100	0.402	0.177
student 6L	100	0.322	0.139

Per-source CER, showing the student is ahead of the teacher on every source independently rather than on one source's average:

speech type	real human	teacher 24L	student 6L
spontaneous, SNR 40-50 dB	0.382	0.389	0.387
spontaneous, SNR > 50 dB	0.334	0.383	0.352
read speech	0.380	0.480	0.397
read speech, long utterances	0.364	0.449	0.367
spontaneous, curated	0.343	0.420	0.380
acted, studio	0.257	0.286	0.275

Do not read "the student's WER is below the human's" as beating human quality. The held-out human audio is largely spontaneous speech with disfluencies scored against a reference transcript, and synthetic read speech is simply easier for an ASR. The honest conclusion is that WER has saturated as a discriminator at this quality level — not that the model is superhuman.

Why the ASR choice matters

The same 100 clips, scored with off-the-shelf whisper-large-v3-turbo (language="ne"):

system	n	WER	CER
real human	98	0.917	0.344
teacher 24L	95	0.991	0.402
student 6L	97	0.937	0.361

At 0.917 WER on genuine human speech, base Whisper cannot rank two TTS systems — the differences are inside the instrument's noise. It also silently romanizes a few percent of Nepali clips, producing CER ≈ 1.0 on audio that is fine; those rows are detected by a Devanagari-character ratio and discarded as broken measurements, which is why n varies. Any Nepali TTS evaluation should score real human audio as a control before trusting a WER delta.

Speaker similarity

Resemblyzer cosine between the generated audio and the voice prompt:

system	n	similarity
real human vs prompt	100	0.807
teacher 24L	100	0.826
student 6L	100	0.841

Both TTS systems score above genuine human cross-utterance similarity. That is an artifact: a clone sits unnaturally close to its own prompt, while two real recordings of one person differ in content, session and channel. Read this column as a teacher-vs-student comparison only, never as an absolute or as "more like the speaker than the speaker".

Speed

Three Nepali sentences, CPU only, torch.set_num_threads(4) on a 16-core machine:

system	real-time factor	AR loop	Mimi decode
teacher 24L	1.53x	39.9 ms/frame	7.5 ms/frame
student 6L	5.04x	12.7 ms/frame	8.1 ms/frame

The frame budget for real-time at 12.5 Hz is 80 ms. The backbone is 3.14x cheaper; Mimi is identical in both (same weights) and is only ~19% of the teacher's per-frame cost, so cutting the backbone pays almost in full.

Limitations

Nepali only. No English or code-switching ability; the text embedding was reset.
Numbers, dates, abbreviations and Latin-script tokens are not normalized. Expand them in the input text yourself.
Emotional control is not available. There is no style or emotion conditioning; prosody comes entirely from the voice prompt. Separate work on this corpus found that emotion is not linearly decodable from a related model's states for Nepali, and no emotion claim is made here.
Prompt quality dominates output quality. A noisy, reverberant or clipped prompt produces a noisy voice. 3-5 s of clean speech is the sweet spot.
Inherited codec ceiling. Mimi at 12.5 Hz caps achievable audio fidelity; this is a compact on-device model, not a studio-quality one.
Speaker coverage is skewed toward the YouTube-derived portion of the corpus.
Occasional generations end without EOS on very long inputs; feed sentence-sized chunks.

Voice prompts

No reference audio ships in this repo. The training corpus contains recordings of identifiable real people who did not consent to having their voices redistributed as cloning prompts, and shipping cloned samples of them would compound that. Supply your own prompt from a speaker who has agreed to it.

Intended use

Nepali speech synthesis on CPU — accessibility tooling, offline assistants, voice interfaces for low-resource deployments, and research on distillation for low-resource TTS.

Out of scope: impersonating real people, generating audio presented as a genuine recording, cloning any voice without that person's informed consent, and any use where synthetic speech is passed off as human. Access to this repo is gated because 3-5 seconds of audio is enough to clone a voice.

Attribution and license

Released under CC-BY-4.0, matching the upstream weights it derives from.

Architecture, training code and the English checkpoint the teacher was fine-tuned from: Kyutai Pocket-TTS (kyutai/pocket-tts, CC-BY-4.0). The Mimi codec weights in this repo are Kyutai's, unmodified.
Nepali speech data: YouTube-derived audio, which is the bulk of the corpus and is not redistributable, plus open-source Nepali speech datasets. Among those, ai4bharat/indicvoices_r and ai4bharat/Rasa are CC-BY-4.0 and are named here because that licence requires attribution. Please honour their terms and cite them if you build on this.

If you use this model, please cite Kyutai's Pocket-TTS alongside this repo.

Downloads last month: -

Safetensors

Model size

0.1B params

Tensor type

F32

Model tree for himalaya-ai/pocket-tts-nepali-6l

Base model

kyutai/pocket-tts

Finetuned

(20)

this model