lazymonster/yobyt5-restoration

Yo-ByT5

A fine-tune of google/byt5-small that restores Yorùbá diacritics (tone marks: ◌́ high, ◌̀ low, unmarked mid; underdots: ẹ ọ ṣ) to undiacritised text. A byte-level model suits the task because the target is a byte-level phenomenon: a combining mark on a base character.

• Language: Yorùbá (yo) · Task: Automatic Diacritic Restoration
• Developed by: Gali Ahmad Samuel (lazymonster)

Intended uses & limitations

Preprocessing for TTS and MT, query normalisation, and corpus annotation. Ambiguous forms are resolved from context (owo → owó money / ọwọ́ hand / ọ̀wọ̀ honour). Byte sequences are long, so chunk inputs beyond ~1024 bytes. Training data is Yorùbá news and scripture, so conversational, social-media, and dialectal Yorùbá are out of distribution. Greedy decoding (num_beams=1) is a good default: the task is near-monotonic, so greedy is fast and accurate. Beam search (num_beams=5) is comparable in quality, but only on transformers<5 (verified on 4.57.6): transformers 5.0.0 has a beam-search regression that truncates this byte-level model's output.

Unicode

Yorùbá diacritics have multiple valid encodings that render identically but differ byte-for-byte. Normalise both hypothesis and reference to the same form (unicodedata.normalize("NFC", ...)) before scoring. Skipping this can shift CER by tens of points.

Training data

Source	Sentences	Licence
MENYO-20k, Yorùbá side, training portion	9,942	CC BY-NC 4.0
Biblica® Open Yorùbá Contemporary Bible 2017	~36,000	CC BY-SA
Total train	46,313
Validation	5,305
Internal test	6,633

The Bible text (Bíbélì Mímọ́ ní Èdè Yorùbá Òde-Òní, Biblica, Inc.) is segmented below verse level on punctuation, so it does not align sentence-for-sentence with other published Yorùbá Bible splits. JW300 was excluded because its Yorùbá diacritisation is inconsistent.

Contamination against YAD

Audited against YAD dev+test (the MENYO-20k test split). The only overlap is one boilerplate header line, itself duplicated within MENYO-20k's own train/test halves and therefore also present in YAD's official train split. The model can be evaluated on YAD test.

Training procedure

Fine-tuned from google/byt5-small on Google Cloud TPU v6e-8 in two phases. Both phases use per-device batch 4, gradient accumulation 2 (effective batch 64), AdamW, linear decay, weight decay 0.01, and gradient clipping 0.5. Phase 1 uses learning rate 2e-4 with 300 warmup steps. Phase 2, which produced the released checkpoint, uses learning rate 1e-4 with no warmup and runs 4 epochs. Frameworks: Transformers 4.53.3, PyTorch 2.6.0+cu124, torch_xla.

Evaluation results

Greedy decoding (num_beams=1), both sides normalised to the same Unicode form. Two test sets are reported and are not interchangeable: YAD is the public benchmark and the only set comparable to published work; the internal test set is reported for continuity with earlier versions of this card and is comparable to nothing else.

Each is scored against two references. Official is the reference as released, the basis of every published YAD number and the column to quote alongside prior work. Corrected repairs the underdot encoding described below on both sides.

The two columns answer different questions rather than one being a discount on the other. Under the official reference, DER counts only positions where the reference carries a mark the metric recognises, so the 891 underdot positions encoded as U+0329 are dropped from the denominator entirely and the model's output there is never examined. Correcting the reference returns those positions to the count. WER and CER can only improve under correction; DER and WDER may move in either direction, because correction enlarges their denominator, which raises the rate if the model handles the recovered positions worse than average.

YAD benchmark test set (3,330 sentences): benchmark-comparable

Metric	Official reference	Corrected reference
WER	16.11%	14.93%
CER	3.82%	3.55%
DER	13.36%	12.88%
DER (tone)	12.91%	12.91%
DER (underdot)	5.74%	5.77%
WDER	16.90%	15.66%
BLEU	0.6837	0.6837
ChrF	0.8431	0.8431
Word accuracy †	83.96%	n/a
Underdot accuracy †	92.36%	n/a

Internal test set (6,633 sentences): not benchmark-comparable

Metric	Official reference	Corrected reference
WER	16.36%	15.44%
CER	3.91%	3.69%
DER	14.93%	14.62%
DER (tone)	14.62%	14.62%
DER (underdot)	6.69%	6.87%
WDER	17.25%	16.27%
BLEU	0.6856	0.6856
ChrF	0.8383	0.8383
Word accuracy †	83.68%	n/a
Underdot accuracy †	92.40%	n/a

† Custom metrics carried over from earlier versions of this card. They perform no encoding repair, so they have an official-reference value only. BLEU and ChrF are computed on raw strings and so do not vary by reference.

All numbers were measured with transformers 4.57.6 and greedy decoding as described above; the YAD figures reproduced identically across three independent evaluation runs.

DER is diacritic-position error (wrong marks ÷ marks that should be present; a no-diacritic output scores 100%), the convention published YAD results use. The YAD reference encodes 1,415 of 138,798 positions (1.02%) with the wrong underdot codepoint (U+0329 for U+0323) or doubled underdots, which normalisation cannot fix.

DER (underdot) against the official reference does not measure those positions at all. The metric takes an underdot to be U+0323, so a reference position carrying U+0329 matches nothing it looks for and is skipped: 891 positions, 2.65% of the benchmark's real underdots, absent from the denominator. That column therefore reports the underdot error rate over the subset of underdots YAD happened to encode correctly. The corrected column measures all of them, and is the one to cite for any claim about underdot behaviour.

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import unicodedata

tok = AutoTokenizer.from_pretrained("lazymonster/yobyt5-restoration")
model = AutoModelForSeq2SeqLM.from_pretrained("lazymonster/yobyt5-restoration")

text = "Eko ni kokoro aseyori."
inputs = tok(text, return_tensors="pt", max_length=1024, truncation=True)
out = model.generate(inputs["input_ids"], max_length=1024, num_beams=1)  # greedy
print(tok.decode(out[0], skip_special_tokens=True))
# Ẹ̀kọ́ ni kọ́kọ́rọ́ àṣeyọrí.
# If scoring: hyp, ref = normalize("NFC", hyp), normalize("NFC", ref)

Licence

Weights are Apache 2.0, matching google/byt5-small. Corpus licences govern the training data and require attribution: MENYO-20k is CC BY-NC 4.0 (Adelani et al., 2021); the Biblica® Bíbélì Mímọ́ ní Èdè Yorùbá Òde-Òní (2017) is CC BY-SA (Biblica, Inc.).

Citation

@misc{gali2025yobyt5,
  author       = {Gali Ahmad, Samuel},
  title        = {Yo-ByT5: Byte-Level Diacritic Restoration for Yorùbá},
  year         = {2025},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/lazymonster/yobyt5-restoration}}
}

Acknowledgments

Trained as a member of the HausaNLP Research Group using Google TPU Research Cloud (TRC) compute.