shekar-ai/byt5-small-tajik-farsi-translit

ByT5 Persian ↔ Tajik Transliterator

Bidirectional transliteration between Persian (Perso-Arabic script) and Tajik (Cyrillic script), fine-tuned from google/byt5-small on the ParsText Persian–Tajik parallel corpus. ByT5 operates on raw UTF-8 bytes, so both scripts are handled natively without script-specific tokenization.

Direction is selected with a task prefix:

• fa2tg: … — Persian → Tajik (Cyrillic)
• tg2fa: … — Tajik → Persian (Perso-Arabic)

Architecture	ByT5-small (byte-level encoder–decoder)
Parameters	299 M
Base model	google/byt5-small
Training data	ParsText Persian–Tajik parallel corpus
License	MIT

Intended Use

• Persian ↔ Tajik script conversion for cross-script search and reading
• Script normalization ahead of Persian/Tajik NLP pipelines that assume a single script
• A starting point for fine-tuning on related low-resource script-conversion tasks

Usage

Shekar

Shekar wraps 8-bit quantized ONNX exports of the encoder and decoder, so it runs on CPU without PyTorch. FarsiToTajik and TajikToFarsi set the direction prefix for you.

from shekar import FarsiToTajik, TajikToFarsi

to_tajik = FarsiToTajik()
to_farsi = TajikToFarsi()

print(to_tajik("ایران مادر است!"))     # Эрон модар аст!
print(to_farsi("Донишгоҳи Теҳрон"))    # دانشگاه تهران

Both accept num_beams (default 1) and max_new_tokens (default 256).

Transformers

import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

repo = "shekar-ai/byt5-small-tajik-farsi-translit"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSeq2SeqLM.from_pretrained(repo).eval()

@torch.no_grad()
def transliterate(text, direction="fa2tg", num_beams=4):
    assert direction in {"fa2tg", "tg2fa"}
    inputs = tokenizer(f"{direction}: {text.strip()}", return_tensors="pt", truncation=True, max_length=1024)
    output = model.generate(**inputs, num_beams=num_beams, max_new_tokens=1024, early_stopping=True)
    return tokenizer.decode(output[0], skip_special_tokens=True).strip()

print(transliterate("دانشگاه تهران", "fa2tg"))     # Донишгоҳи Теҳрон
print(transliterate("Эрон модар аст!", "tg2fa"))   # ایران مادر است!

Training

Data. The ParsText Persian–Tajik parallel dataset. After cleaning (non-empty pairs, length ≤ 200 characters) and bidirectional expansion (one example per direction), the corpus contains 751,650 examples, split 90/5/5 into train/validation/test. Persian text was normalized with Shekar's preprocessing pipeline (AlphabetNormalizer → YaNormalizer → SpacingNormalizer).

Hyperparameter	Value
Epochs	5 (early-stopped on best chrF++)
Batch size	32 per device
Learning rate	5e-4
LR scheduler	Linear, 5% warmup
Weight decay	0.01
Precision	bf16 mixed
Beam search (eval)	4 beams
Max sequence length	1024 bytes

Evaluation

Held-out test set of 37,582 examples, scored with chrF++ (character n-gram F-score, word_order=2), CER (character error rate), and exact sequence match:

Direction	n	chrF++ ↑	CER ↓	Sequence accuracy ↑
Overall	37,582	89.68	0.0382	0.537
fa2tg (Persian → Tajik)	18,705	87.90	0.0467	0.423
tg2fa (Tajik → Persian)	18,877	91.76	0.0283	0.649

tg2fa scores higher because the Perso-Arabic script collapses several Tajik vowel distinctions, making the target more deterministic than in the reverse direction.

Limitations

Persian → Tajik is the harder direction: Persian orthography leaves short vowels unwritten, so the model must infer them, and exact-match accuracy is correspondingly lower (42% vs 65%). Training examples were capped at 200 characters — transliterate long text sentence by sentence. Proper nouns and loanwords absent from ParsText are the most common source of errors.

Citation

@article{Amirivojdan2025Shekar,
  author  = {Amirivojdan, Ahmad},
  title   = {{Shekar: A Python Toolkit for Persian Natural Language Processing}},
  journal = {Journal of Open Source Software},
  volume  = {10},
  number  = {114},
  pages   = {9128},
  year    = {2025},
  doi     = {10.21105/joss.09128},
  url     = {https://joss.theoj.org/papers/10.21105/joss.09128}
}