Hailay/movoc-mt-en-am-ti

MoVoC_MT: From-Scratch MarianMT for English-Amharic-Tigrinya, Zero-Shot Tigre

Downstream MT evaluation of MoVoC_Tok (a 120k-vocab shared Ge'ez-script + English SentencePiece Unigram tokenizer, built for the MoVoC project, arXiv:2509.08812). A from-scratch MarianMT model -- matching the paper's own architecture exactly (6+6 layers, 8 heads, d_model=512, ffn=2048, Swish activation, shared embeddings, static position embeddings) -- trained bidirectionally on English-Amharic and English-Tigrinya, then evaluated zero-shot on English-Tigre (a third Ge'ez-script language never seen during training).

Code: github.com/hailaykidu/MoVoC_MT

Training

• From-scratch (random init), NOT a fine-tune of any pretrained checkpoint -- no pretrained MarianMT model has a MoVoC_Tok-compatible (120k) vocabulary
• 106,104,832 parameters
• Bidirectional: >>amh<< / >>tir<< / >>eng<< direction tags
• 4,437,752 examples/epoch (2,157,134 Amharic-derived + 2,280,618 Tigrinya-derived), 3 epochs, 416,040 total steps
• Batch size 32, peak LR 5e-05, linear decay, 500 warmup steps, fp16
• Real runtime: 7h 25m 41s (SLURM job 52623, COMPLETED); throughput 502.8 samples/sec
• Per-epoch training loss: 3.6062 (epoch 1) -> 3.2935 (epoch 2) -> 3.1292 (epoch 3)
• Gradient-norm range: 1.33-5.61 (finite steps); 2 isolated inf events (fp16 loss-scaler overflow, confirmed benign -- loss/eval continued improving smoothly around both)

Results (real, not adjusted)

Amharic / Tigrinya (2,000 held-out dev pairs each, beam search num_beams=4)

Direction	BLEU	chrF
en -> am	24.699	33.655
am -> en	20.485	45.554
en -> ti	20.556	18.634
ti -> en	10.571	31.945

Tigre, zero-shot (43 of 45 Tatoeba pairs; Tigre never in training data)

Direction	BLEU	chrF
en -> tig	11.713	19.405
tig -> en	17.628	32.187

X->English is consistently stronger than English->X in every language pair tested, including zero-shot Tigre. tig->en zero-shot transfer is genuinely reasonable (e.g. a source meaning "Technology has failed to ease the conflict between man and nature" produces "Technology can't stop the conflict between humans and nature" -- capturing the real meaning despite zero Tigre training exposure). en->tig, by contrast, shows real repetition-loop degeneration under beam search (e.g. "his stomach is weak" produces a single Ge'ez word looping eight times) -- reported as observed, not smoothed over. 43-45 pairs is a small qualitative check, not a statistically robust benchmark; no larger English-Tigre parallel dataset exists anywhere to draw a stronger conclusion from.

Usage

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model = AutoModelForSeq2SeqLM.from_pretrained("Hailay/movoc-mt-en-am-ti")
tokenizer = AutoTokenizer.from_pretrained("Hailay/movoc-mt-en-am-ti")

text = ">>amh<< The weather is nice today."
inputs = tokenizer(text, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=64, num_beams=4)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Direction tags: >>amh<< (target Amharic), >>tir<< (target Tigrinya -- also used for zero-shot Tigre, the closest trained relative), >>eng<< (target English).

Reproducibility

Partially reproducible. Data splitting/shuffling is seeded (SEED=42) and deterministic, but exact metric recovery is not guaranteed:

• Model weight initialization is not seeded -- the reported numbers correspond to one unrepeatable random init.
• Training uses fp16 with no forced-determinism flags (cuDNN algorithm selection / reduction order can vary run to run).
• No dependency lockfile is published; results were produced with transformers==4.57.6, torch==2.9.0+cu128, datasets==5.0.0, tokenizers==0.22.2, sacrebleu==2.6.0.
• The tokenizer and training corpora are referenced from external, unversioned paths, not vendored into the training repo.

Full details and code: github.com/hailaykidu/MoVoC_MT.

Limitations

• Amharic corpus capped at 1.5M raw NLLB lines (of 16.1M available), to balance against Tigrinya's corpus size.
• Batch size (32) deviates from the paper's original run (8) for wall-clock reasons on this larger, bidirectional, multi-pair corpus.
• Tigre zero-shot eval is only 43-45 pairs -- a small qualitative check, not a robust benchmark.
• En->X translation is weaker than X->En for every pair tested, and en->tig zero-shot shows real repetition-loop degeneration.

Installation

pip install transformers torch sentencepiece

Citation

If you use this artifact, please cite:

@inproceedings{teklehaymanot2025movoc,
  title     = {MoVoC: Morphology-Aware Subword Construction for Ge'ez Script Languages},
  author    = {Teklehaymanot, Hailay Kidu and Fazlija, Dren and Nejdl, Wolfgang},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2025},
  year      = {2025},
  url       = {https://arxiv.org/abs/2509.08812}
}