ebible_m2o-nllb600m-ton

NLLB-200-distilled-600M fine-tuned to translate Bible verses into ton from 4 related source languages. Part of a series testing whether a pretrained multilingual model can draft Old Testament books for a language it has only seen the New Testament of — the practical "no OT exists yet" scenario. Project: reproduction and extension of Sami Liedes' 2018 closed-text Bible translation experiment.

Training used New Testament verses only (ton target, whole OT withheld); the scores below are on withheld OT material (Ruth, Jonah, Genesis 1) the model never saw in the target language. Target token ton_Latn (scratch init — the token-init comparison in the experiment showed the init method makes no measurable difference at an adequate learning rate).

How to use

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model = AutoModelForSeq2SeqLM.from_pretrained("DavidCBaines/ebible_m2o-nllb600m-ton")
tokenizer = AutoTokenizer.from_pretrained("DavidCBaines/ebible_m2o-nllb600m-ton")

tokenizer.src_lang = "mri_Latn"  # any source language's FLORES code
batch = tokenizer(["<a mri verse>"], return_tensors="pt")
out = model.generate(
    **batch,
    forced_bos_token_id=tokenizer.convert_tokens_to_ids("ton_Latn"),
    num_beams=5, max_length=128,
)
print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])

The forced_bos_token_id is also baked into generation_config.json, so omitting it works too.

Evaluation (withheld OT books, never seen in ton)

chrF3, sacreBLEU conventions. The source-copy floor is the chrF3 of the best source language's own text against the ton reference — the score you would get by simply copying the closest relative. The model must beat it for the run to be publishable. Validation (250 NT verses): best chrF3 59.00.

book	verses	best source	chrF3	spBLEU	mean chrF3 over sources	source-copy floor
RUT	85	mri	46.59	32.67	45.58	20.21
JON	48	tgl	46.78	33.61	45.66	18.74
GEN 1:	31	ind	48.5	33.31	47.07	18.57

Per-source detail:

• RUT: chrF3 by source — ceb 45.17, ind 44.73, mri 46.59, tgl 45.84
• JON: chrF3 by source — ceb 44.79, ind 45.29, mri 45.78, tgl 46.78
• GEN 1:: chrF3 by source — ceb 46.65, ind 48.5, mri 47.55, tgl 45.6

Training data and licensing

Fine-tuned on eBible-corpus translations. The base model (NLLB-200) is CC-BY-NC-4.0, so this model is non-commercial regardless of the data licences below; ShareAlike sources additionally propagate SA. Released under cc-by-nc-sa-4.0.

translation	language	FLORES code	licence
ton	ton (target)	—	Public Domain
mri2012	mri	mri_Latn	by-nc
indayt	ind	ind_Latn	by-nc
cebulb	ceb	ceb_Latn	by-sa
tglulb	tgl	tgl_Latn	by-sa

Reproducibility

• Experiment: m2o_ton, init scratch, lr 0.0003, max 8000 steps, generation-based early stopping (chrF3 on a fixed 250-verse NT set, patience 3, min-delta 0.2).
• Git commit: 423a6b234f489c111f6d44f0e45b53d621d8d720
• Code, configs and the full 15-run comparison: the project repository and the companion results dataset.

Acknowledgement

This series of experiments is inspired by the closed-text Bible translation experiment described by Sami Liedes in his blog. The repo was created with Claude Code using Opus 4.8 and Fable 5 with no code written by hand. There may well be errors present, but I trust Claude to do a better job than I can. Please let me know of any issues.