Publish NLLB SPM8k private no-Bible v3 best checkpoint (20260809-210523)

7fda2a5 verified 1 day ago

5.88 kB

license: cc-by-nc-4.0
library_name: transformers
pipeline_tag: translation
base_model: facebook/nllb-200-distilled-600M
language:
  - ami
  - bnn
  - ckv
  - dru
  - pwn
  - pyu
  - ssf
  - sxr
  - szy
  - tao
  - tay
  - trv
  - tsu
  - xnb
  - xsy
  - en
tags:
  - translation
  - nllb-200
  - formosan-languages
  - low-resource
metrics:
  - bleu
  - chrf
  - ter
model-index:
  - name: nllb200-formosan-en-spm8k
    results:
      - task:
          type: translation
          name: Translation
        dataset:
          name: FormosanBank private no-Bible hard test
          type: private-no-bible-hard-test
          split: test
        metrics:
          - type: bleu
            name: sacreBLEU
            value: 9.324037
          - type: chrf
            name: chrF2
            value: 26.92985
          - type: ter
            name: TER
            value: 96.958093

nllb200-formosan-en-spm8k

Direction: Formosan to English
Base model: facebook/nllb-200-distilled-600M
Recipe: nllb200-spm8k-directional-v3
Release: 20260809-210523, validation-selected step 210,000

This is a directional model for 15 Formosan languages. It uses the private_no_bible leakage-controlled corpus, Formosan-aware 8k SentencePiece extension, balanced language/source sampling, and direction/domain/dialect control tags. The model weights are public, but the private training corpus is not included.

Model details

Item	Value
Base revision	`f8d333a098d19b4fd9a8b18f94170487ad3f821d`
Training rows	573,657
Effective batch size	64
Maximum sequence length	384
Learning rate	2e-05
Precision	`bf16`
Checkpoint selection	Human validation `chrF2`
Formosan text	`kindOf=standard`, `formosan-mt-standard-v3`
Corpus SHA-256	`e3feeaf7c3c51b9cd5c4b0537ffe44a370e7d02723f34b467479b6c4f5f0ea77`
Training profile SHA-256	`34f45832bdedc1b8322b26936dd88667dfecb360d941a7f54f60ae31a1fb4e10`

Usage

import torch
from transformers import AutoModelForSeq2SeqLM, NllbTokenizer
from formosan_mt_inference import normalize_formosan


model_id = "FormosanBank/nllb200-formosan-en-spm8k"
tokenizer = NllbTokenizer.from_pretrained(model_id, use_fast=False)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
model.to("cuda" if torch.cuda.is_available() else "cpu")
NLLB_LIDS = {'ami': 'ami_Latn', 'bnn': 'bnn_Latn', 'ckv': 'ckv_Latn', 'dru': 'dru_Latn', 'pwn': 'pwn_Latn', 'pyu': 'pyu_Latn', 'ssf': 'ssf_Latn', 'sxr': 'sxr_Latn', 'szy': 'szy_Latn', 'tao': 'tao_Latn', 'tay': 'tay_Latn', 'trv': 'trv_Latn', 'tsu': 'tsu_Latn', 'xnb': 'xnb_Latn', 'xsy': 'xsy_Latn'}

def translate(text, lang_code, source_bucket="unknown", dialect="default"):
    text = normalize_formosan(text, lang_code)
    tokenizer.src_lang = NLLB_LIDS[lang_code]
    prompt = (
        f"<to_eng> <src_{lang_code}> <dom_{source_bucket}> "
        f"<dialect_{dialect}> {text}"
    )
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    output = model.generate(
        **inputs,
        decoder_start_token_id=tokenizer.eos_token_id,
        forced_bos_token_id=tokenizer.convert_tokens_to_ids('eng_Latn'),
        max_new_tokens=256,
        num_beams=4,
    )
    return tokenizer.batch_decode(output, skip_special_tokens=True)[0]

print(translate("Pa'araw cingra.", "ami"))

The control tags are part of the training contract. Use unknown and default when source bucket or dialect metadata is unavailable.

Evaluation

The best checkpoint was selected on human validation chrF2. Test references are human sentence pairs; synthetic pivots and lexical entries are train-only. The headline result uses default metadata controls, so it does not assume access to test-set domain or dialect labels.

Split	Rows
Train	573,657
Test	63,140
Validate	20,733

Scope	BLEU	chrF2	TER
Hard test	9.32	26.93	96.96
Selection validation	9.53	29.70	92.68

Test empty-output rate: 0.0000%.

Confidence intervals

Stratified bootstrap, 200 samples, 95% confidence.

Metric	Lower	Upper
BLEU	9.18	9.48
chrF2	26.78	27.08
TER	96.48	97.54

Language	Samples	BLEU	chrF2	TER
`ami`	11,017	7.62	26.29	93.14
`bnn`	6,049	14.68	33.73	77.16
`ckv`	2,910	10.84	30.22	96.13
`dru`	5,846	1.07	13.93	135.78
`pwn`	5,811	7.65	24.46	106.90
`pyu`	4,315	12.50	29.72	107.12
`ssf`	1,630	14.05	31.21	87.89
`sxr`	2,081	11.33	26.27	96.93
`szy`	1,852	13.59	32.77	90.81
`tao`	2,257	9.04	27.97	93.45
`tay`	6,411	10.46	26.58	104.63
`trv`	6,502	12.81	31.71	96.50
`tsu`	2,003	2.39	19.11	95.33
`xnb`	2,682	5.62	25.89	93.22
`xsy`	1,774	16.10	35.31	83.12

The corpus gate enforces standard-tier Formosan text, at least 7.5% test and 2.5% validation per language, human sentence-only evaluation, and zero exact, skeleton, one-edit, configured high character n-gram, or document train/evaluation conflicts. This release passed all gates: exact 0, skeleton 0, one-edit 0, character n-gram 0, and document 0.

See eval/metrics.json for sacreBLEU signatures, per-language, source, dialect, and length diagnostics. publication.json records the corpus, profile, run, and checkpoint hashes used for this release.

Intended use

This model supports research, corpus development, and assisted translation for the 15 included Formosan languages. It is designed for the exact prompt and generation contract shown above.

Limitations

Outputs require knowledgeable speaker review. Aggregate metrics hide large differences among languages and domains. This model is not suitable for authoritative, medical, legal, or safety-critical translation.