Publish NLLB SPM8k private no-Bible v3 best checkpoint (20260809-210523)

052b723 verified 1 day ago

5.7 kB

license: cc-by-nc-4.0
library_name: transformers
pipeline_tag: translation
base_model: facebook/nllb-200-distilled-600M
language:
  - ami
  - bnn
  - ckv
  - dru
  - pwn
  - pyu
  - ssf
  - sxr
  - szy
  - tao
  - tay
  - trv
  - tsu
  - xnb
  - xsy
  - zh
tags:
  - translation
  - nllb-200
  - formosan-languages
  - low-resource
metrics:
  - bleu
  - chrf
  - ter
model-index:
  - name: nllb200-formosan-zh-spm8k
    results:
      - task:
          type: translation
          name: Translation
        dataset:
          name: FormosanBank private no-Bible hard test
          type: private-no-bible-hard-test
          split: test
        metrics:
          - type: bleu
            name: sacreBLEU
            value: 12.580739
          - type: chrf
            name: chrF2
            value: 12.964706
          - type: ter
            name: TER
            value: 82.854846

nllb200-formosan-zh-spm8k

Direction: Formosan to Traditional Chinese
Base model: facebook/nllb-200-distilled-600M
Recipe: nllb200-spm8k-directional-v3
Release: 20260809-210523, validation-selected step 220,000

This is a directional model for 15 Formosan languages. It uses the private_no_bible leakage-controlled corpus, Formosan-aware 8k SentencePiece extension, balanced language/source sampling, and direction/domain/dialect control tags. The model weights are public, but the private training corpus is not included.

Model details

Item	Value
Base revision	`f8d333a098d19b4fd9a8b18f94170487ad3f821d`
Training rows	702,520
Effective batch size	64
Maximum sequence length	384
Learning rate	2e-05
Precision	`bf16`
Checkpoint selection	Human validation `chrF2`
Formosan text	`kindOf=standard`, `formosan-mt-standard-v3`
Corpus SHA-256	`238b96ea87716a749391d7157ce99f1611c1d4a489c4c108c315553505ca1f29`
Training profile SHA-256	`34f45832bdedc1b8322b26936dd88667dfecb360d941a7f54f60ae31a1fb4e10`

Usage

import torch
from transformers import AutoModelForSeq2SeqLM, NllbTokenizer
from formosan_mt_inference import normalize_formosan


model_id = "FormosanBank/nllb200-formosan-zh-spm8k"
tokenizer = NllbTokenizer.from_pretrained(model_id, use_fast=False)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
model.to("cuda" if torch.cuda.is_available() else "cpu")
NLLB_LIDS = {'ami': 'ami_Latn', 'bnn': 'bnn_Latn', 'ckv': 'ckv_Latn', 'dru': 'dru_Latn', 'pwn': 'pwn_Latn', 'pyu': 'pyu_Latn', 'ssf': 'ssf_Latn', 'sxr': 'sxr_Latn', 'szy': 'szy_Latn', 'tao': 'tao_Latn', 'tay': 'tay_Latn', 'trv': 'trv_Latn', 'tsu': 'tsu_Latn', 'xnb': 'xnb_Latn', 'xsy': 'xsy_Latn'}

def translate(text, lang_code, source_bucket="unknown", dialect="default"):
    text = normalize_formosan(text, lang_code)
    tokenizer.src_lang = NLLB_LIDS[lang_code]
    prompt = (
        f"<to_zh> <src_{lang_code}> <dom_{source_bucket}> "
        f"<dialect_{dialect}> {text}"
    )
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    output = model.generate(
        **inputs,
        decoder_start_token_id=tokenizer.eos_token_id,
        forced_bos_token_id=tokenizer.convert_tokens_to_ids('zho_Hant'),
        max_new_tokens=256,
        num_beams=4,
    )
    return tokenizer.batch_decode(output, skip_special_tokens=True)[0]

print(translate("Pa'araw cingra.", "ami"))

The control tags are part of the training contract. Use unknown and default when source bucket or dialect metadata is unavailable.

Evaluation

The best checkpoint was selected on human validation chrF2. Test references are human sentence pairs; synthetic pivots and lexical entries are train-only. The headline result uses default metadata controls, so it does not assume access to test-set domain or dialect labels.

Split	Rows
Train	702,520
Test	64,356
Validate	21,450

Scope	BLEU	chrF2	TER
Hard test	12.58	12.96	82.85
Selection validation	9.41	11.26	90.38

Test empty-output rate: 0.0140%.

Language	Samples	BLEU	chrF2	TER
`ami`	11,271	13.19	13.38	80.51
`bnn`	6,112	11.10	11.85	88.68
`ckv`	3,038	15.17	14.81	76.72
`dru`	6,109	5.94	7.04	102.81
`pwn`	5,718	8.52	9.67	93.46
`pyu`	4,280	19.80	18.77	73.94
`ssf`	1,673	15.88	17.04	70.99
`sxr`	1,781	13.11	13.68	76.03
`szy`	2,055	12.49	14.56	75.39
`tao`	1,838	10.60	12.10	86.38
`tay`	6,723	10.03	10.98	78.01
`trv`	6,983	14.43	15.24	80.84
`tsu`	2,164	11.90	12.90	80.67
`xnb`	2,728	19.40	18.49	69.28
`xsy`	1,883	11.52	12.48	77.75

The corpus gate enforces standard-tier Formosan text, at least 7.5% test and 2.5% validation per language, human sentence-only evaluation, and zero exact, skeleton, one-edit, configured high character n-gram, or document train/evaluation conflicts. This release passed all gates: exact 0, skeleton 0, one-edit 0, character n-gram 0, and document 0.

See eval/metrics.json for sacreBLEU signatures, per-language, source, dialect, and length diagnostics. publication.json records the corpus, profile, run, and checkpoint hashes used for this release.

Intended use

This model supports research, corpus development, and assisted translation for the 15 included Formosan languages. It is designed for the exact prompt and generation contract shown above.

Limitations

Outputs require knowledgeable speaker review. Aggregate metrics hide large differences among languages and domains. This model is not suitable for authoritative, medical, legal, or safety-critical translation.