README.md

---
license: cc-by-nc-4.0
library_name: transformers
pipeline_tag: translation
base_model: facebook/nllb-200-distilled-600M
language:
- zh
- ami
- bnn
- ckv
- dru
- pwn
- pyu
- ssf
- sxr
- szy
- tao
- tay
- trv
- tsu
- xnb
- xsy
tags:
- translation
- nllb
- nllb-200
- low-resource
- endangered-languages
- formosan-languages
- sentencepiece
- private-no-bible
metrics:
- bleu
- chrf2
- ter
model-index:
- name: nllb200-formosan-zh-spm8k
  results:
  - task:
      name: Machine Translation
      type: translation
    dataset:
      name: FormosanBank Traditional Chinese private no-Bible hard split
      type: custom
    metrics:
    - name: BLEU
      type: bleu
      value: 10.5767
      args:
        direction: f2zh
        samples: 61384
        tokenize: zh
    - name: chrF2
      type: chrf2
      value: 12.3912
      args:
        direction: f2zh
        samples: 61384
    - name: TER
      type: ter
      value: 115.7521
      args:
        direction: f2zh
        samples: 61384
---

# nllb200-formosan-zh-spm8k

**Base model:** [`facebook/nllb-200-distilled-600M`](https://huggingface.co/facebook/nllb-200-distilled-600M)  
**Direction:** **Formosan -> Traditional Chinese**  
**Companion model:** [`FormosanBank/nllb200-zh-formosan-spm8k`](https://huggingface.co/FormosanBank/nllb200-zh-formosan-spm8k)  
**Release:** private no-Bible SPM8k flight `20260712-232900`, validation-selected step `270,000`

This directional checkpoint replaces the earlier release with the strongest `private_no_bible` model from the
fully rebuilt FormosanBank MT pipeline. It uses an 8,192-piece Formosan-aware SentencePiece extension and explicit
direction, source-language, source-domain, and dialect control tags.

## Supported Languages

| Language | NLLB code |
|---|---|
| Traditional Chinese | `zho_Hant` |
| Amis | `ami_Latn` |
| Bunun | `bnn_Latn` |
| Kavalan | `ckv_Latn` |
| Rukai | `dru_Latn` |
| Paiwan | `pwn_Latn` |
| Puyuma | `pyu_Latn` |
| Thao | `ssf_Latn` |
| Saaroa | `sxr_Latn` |
| Sakizaya | `szy_Latn` |
| Tao / Yami | `tao_Latn` |
| Atayal | `tay_Latn` |
| Seediq | `trv_Latn` |
| Tsou | `tsu_Latn` |
| Kanakanavu | `xnb_Latn` |
| Saisiyat | `xsy_Latn` |

## Input Format

Prefix every source with:

`<to_zh> <src_LANG> <dom_BUCKET> <dialect_DIALECT>`

Example:

`<to_zh> <src_ami> <dom_unknown> <dialect_default> Pa'araw cingra to demak nira.`

Use `<dom_unknown>` and `<dialect_default>` when metadata is unavailable.

## Usage

Use the slow `NllbTokenizer` (`use_fast=False` with `AutoTokenizer`). These checkpoints were trained with
`transformers==4.56.1`; fast-tokenizer added-token IDs can differ from the slow tokenizer IDs used in training.
NLLB generation must start with the tokenizer EOS ID and force the target-language BOS ID.

```python
import torch
from transformers import AutoModelForSeq2SeqLM, NllbTokenizer

model_id = "FormosanBank/nllb200-formosan-zh-spm8k"
tokenizer = NllbTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
model.to("cuda" if torch.cuda.is_available() else "cpu")

FORMOSAN_TO_LID = {
    "ami": "ami_Latn", "bnn": "bnn_Latn", "ckv": "ckv_Latn", "dru": "dru_Latn",
    "pwn": "pwn_Latn", "pyu": "pyu_Latn", "ssf": "ssf_Latn", "sxr": "sxr_Latn",
    "szy": "szy_Latn", "tao": "tao_Latn", "tay": "tay_Latn", "trv": "trv_Latn",
    "tsu": "tsu_Latn", "xnb": "xnb_Latn", "xsy": "xsy_Latn",
}

def translate_formosan_to_chinese(text: str, lang_code: str, source_bucket: str = "unknown", dialect: str = "default") -> str:
    tokenizer.src_lang = FORMOSAN_TO_LID[lang_code]
    prompt = f"<to_zh> <src_{lang_code}> <dom_{source_bucket}> <dialect_{dialect}> {text}"
    inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=384).to(model.device)
    outputs = model.generate(
        **inputs,
        forced_bos_token_id=tokenizer.convert_tokens_to_ids("zho_Hant"),
        decoder_start_token_id=tokenizer.eos_token_id,
        max_new_tokens=128,
        num_beams=4,
        no_repeat_ngram_size=3,
        repetition_penalty=1.15,
        early_stopping=True,
    )
    return tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]

print(translate_formosan_to_chinese("Pa'araw cingra to demak nira.", "ami"))
```

## Checkpoint Selection

The published checkpoint was selected **only on validation chrF2**, not on the hard test set.

| Selection step | Validation samples | Validation loss | Perplexity | BLEU | chrF2 | TER |
|---:|---:|---:|---:|---:|---:|---:|
| 270,000 | 1,920 | 2.4207 | 11.25 | 23.58 | 22.94 | 103.94 |

Validation generation sampled 128 rows per Formosan language every 10,000 updates. The full hard test was evaluated
only after selection.

## Training Setup

| Setting | Value |
|---|---|
| Corpus | `private_no_bible` (Traditional Chinese) |
| Base model | `facebook/nllb-200-distilled-600M` |
| Maximum updates | 300,000 |
| Published best step | 270,000 |
| Microbatch / accumulation | 16 / 4 |
| Effective batch | 64 |
| Maximum length | 384 |
| Learning rate | `2e-05` |
| Precision | `bf16` |
| Easy-source weight | 0.05 |
| Language sampling alpha | 0.5 |
| Metadata control tags | enabled and validated as single tokenizer IDs |

## Corpus and Split Integrity

| Total | Train | Test | Validate | Minimum per-language test | Minimum per-language validate |
|---:|---:|---:|---:|---:|---:|
| 791,330 | 708,496 | 61,384 | 21,450 | 7.5% | 2.5% |

The exact `Formosan-Taiwan-Bible-Society-Bibles` repository is excluded. Lexical entries are train-only. Independent
validation found zero normalized source, target, or pair overlap; zero punctuation/spacing skeleton overlap; and zero
one-edit source or target conflicts across train and evaluation. Connected similarity groups are assigned as units so
variants cannot be split independently merely because they are not exact duplicates.

All hard-test references in this Chinese corpus are original rather than pivot-generated.

## Hard-Test Results

SacreBLEU was computed with `zh` tokenization; chrF uses beta 2; TER is lower-is-better.

| Direction | Samples | BLEU | chrF2 | TER | Exact match | Empty output |
|---|---:|---:|---:|---:|---:|---:|
| Formosan -> Traditional Chinese | 61,384 | 10.58 | 12.39 | 115.75 | 1.40% | 0.00% |

### Per-Language Results

| Language | Code | Samples | BLEU | chrF2 | TER |
|---|---:|---:|---:|---:|---:|
| Amis | `ami_Latn` | 10,929 | 7.91 | 10.07 | 115.26 |
| Bunun | `bnn_Latn` | 5,450 | 9.24 | 10.91 | 120.58 |
| Kavalan | `ckv_Latn` | 2,852 | 15.76 | 16.77 | 108.05 |
| Rukai | `dru_Latn` | 5,524 | 8.14 | 10.05 | 127.33 |
| Paiwan | `pwn_Latn` | 5,519 | 8.81 | 10.99 | 111.15 |
| Puyuma | `pyu_Latn` | 4,105 | 13.10 | 15.12 | 107.76 |
| Thao | `ssf_Latn` | 1,670 | 14.87 | 16.29 | 110.56 |
| Saaroa | `sxr_Latn` | 1,539 | 8.57 | 11.04 | 109.41 |
| Sakizaya | `szy_Latn` | 2,147 | 12.58 | 15.79 | 110.10 |
| Tao / Yami | `tao_Latn` | 1,802 | 8.32 | 11.30 | 109.37 |
| Atayal | `tay_Latn` | 6,395 | 8.92 | 10.24 | 125.92 |
| Seediq | `trv_Latn` | 6,689 | 17.66 | 18.01 | 107.97 |
| Tsou | `tsu_Latn` | 1,889 | 8.00 | 10.40 | 115.47 |
| Kanakanavu | `xnb_Latn` | 2,927 | 17.48 | 18.51 | 111.07 |
| Saisiyat | `xsy_Latn` | 1,947 | 9.94 | 12.25 | 125.58 |

Full source-bucket and length-bin breakdowns are in [`eval/metrics.json`](eval/metrics.json).

## Intended Use

- Research, teaching, and prototyping for Formosan-language machine translation.
- Draft translation assistance where knowledgeable speakers can review the output.
- Comparative low-resource MT evaluation on the documented leakage-controlled split.

## Limitations

- Output may be incorrect, ungrammatical, incomplete, or culturally inappropriate.
- Formosan generation is draft-only and requires speaker review.
- Aggregate scores across 15 languages conceal substantial per-language variation.
- This model is unsuitable for legal, medical, safety-critical, or authoritative community-facing use without expert review.
- Hard-split scores are not directly comparable with earlier evaluations that allowed stronger train-test similarity.

## License

Released under `cc-by-nc-4.0`. Underlying corpus sources may impose additional restrictions. Confirm the rights needed
for your use case.

## Citation

```bibtex
@misc{formosanbank_nllb200_formosan_zh_spm8k_2026,
  title  = {nllb200-formosan-zh-spm8k: Directional NLLB-200 MT on the FormosanBank private no-Bible corpus},
  author = {FormosanBank contributors},
  year   = {2026},
  url    = {https://huggingface.co/FormosanBank/nllb200-formosan-zh-spm8k}
}
```