Publish NLLB SPM8k private no-Bible v3 best checkpoint (20260809-210523)

Browse files

Files changed (14) hide show

README.md +129 -188
added_tokens.json +225 -180
config.json +1 -1
eval/metrics.json +0 -0
experiment_metadata.json +190 -102
formosan_mt_inference.py +79 -0
model.safetensors +2 -2
mt_standardization.py +458 -0
mt_standardization_profile.json +32 -0
publication.json +30 -0
sentencepiece.bpe.model +2 -2
special_tokens_map.json +71 -27
tokenizer_config.json +649 -245
training_profile.json +94 -0

README.md CHANGED Viewed

@@ -4,7 +4,6 @@ library_name: transformers
 pipeline_tag: translation
 base_model: facebook/nllb-200-distilled-600M
 language:
-- eng
 - ami
 - bnn
 - ckv
@@ -20,232 +19,174 @@ language:
 - tsu
 - xnb
 - xsy
 tags:
 - translation
-- nllb
 - nllb-200
-- low-resource
-- endangered-languages
 - formosan-languages
-- sentencepiece
-- private-no-bible
 metrics:
 - bleu
-- chrf2
 - ter
 model-index:
 - name: nllb200-formosan-en-spm8k
   results:
   - task:
-      name: Machine Translation
       type: translation
     dataset:
-      name: FormosanBank English private no-Bible hard split
-      type: custom
     metrics:
-    - name: BLEU
-      type: bleu
-      value: 13.0773
-      args:
-        direction: f2en
-        samples: 58129
-        tokenize: 13a
-    - name: chrF2
-      type: chrf2
-      value: 31.1654
-      args:
-        direction: f2en
-        samples: 58129
-    - name: TER
-      type: ter
-      value: 86.4966
-      args:
-        direction: f2en
-        samples: 58129
 ---
 # nllb200-formosan-en-spm8k
-**Base model:** [`facebook/nllb-200-distilled-600M`](https://huggingface.co/facebook/nllb-200-distilled-600M)
-**Direction:** **Formosan -> English**
-**Companion model:** [`FormosanBank/nllb200-en-formosan-spm8k`](https://huggingface.co/FormosanBank/nllb200-en-formosan-spm8k)
-**Release:** private no-Bible SPM8k flight `20260712-232900`, validation-selected step `270,000`
-This directional checkpoint replaces the earlier release with the strongest `private_no_bible` model from the
-fully rebuilt FormosanBank MT pipeline. It uses an 8,192-piece Formosan-aware SentencePiece extension and explicit
-direction, source-language, source-domain, and dialect control tags.
-## Supported Languages
-| Language | NLLB code |
 |---|---|
-| English | `eng_Latn` |
-| Amis | `ami_Latn` |
-| Bunun | `bnn_Latn` |
-| Kavalan | `ckv_Latn` |
-| Rukai | `dru_Latn` |
-| Paiwan | `pwn_Latn` |
-| Puyuma | `pyu_Latn` |
-| Thao | `ssf_Latn` |
-| Saaroa | `sxr_Latn` |
-| Sakizaya | `szy_Latn` |
-| Tao / Yami | `tao_Latn` |
-| Atayal | `tay_Latn` |
-| Seediq | `trv_Latn` |
-| Tsou | `tsu_Latn` |
-| Kanakanavu | `xnb_Latn` |
-| Saisiyat | `xsy_Latn` |
-## Input Format
-Prefix every source with:
-`<to_eng> <src_LANG> <dom_BUCKET> <dialect_DIALECT>`
-Example:
-`<to_eng> <src_ami> <dom_unknown> <dialect_default> Pa'araw cingra to demak nira.`
-Use `<dom_unknown>` and `<dialect_default>` when metadata is unavailable.
 ## Usage
-Use the slow `NllbTokenizer` (`use_fast=False` with `AutoTokenizer`). These checkpoints were trained with
-`transformers==4.56.1`; fast-tokenizer added-token IDs can differ from the slow tokenizer IDs used in training.
-NLLB generation must start with the tokenizer EOS ID and force the target-language BOS ID.
 ```python
 import torch
 from transformers import AutoModelForSeq2SeqLM, NllbTokenizer
 model_id = "FormosanBank/nllb200-formosan-en-spm8k"
-tokenizer = NllbTokenizer.from_pretrained(model_id)
 model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
 model.to("cuda" if torch.cuda.is_available() else "cpu")
-FORMOSAN_TO_LID = {
-    "ami": "ami_Latn", "bnn": "bnn_Latn", "ckv": "ckv_Latn", "dru": "dru_Latn",
-    "pwn": "pwn_Latn", "pyu": "pyu_Latn", "ssf": "ssf_Latn", "sxr": "sxr_Latn",
-    "szy": "szy_Latn", "tao": "tao_Latn", "tay": "tay_Latn", "trv": "trv_Latn",
-    "tsu": "tsu_Latn", "xnb": "xnb_Latn", "xsy": "xsy_Latn",
-}
-def translate_formosan_to_english(text: str, lang_code: str, source_bucket: str = "unknown", dialect: str = "default") -> str:
-    tokenizer.src_lang = FORMOSAN_TO_LID[lang_code]
-    prompt = f"<to_eng> <src_{lang_code}> <dom_{source_bucket}> <dialect_{dialect}> {text}"
-    inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=384).to(model.device)
-    outputs = model.generate(
         **inputs,
-        forced_bos_token_id=tokenizer.convert_tokens_to_ids("eng_Latn"),
         decoder_start_token_id=tokenizer.eos_token_id,
-        max_new_tokens=128,
         num_beams=4,
-        no_repeat_ngram_size=3,
-        repetition_penalty=1.15,
-        early_stopping=True,
     )
-    return tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]
-print(translate_formosan_to_english("Pa'araw cingra to demak nira.", "ami"))
 ```
-## Checkpoint Selection
-The published checkpoint was selected **only on validation chrF2**, not on the hard test set.
-| Selection step | Validation samples | Validation loss | Perplexity | BLEU | chrF2 | TER |
-|---:|---:|---:|---:|---:|---:|---:|
-| 270,000 | 1,920 | 2.2070 | 9.09 | 19.15 | 35.33 | 80.93 |
-Validation generation sampled 128 rows per Formosan language every 10,000 updates. The full hard test was evaluated
-only after selection.
-## Training Setup
-| Setting | Value |
-|---|---|
-| Corpus | `private_no_bible` (English) |
-| Base model | `facebook/nllb-200-distilled-600M` |
-| Maximum updates | 300,000 |
-| Published best step | 270,000 |
-| Microbatch / accumulation | 16 / 4 |
-| Effective batch | 64 |
-| Maximum length | 384 |
-| Learning rate | `2e-05` |
-| Precision | `bf16` |
-| Easy-source weight | 0.05 |
-| Language sampling alpha | 0.5 |
-| Metadata control tags | enabled and validated as single tokenizer IDs |
-## Corpus and Split Integrity
-| Total | Train | Test | Validate | Minimum per-language test | Minimum per-language validate |
-|---:|---:|---:|---:|---:|---:|
-| 759,493 | 681,034 | 58,129 | 20,330 | 7.5% | 2.5% |
-The exact `Formosan-Taiwan-Bible-Society-Bibles` repository is excluded. Lexical entries are train-only. Independent
-validation found zero normalized source, target, or pair overlap; zero punctuation/spacing skeleton overlap; and zero
-one-edit source or target conflicts across train and evaluation. Connected similarity groups are assigned as units so
-variants cannot be split independently merely because they are not exact duplicates.
-The English hard test contains 51,206 original-reference rows and 6,923 DeepL-pivoted reference rows. Synthetic rows were admitted only when the available human sentence groups could not satisfy a language's minimum evaluation floor; metrics should not be described as human-only.
-## Hard-Test Results
-SacreBLEU was computed with `13a` tokenization; chrF uses beta 2; TER is lower-is-better.
-| Direction | Samples | BLEU | chrF2 | TER | Exact match | Empty output |
-|---|---:|---:|---:|---:|---:|---:|
-| Formosan -> English | 58,129 | 13.08 | 31.17 | 86.50 | 2.42% | 0.00% |
-### Per-Language Results
-| Language | Code | Samples | BLEU | chrF2 | TER |
-|---|---:|---:|---:|---:|---:|
-| Amis | `ami_Latn` | 10,741 | 10.50 | 28.90 | 93.74 |
-| Bunun | `bnn_Latn` | 5,319 | 12.84 | 32.45 | 80.82 |
-| Kavalan | `ckv_Latn` | 2,728 | 13.74 | 32.26 | 85.13 |
-| Rukai | `dru_Latn` | 5,340 | 9.20 | 26.56 | 88.63 |
-| Paiwan | `pwn_Latn` | 5,349 | 11.78 | 31.27 | 84.88 |
-| Puyuma | `pyu_Latn` | 3,480 | 15.26 | 34.25 | 79.45 |
-| Thao | `ssf_Latn` | 1,521 | 18.84 | 36.02 | 80.85 |
-| Saaroa | `sxr_Latn` | 1,466 | 16.72 | 33.26 | 84.19 |
-| Sakizaya | `szy_Latn` | 1,983 | 15.14 | 34.29 | 89.05 |
-| Tao / Yami | `tao_Latn` | 1,729 | 16.97 | 33.51 | 83.90 |
-| Atayal | `tay_Latn` | 5,985 | 16.31 | 33.12 | 80.59 |
-| Seediq | `trv_Latn` | 6,386 | 14.66 | 32.55 | 86.39 |
-| Tsou | `tsu_Latn` | 1,784 | 11.32 | 29.23 | 89.88 |
-| Kanakanavu | `xnb_Latn` | 2,688 | 8.31 | 29.18 | 86.57 |
-| Saisiyat | `xsy_Latn` | 1,630 | 10.52 | 29.28 | 90.05 |
-Full source-bucket and length-bin breakdowns are in [`eval/metrics.json`](eval/metrics.json).
-## Intended Use
-- Research, teaching, and prototyping for Formosan-language machine translation.
-- Draft translation assistance where knowledgeable speakers can review the output.
-- Comparative low-resource MT evaluation on the documented leakage-controlled split.
 ## Limitations
-- Output may be incorrect, ungrammatical, incomplete, or culturally inappropriate.
-- Formosan generation is draft-only and requires speaker review.
-- Aggregate scores across 15 languages conceal substantial per-language variation.
-- This model is unsuitable for legal, medical, safety-critical, or authoritative community-facing use without expert review.
-- Hard-split scores are not directly comparable with earlier evaluations that allowed stronger train-test similarity.
-## License
-Released under `cc-by-nc-4.0`. Underlying corpus sources may impose additional restrictions. Confirm the rights needed
-for your use case.
-## Citation
-```bibtex
-@misc{formosanbank_nllb200_formosan_en_spm8k_2026,
-  title  = {nllb200-formosan-en-spm8k: Directional NLLB-200 MT on the FormosanBank private no-Bible corpus},
-  author = {FormosanBank contributors},
-  year   = {2026},
-  url    = {https://huggingface.co/FormosanBank/nllb200-formosan-en-spm8k}
-}
-```

 pipeline_tag: translation
 base_model: facebook/nllb-200-distilled-600M
 language:
 - ami
 - bnn
 - ckv
 - tsu
 - xnb
 - xsy
+- en
 tags:
 - translation
 - nllb-200
 - formosan-languages
+- low-resource
 metrics:
 - bleu
+- chrf
 - ter
 model-index:
 - name: nllb200-formosan-en-spm8k
   results:
   - task:
       type: translation
+      name: Translation
     dataset:
+      name: FormosanBank private no-Bible hard test
+      type: private-no-bible-hard-test
+      split: test
     metrics:
+    - type: bleu
+      name: sacreBLEU
+      value: 9.324037
+    - type: chrf
+      name: chrF2
+      value: 26.929850
+    - type: ter
+      name: TER
+      value: 96.958093
 ---
 # nllb200-formosan-en-spm8k
+**Direction:** Formosan to English<br>
+**Base model:** [`facebook/nllb-200-distilled-600M`](https://huggingface.co/facebook/nllb-200-distilled-600M)<br>
+**Recipe:** `nllb200-spm8k-directional-v3`<br>
+**Release:** `20260809-210523`, validation-selected step 210,000
+This is a directional model for 15 Formosan languages. It uses the
+`private_no_bible` leakage-controlled corpus, Formosan-aware 8k SentencePiece extension, balanced
+language/source sampling, and direction/domain/dialect control tags. The model
+weights are public, but the private training corpus is not included.
+## Model details
+| Item | Value |
 |---|---|
+| Base revision | `f8d333a098d19b4fd9a8b18f94170487ad3f821d` |
+| Training rows | 573,657 |
+| Effective batch size | 64 |
+| Maximum sequence length | 384 |
+| Learning rate | 2e-05 |
+| Precision | `bf16` |
+| Checkpoint selection | Human validation `chrF2` |
+| Formosan text | `kindOf=standard`, `formosan-mt-standard-v3` |
+| Corpus SHA-256 | `e3feeaf7c3c51b9cd5c4b0537ffe44a370e7d02723f34b467479b6c4f5f0ea77` |
+| Training profile SHA-256 | `34f45832bdedc1b8322b26936dd88667dfecb360d941a7f54f60ae31a1fb4e10` |
 ## Usage
 ```python
 import torch
 from transformers import AutoModelForSeq2SeqLM, NllbTokenizer
+from formosan_mt_inference import normalize_formosan
 model_id = "FormosanBank/nllb200-formosan-en-spm8k"
+tokenizer = NllbTokenizer.from_pretrained(model_id, use_fast=False)
 model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
 model.to("cuda" if torch.cuda.is_available() else "cpu")
+NLLB_LIDS = {'ami': 'ami_Latn', 'bnn': 'bnn_Latn', 'ckv': 'ckv_Latn', 'dru': 'dru_Latn', 'pwn': 'pwn_Latn', 'pyu': 'pyu_Latn', 'ssf': 'ssf_Latn', 'sxr': 'sxr_Latn', 'szy': 'szy_Latn', 'tao': 'tao_Latn', 'tay': 'tay_Latn', 'trv': 'trv_Latn', 'tsu': 'tsu_Latn', 'xnb': 'xnb_Latn', 'xsy': 'xsy_Latn'}
+def translate(text, lang_code, source_bucket="unknown", dialect="default"):
+    text = normalize_formosan(text, lang_code)
+    tokenizer.src_lang = NLLB_LIDS[lang_code]
+    prompt = (
+        f"<to_eng> <src_{lang_code}> <dom_{source_bucket}> "
+        f"<dialect_{dialect}> {text}"
+    )
+    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+    output = model.generate(
         **inputs,
         decoder_start_token_id=tokenizer.eos_token_id,
+        forced_bos_token_id=tokenizer.convert_tokens_to_ids('eng_Latn'),
+        max_new_tokens=256,
         num_beams=4,
     )
+    return tokenizer.batch_decode(output, skip_special_tokens=True)[0]
+print(translate("Pa'araw cingra.", "ami"))
 ```
+The control tags are part of the training contract. Use `unknown` and `default`
+when source bucket or dialect metadata is unavailable.
+## Evaluation
+The best checkpoint was selected on human validation chrF2. Test references
+are human sentence pairs; synthetic pivots and lexical entries are train-only.
+The headline result uses `default` metadata controls, so it does not
+assume access to test-set domain or dialect labels.
+| Split | Rows |
+|---|---:|
+| Train | 573,657 |
+| Test | 63,140 |
+| Validate | 20,733 |
+| Scope | BLEU | chrF2 | TER |
+|---|---:|---:|---:|
+| Hard test | 9.32 | 26.93 | 96.96 |
+| Selection validation | 9.53 | 29.70 | 92.68 |
+Test empty-output rate: 0.0000%.
+### Confidence intervals
+Stratified bootstrap, 200 samples, 95% confidence.
+| Metric | Lower | Upper |
+|---|---:|---:|
+| BLEU | 9.18 | 9.48 |
+| chrF2 | 26.78 | 27.08 |
+| TER | 96.48 | 97.54 |
+| Language | Samples | BLEU | chrF2 | TER |
+|---|---:|---:|---:|---:|
+| `ami` | 11,017 | 7.62 | 26.29 | 93.14 |
+| `bnn` | 6,049 | 14.68 | 33.73 | 77.16 |
+| `ckv` | 2,910 | 10.84 | 30.22 | 96.13 |
+| `dru` | 5,846 | 1.07 | 13.93 | 135.78 |
+| `pwn` | 5,811 | 7.65 | 24.46 | 106.90 |
+| `pyu` | 4,315 | 12.50 | 29.72 | 107.12 |
+| `ssf` | 1,630 | 14.05 | 31.21 | 87.89 |
+| `sxr` | 2,081 | 11.33 | 26.27 | 96.93 |
+| `szy` | 1,852 | 13.59 | 32.77 | 90.81 |
+| `tao` | 2,257 | 9.04 | 27.97 | 93.45 |
+| `tay` | 6,411 | 10.46 | 26.58 | 104.63 |
+| `trv` | 6,502 | 12.81 | 31.71 | 96.50 |
+| `tsu` | 2,003 | 2.39 | 19.11 | 95.33 |
+| `xnb` | 2,682 | 5.62 | 25.89 | 93.22 |
+| `xsy` | 1,774 | 16.10 | 35.31 | 83.12 |
+The corpus gate enforces standard-tier Formosan text, at least 7.5% test and
+2.5% validation per language, human sentence-only evaluation, and zero exact,
+skeleton, one-edit, configured high character n-gram, or document
+train/evaluation conflicts. This release passed all gates: exact
+0, skeleton
+0, one-edit
+0, character n-gram
+0, and document
+0.
+See `eval/metrics.json` for sacreBLEU signatures, per-language, source,
+dialect, and length diagnostics. `publication.json` records the corpus,
+profile, run, and checkpoint hashes used for this release.
+## Intended use
+This model supports research, corpus development, and assisted translation for
+the 15 included Formosan languages. It is designed for the exact prompt and
+generation contract shown above.
 ## Limitations
+Outputs require knowledgeable speaker review. Aggregate metrics hide large
+differences among languages and domains. This model is not suitable for
+authoritative, medical, legal, or safety-critical translation.

added_tokens.json CHANGED Viewed

@@ -1,182 +1,227 @@
 {
-  "<dae>": 259232,
-  "<dialect_central>": 259233,
-  "<dialect_coastal>": 259234,
-  "<dialect_dawu>": 259235,
-  "<dialect_default>": 259236,
-  "<dialect_deluvalley>": 259237,
-  "<dialect_dona>": 259238,
-  "<dialect_duda>": 259239,
-  "<dialect_eastern>": 259240,
-  "<dialect_fourseasons>": 259241,
-  "<dialect_hengchun>": 259242,
-  "<dialect_jianhe>": 259243,
-  "<dialect_junqun>": 259244,
-  "<dialect_kanakanavu>": 259245,
-  "<dialect_kaqun>": 259246,
-  "<dialect_kavalan>": 259247,
-  "<dialect_luanqun>": 259248,
-  "<dialect_malan>": 259249,
-  "<dialect_maolin>": 259250,
-  "<dialect_nanwang>": 259251,
-  "<dialect_northern>": 259252,
-  "<dialect_saaroa>": 259253,
-  "<dialect_saisiyat>": 259254,
-  "<dialect_sakizaya>": 259255,
-  "<dialect_sekolik>": 259256,
-  "<dialect_southern>": 259257,
-  "<dialect_tanqun>": 259258,
-  "<dialect_tegudaya>": 259259,
-  "<dialect_thao>": 259260,
-  "<dialect_truku>": 259261,
-  "<dialect_tsou>": 259262,
-  "<dialect_unknown>": 259263,
-  "<dialect_wanda>": 259264,
-  "<dialect_wanshan>": 259265,
-  "<dialect_wenshui>": 259266,
-  "<dialect_wutai>": 259267,
-  "<dialect_xiqun>": 259268,
-  "<dialect_xiuguluan>": 259269,
-  "<dialect_yami>": 259270,
-  "<dialect_yilanzeaol>": 259271,
-  "<dialect_zeaol>": 259272,
-  "<dialect_zhiben>": 259273,
-  "<dialect_zhuoqun>": 259274,
-  "<dom_classroom_context>": 259275,
-  "<dom_culture>": 259276,
-  "<dom_dictionary>": 259277,
-  "<dom_essays>": 259278,
-  "<dom_ethnography_journal_ritual_song>": 259279,
-  "<dom_formosan_100_paiwan_texts>": 259280,
-  "<dom_formosan_academia_sinica_oral_legends>": 259281,
-  "<dom_formosan_amis_adversative_constructions>": 259282,
-  "<dom_formosan_amis_kavalan_lin_interrogative_verbs>": 259283,
-  "<dom_formosan_amis_kuo_sung_comparative_constructions>": 259284,
-  "<dom_formosan_amis_lifok_dongi_saopo_kimakimad>": 259285,
-  "<dom_formosan_amis_myths_and_customs>": 259286,
-  "<dom_formosan_amis_pa_verbs>": 259287,
-  "<dom_formosan_amis_serial_verb_constructions>": 259288,
-  "<dom_formosan_amis_tung_chiou>": 259289,
-  "<dom_formosan_asai_sedik_language>": 259290,
-  "<dom_formosan_bunun_debusser_dissertation>": 259291,
-  "<dom_formosan_bunun_moriguchi_northern_texts>": 259292,
-  "<dom_formosan_bunun_topic_focus>": 259293,
-  "<dom_formosan_chen_fukuda_relabeling_ergative>": 259294,
-  "<dom_formosan_chen_fukuda_three_ways_steal>": 259295,
-  "<dom_formosan_cip_atayal_grammar_overview>": 259296,
-  "<dom_formosan_dean_johnson_year_clouds_smangus>": 259297,
-  "<dom_formosan_dorinda_tsai_hsiu_liu_neutral_imperfect>": 259298,
-  "<dom_formosan_egerod_origin_headhunting_atayal_text_v>": 259299,
-  "<dom_formosan_egerod_soren_word_order_word_classes_at>": 259300,
-  "<dom_formosan_elizabeth_zeitoun_chen_huei_wu_overview>": 259301,
-  "<dom_formosan_epark>": 259302,
-  "<dom_formosan_ferrell_taiwan_aboriginal_groups_proble>": 259303,
-  "<dom_formosan_gitbook_translations>": 259304,
-  "<dom_formosan_glosbe>": 259305,
-  "<dom_formosan_hala_saku_la_videos>": 259306,
-  "<dom_formosan_holmer_seediq>": 259307,
-  "<dom_formosan_huang_grammaticalization_squliq_atayal>": 259308,
-  "<dom_formosan_huang_mei_chin_atayal_reference_grammar>": 259309,
-  "<dom_formosan_huteson_rukai_survey>": 259310,
-  "<dom_formosan_ilrdf_42_language_practice_word_lists>": 259311,
-  "<dom_formosan_ilrdf_tousvusvutu_kita>": 259312,
-  "<dom_formosan_indigenous_language_literary_awards>": 259313,
-  "<dom_formosan_kanakanavu_texts>": 259314,
-  "<dom_formosan_kavalan_zhang_reference_grammar>": 259315,
-  "<dom_formosan_li_peirong_xu_weisheng_introduction_tru>": 259316,
-  "<dom_formosan_li_peizhen_wu_xingyu_shouhu_xing>": 259317,
-  "<dom_formosan_naomi_tsukida_correlative_clauses_seedi>": 259318,
-  "<dom_formosan_nowbucyang_truku_thesis>": 259319,
-  "<dom_formosan_old_texts>": 259320,
-  "<dom_formosan_paiwan_collart_zeitoun_time_reference>": 259321,
-  "<dom_formosan_paiwan_ho_five_dialects>": 259322,
-  "<dom_formosan_paiwanstories>": 259323,
-  "<dom_formosan_puyuma_chen_raising_to_object>": 259324,
-  "<dom_formosan_puyuma_katipol_kopfjagdriten>": 259325,
-  "<dom_formosan_puyuma_teng_pronominal_systems>": 259326,
-  "<dom_formosan_puyuma_teng_reference_grammar>": 259327,
-  "<dom_formosan_rik_bunun>": 259328,
-  "<dom_formosan_robert_blust_austronesian_homeland_ling>": 259329,
-  "<dom_formosan_rukai_mantauran_stories>": 259330,
-  "<dom_formosan_rukai_texts>": 259331,
-  "<dom_formosan_rukai_zeitoun_saying>": 259332,
-  "<dom_formosan_saaroa_pan_grammar>": 259333,
-  "<dom_formosan_sakizaya_affixes>": 259334,
-  "<dom_formosan_seals>": 259335,
-  "<dom_formosan_seediq_zhang_reference_grammar>": 259336,
-  "<dom_formosan_shigeru_tsuchida_kanakanavu_texts>": 259337,
-  "<dom_formosan_shigeru_tsuchida_reconstruction_proto_t>": 259338,
-  "<dom_formosan_shih_rukai_adverbial>": 259339,
-  "<dom_formosan_song_limei_introduction_kanakanavu_gram>": 259340,
-  "<dom_formosan_song_limei_introduction_seediq_grammar>": 259341,
-  "<dom_formosan_sung_kuo_descriptive_comparative_constr>": 259342,
-  "<dom_formosan_taoshan_elementary_school_atelier_hui_k>": 259343,
-  "<dom_formosan_tsukida_naomi_verb_classification_amis>": 259344,
-  "<dom_formosan_wei_matri_clan_lineage_system_ami>": 259345,
-  "<dom_formosan_wilang_yutas_videos>": 259346,
-  "<dom_formosan_wu_jinglan_introduction_amis_grammar>": 259347,
-  "<dom_formosan_yedda_palemeq_blog>": 259348,
-  "<dom_formosan_yeddas_blog>": 259349,
-  "<dom_formosan_yeh_meili_introduction_saisiyat_grammar>": 259350,
-  "<dom_formosan_zhang_yongli_pan_jiarong_introduction_t>": 259351,
-  "<dom_formosan_zheng_acl_2024>": 259352,
-  "<dom_formosan_zheng_data>": 259353,
-  "<dom_learning_vocab>": 259354,
-  "<dom_nine_level>": 259355,
-  "<dom_ntu>": 259356,
-  "<dom_picture_book>": 259357,
-  "<dom_picture_story>": 259358,
-  "<dom_presidential_apology>": 259359,
-  "<dom_reading_writing>": 259360,
-  "<dom_unknown>": 259361,
-  "<dom_youtube>": 259362,
-  "<src_ami>": 259363,
-  "<src_bnn>": 259364,
-  "<src_ckv>": 259365,
-  "<src_dru>": 259366,
-  "<src_eng>": 259367,
-  "<src_pwn>": 259368,
-  "<src_pyu>": 259369,
-  "<src_ssf>": 259370,
-  "<src_sxr>": 259371,
-  "<src_szy>": 259372,
-  "<src_tao>": 259373,
-  "<src_tay>": 259374,
-  "<src_trv>": 259375,
-  "<src_tsu>": 259376,
-  "<src_xnb>": 259377,
-  "<src_xsy>": 259378,
-  "<src_zh>": 259379,
-  "<to_ami>": 259380,
-  "<to_bnn>": 259381,
-  "<to_ckv>": 259382,
-  "<to_dru>": 259383,
-  "<to_eng>": 259384,
-  "<to_pwn>": 259385,
-  "<to_pyu>": 259386,
-  "<to_ssf>": 259387,
-  "<to_sxr>": 259388,
-  "<to_szy>": 259389,
-  "<to_tao>": 259390,
-  "<to_tay>": 259391,
-  "<to_trv>": 259392,
-  "<to_tsu>": 259393,
-  "<to_xnb>": 259394,
-  "<to_xsy>": 259395,
-  "<to_zh>": 259396,
-  "ami_Latn": 259217,
-  "bnn_Latn": 259218,
-  "ckv_Latn": 259219,
-  "dru_Latn": 259220,
-  "pwn_Latn": 259221,
-  "pyu_Latn": 259222,
-  "ssf_Latn": 259223,
-  "sxr_Latn": 259224,
-  "szy_Latn": 259225,
-  "tao_Latn": 259226,
-  "tay_Latn": 259227,
-  "trv_Latn": 259228,
-  "tsu_Latn": 259229,
-  "xnb_Latn": 259230,
-  "xsy_Latn": 259231
 }

 {
+  "<dialect_central>": 261410,
+  "<dialect_coastal>": 261411,
+  "<dialect_dawu>": 261412,
+  "<dialect_default>": 261413,
+  "<dialect_deluvalley>": 261414,
+  "<dialect_dona>": 261415,
+  "<dialect_duda>": 261416,
+  "<dialect_eastern>": 261417,
+  "<dialect_fourseasons>": 261418,
+  "<dialect_hengchun>": 261419,
+  "<dialect_jianhe>": 261420,
+  "<dialect_junqun>": 261421,
+  "<dialect_kanakanavu>": 261422,
+  "<dialect_kaqun>": 261423,
+  "<dialect_kavalan>": 261424,
+  "<dialect_luanqun>": 261425,
+  "<dialect_malan>": 261426,
+  "<dialect_maolin>": 261427,
+  "<dialect_nanwang>": 261428,
+  "<dialect_northern>": 261429,
+  "<dialect_saaroa>": 261430,
+  "<dialect_saisiyat>": 261431,
+  "<dialect_sakizaya>": 261432,
+  "<dialect_sekolik>": 261433,
+  "<dialect_southern>": 261434,
+  "<dialect_tanqun>": 261435,
+  "<dialect_tegudaya>": 261436,
+  "<dialect_thao>": 261437,
+  "<dialect_truku>": 261438,
+  "<dialect_tsou>": 261439,
+  "<dialect_unknown>": 261440,
+  "<dialect_wanda>": 261441,
+  "<dialect_wanshan>": 261442,
+  "<dialect_wenshui>": 261443,
+  "<dialect_wutai>": 261444,
+  "<dialect_xiqun>": 261445,
+  "<dialect_xiuguluan>": 261446,
+  "<dialect_yami>": 261447,
+  "<dialect_yilanzeaol>": 261448,
+  "<dialect_zeaol>": 261449,
+  "<dialect_zhiben>": 261450,
+  "<dialect_zhuoqun>": 261451,
+  "<dom_classroom_context>": 261452,
+  "<dom_culture>": 261453,
+  "<dom_dictionary>": 261454,
+  "<dom_essays>": 261455,
+  "<dom_formosan_100_paiwan_texts>": 261456,
+  "<dom_formosan_academia_sinica_oral_legends>": 261457,
+  "<dom_formosan_amis_adversative_constructions>": 261458,
+  "<dom_formosan_amis_huang_english_phonology_grammar>": 261459,
+  "<dom_formosan_amis_kavalan_lin_interrogative_verbs>": 261460,
+  "<dom_formosan_amis_kuo_sung_comparative_constructions>": 261461,
+  "<dom_formosan_amis_lifok_dongi_saopo_kimakimad>": 261462,
+  "<dom_formosan_amis_myths_and_customs>": 261463,
+  "<dom_formosan_amis_pa_verbs>": 261464,
+  "<dom_formosan_anna_hsiou_chuan_chang_reference_gramma>": 261465,
+  "<dom_formosan_anton_quack_puyuma_pulingaw>": 261466,
+  "<dom_formosan_atayal_you_nao_savak_raxayal>": 261467,
+  "<dom_formosan_boyizhenu_narrative_oral_literature_tap>": 261468,
+  "<dom_formosan_bunun_debusser_dissertation>": 261469,
+  "<dom_formosan_bunun_topic_focus>": 261470,
+  "<dom_formosan_catherine_tseng_seediq_atayal_foods_the>": 261471,
+  "<dom_formosan_chen_yaohan_miss_virginia_fey_played_es>": 261472,
+  "<dom_formosan_chun_mei_chen_comparative_phonology_pai>": 261473,
+  "<dom_formosan_chunming_wu_two_types_of_noun_incorpora>": 261474,
+  "<dom_formosan_cip_atayal_grammar_overview>": 261475,
+  "<dom_formosan_claire_mcgill_a_brief_tayal_vocabulary>": 261476,
+  "<dom_formosan_david_blundell_ed_austronesian_taiwan>": 261477,
+  "<dom_formosan_dean_johnson_year_clouds_smangus>": 261478,
+  "<dom_formosan_deng_fangqing_introduction_puyuma_gramm>": 261479,
+  "<dom_formosan_der_hwa_victoria_rau_grammar_atayal>": 261480,
+  "<dom_formosan_dong_manv_yami_ode_to_taro>": 261481,
+  "<dom_formosan_dong_yijia_tjuwabar_paiwan_ritual_langu>": 261482,
+  "<dom_formosan_elizabeth_zeitoun_chen_huei_wu_overview>": 261483,
+  "<dom_formosan_elizabeth_zeitoun_lillian_huang_marie_y>": 261484,
+  "<dom_formosan_elizabeth_zeitoun_pronominal_system_man>": 261485,
+  "<dom_formosan_elizabeth_zeitoun_squib_dynamic_vs_stat>": 261486,
+  "<dom_formosan_epark>": 261487,
+  "<dom_formosan_gitbook_translations>": 261488,
+  "<dom_formosan_glosbe>": 261489,
+  "<dom_formosan_ho_dan_phonological_system_butonglu_pai>": 261490,
+  "<dom_formosan_holmer_seediq>": 261491,
+  "<dom_formosan_hsiu_chuan_liao_transitivity_ergativity>": 261492,
+  "<dom_formosan_huang_grammaticalization_squliq_atayal>": 261493,
+  "<dom_formosan_huang_hui_chuan_glide_formation_takitud>": 261494,
+  "<dom_formosan_huang_huijuan_shi_chaokai_introduction>": 261495,
+  "<dom_formosan_huang_mei_chin_atayal_reference_grammar>": 261496,
+  "<dom_formosan_huteson_rukai_survey>": 261497,
+  "<dom_formosan_ilrdf_42_language_practice_word_lists>": 261498,
+  "<dom_formosan_ilrdf_tousvusvutu_kita>": 261499,
+  "<dom_formosan_indigenous_education_resource_center_ke>": 261500,
+  "<dom_formosan_indigenous_language_literary_awards>": 261501,
+  "<dom_formosan_isabelle_bril_roots_and_stems_lexical_a>": 261502,
+  "<dom_formosan_jian_iwan_guo_qingliu_life_history>": 261503,
+  "<dom_formosan_jian_shilang_introduction_thao_grammar>": 261504,
+  "<dom_formosan_john_u_wolff_the_proto_austronesian_pho>": 261505,
+  "<dom_formosan_josiane_cauquelin_aborigines_taiwan_puy>": 261506,
+  "<dom_formosan_kanakanavu_texts>": 261507,
+  "<dom_formosan_kavalan_zhang_reference_grammar>": 261508,
+  "<dom_formosan_kolas_foting_aboriginal_education_world>": 261509,
+  "<dom_formosan_kucapungane>": 261510,
+  "<dom_formosan_kumu_tapas_tribal_memory_oral_history_w>": 261511,
+  "<dom_formosan_lei_shih_family_system_paiwan_at_su_pai>": 261512,
+  "<dom_formosan_li_may_sung_budai_rukai_exclamatives>": 261513,
+  "<dom_formosan_li_may_sung_clausal_nominalization_buda>": 261514,
+  "<dom_formosan_li_peirong_xu_weisheng_introduction_tru>": 261515,
+  "<dom_formosan_li_peizhen_wu_xingyu_shouhu_xing>": 261516,
+  "<dom_formosan_lillian_huang_atayal_participants_cross>": 261517,
+  "<dom_formosan_liu_manyi_kaskun_ata_matas_i_ki_quaz>": 261518,
+  "<dom_formosan_malcom_ross_proto_austronesian_verbal_m>": 261519,
+  "<dom_formosan_maya_yeh_blaq_uv_construction_atayal>": 261520,
+  "<dom_formosan_nanang_tadaw_naci_mowna_pgagu>": 261521,
+  "<dom_formosan_naomi_tsukida_correlative_clauses_seedi>": 261522,
+  "<dom_formosan_nowbucyang_truku_thesis>": 261523,
+  "<dom_formosan_ochiai_izumi_numerals_paran_seediq_rela>": 261524,
+  "<dom_formosan_old_texts>": 261525,
+  "<dom_formosan_paiwan_collart_zeitoun_time_reference>": 261526,
+  "<dom_formosan_paiwan_ho_five_dialects>": 261527,
+  "<dom_formosan_paiwanstories>": 261528,
+  "<dom_formosan_patricia_stanley_morphophonemics_of_ver>": 261529,
+  "<dom_formosan_paul_jen_kuei_li_internal_relationships>": 261530,
+  "<dom_formosan_paul_jen_kuei_li_rukai_structure>": 261531,
+  "<dom_formosan_paul_jen_kuei_li_the_preglottalised_sto>": 261532,
+  "<dom_formosan_paul_li_atayalic_final_voiced_stops>": 261533,
+  "<dom_formosan_paul_li_position_atayal_austronesian>": 261534,
+  "<dom_formosan_pingdong_county_government_kai_na_sepuc>": 261535,
+  "<dom_formosan_puyuma_katipol_kopfjagdriten>": 261536,
+  "<dom_formosan_puyuma_teng_reference_grammar>": 261537,
+  "<dom_formosan_raleigh_ferrell_construction_markers_fo>": 261538,
+  "<dom_formosan_rik_bunun>": 261539,
+  "<dom_formosan_robert_blust_the_austronesian_languages>": 261540,
+  "<dom_formosan_saaroa_pan_grammar>": 261541,
+  "<dom_formosan_sakizaya_affixes>": 261542,
+  "<dom_formosan_seals>": 261543,
+  "<dom_formosan_seediq_zhang_reference_grammar>": 261544,
+  "<dom_formosan_shen_wenqi_introduction_sakizaya_gramma>": 261545,
+  "<dom_formosan_shigeru_tsuchida_reconstruction_proto_t>": 261546,
+  "<dom_formosan_shih_rukai_adverbial>": 261547,
+  "<dom_formosan_sing_olam_hong_tinglan_aboriginal_educa>": 261548,
+  "<dom_formosan_song_limei_introduction_kanakanavu_gram>": 261549,
+  "<dom_formosan_song_limei_introduction_seediq_grammar>": 261550,
+  "<dom_formosan_soren_egerod_statement_atayal_phonology>": 261551,
+  "<dom_formosan_stacy_teng_malcom_ross_is_puyuma_primar>": 261552,
+  "<dom_formosan_truku_lowking_demonstratives>": 261553,
+  "<dom_formosan_tsai_hsui_liu_complementation_three_lan>": 261554,
+  "<dom_formosan_tsai_wei_tien_dylan_conjunctive_reducti>": 261555,
+  "<dom_formosan_tsou_descriptive_study>": 261556,
+  "<dom_formosan_tung_tsuchida_ting_li_pan_saaroa_texts>": 261557,
+  "<dom_formosan_wei_huilin_liu_social_structure_yami>": 261558,
+  "<dom_formosan_wilang_yutas_videos>": 261559,
+  "<dom_formosan_wu_chunming_adverbials_in_paiwan>": 261560,
+  "<dom_formosan_wu_jinglan_introduction_amis_grammar>": 261561,
+  "<dom_formosan_xie_fuhui_introduction_kavalan_grammar>": 261562,
+  "<dom_formosan_yedda_palemeq_blog>": 261563,
+  "<dom_formosan_yeddas_blog>": 261564,
+  "<dom_formosan_yeh_meili_introduction_saisiyat_grammar>": 261565,
+  "<dom_formosan_yi_yang_cheng_kanakanavu_word_level_pro>": 261566,
+  "<dom_formosan_yi_yang_cheng_tense_aspect_agent_markin>": 261567,
+  "<dom_formosan_yoshiro_nihira_bunun_vocabulary>": 261568,
+  "<dom_formosan_zeng_shifen_reduplication_affixation_pa>": 261569,
+  "<dom_formosan_zhan_sujuan_taiwan_indigenous_peoples_h>": 261570,
+  "<dom_formosan_zhang_xiujuan_introduction_paiwan_gramm>": 261571,
+  "<dom_formosan_zhang_yongli_pan_jiarong_introduction_t>": 261572,
+  "<dom_formosan_zhao_shanhe_li_taiyuan_zhu_qingyi_abori>": 261573,
+  "<dom_formosan_zheng_acl_2024>": 261574,
+  "<dom_formosan_zheng_data>": 261575,
+  "<dom_formosan_zheng_yumei_zhu_qingyi_bo_hongming_abor>": 261576,
+  "<dom_learning_vocab>": 261577,
+  "<dom_nine_level>": 261578,
+  "<dom_ntu>": 261579,
+  "<dom_picture_book>": 261580,
+  "<dom_picture_story>": 261581,
+  "<dom_presidential_apology>": 261582,
+  "<dom_reading_writing>": 261583,
+  "<dom_unknown>": 261584,
+  "<dom_youtube>": 261585,
+  "<src_ami>": 261586,
+  "<src_bnn>": 261587,
+  "<src_ckv>": 261588,
+  "<src_dru>": 261589,
+  "<src_eng>": 261590,
+  "<src_pwn>": 261591,
+  "<src_pyu>": 261592,
+  "<src_ssf>": 261593,
+  "<src_sxr>": 261594,
+  "<src_szy>": 261595,
+  "<src_tao>": 261596,
+  "<src_tay>": 261597,
+  "<src_trv>": 261598,
+  "<src_tsu>": 261599,
+  "<src_xnb>": 261600,
+  "<src_xsy>": 261601,
+  "<src_zh>": 261602,
+  "<to_ami>": 261603,
+  "<to_bnn>": 261604,
+  "<to_ckv>": 261605,
+  "<to_dru>": 261606,
+  "<to_eng>": 261607,
+  "<to_pwn>": 261608,
+  "<to_pyu>": 261609,
+  "<to_ssf>": 261610,
+  "<to_sxr>": 261611,
+  "<to_szy>": 261612,
+  "<to_tao>": 261613,
+  "<to_tay>": 261614,
+  "<to_trv>": 261615,
+  "<to_tsu>": 261616,
+  "<to_xnb>": 261617,
+  "<to_xsy>": 261618,
+  "<to_zh>": 261619,
+  "ami_Latn": 261395,
+  "bnn_Latn": 261396,
+  "ckv_Latn": 261397,
+  "dru_Latn": 261398,
+  "pwn_Latn": 261399,
+  "pyu_Latn": 261400,
+  "ssf_Latn": 261401,
+  "sxr_Latn": 261402,
+  "szy_Latn": 261403,
+  "tao_Latn": 261404,
+  "tay_Latn": 261405,
+  "trv_Latn": 261406,
+  "tsu_Latn": 261407,
+  "xnb_Latn": 261408,
+  "xsy_Latn": 261409
 }

config.json CHANGED Viewed

@@ -30,5 +30,5 @@
   "tokenizer_class": "NllbTokenizer",
   "transformers_version": "4.56.1",
   "use_cache": true,
-  "vocab_size": 259397
 }

   "tokenizer_class": "NllbTokenizer",
   "transformers_version": "4.56.1",
   "use_cache": true,
+  "vocab_size": 261620
 }

eval/metrics.json CHANGED Viewed

The diff for this file is too large to render. See raw diff

experiment_metadata.json CHANGED Viewed

@@ -1,183 +1,271 @@
 {
-  "step": 270000,
   "best_metric": "chrF2",
-  "best_value": 35.32946603492116,
   "direction": "f2en",
   "validation": {
-    "mean_token_loss": 2.206974470427195,
-    "ppl": 9.088178203326196,
     "by_language": {
-      "ami": 2.0968445140426444,
-      "bnn": 1.7874308878638943,
-      "ckv": 2.087236094678569,
-      "dru": 2.190871106935116,
-      "pwn": 1.9686841008042086,
-      "pyu": 1.6745833827700425,
-      "ssf": 1.9486271316560955,
-      "sxr": 2.4568148167406925,
-      "szy": 1.987393990573504,
-      "tao": 2.2040318672348853,
-      "tay": 1.76902462893994,
-      "trv": 2.325485009128127,
-      "tsu": 3.0212691680022647,
-      "xnb": 2.2393458958157524,
-      "xsy": 2.0707779328778106
     },
     "generation": {
       "samples": 1920,
       "bleu_tokenize": "13a",
       "global": {
-        "BLEU": 19.15281806270227,
-        "chrF2": 35.32946603492116,
-        "TER": 80.93177423809962,
-        "exact_match_rate": 0.06302083333333333,
         "empty_output_rate": 0.0,
-        "character_length_ratio": 0.9119322440413731
       },
       "by_language": {
         "ami": {
           "samples": 128,
-          "BLEU": 15.178042998761052,
-          "chrF2": 32.515086875463034,
-          "TER": 85.50185873605948,
-          "exact_match_rate": 0.0703125,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.8890985042447109
         },
         "bnn": {
           "samples": 128,
-          "BLEU": 21.846742757444925,
-          "chrF2": 40.24080207081615,
-          "TER": 71.03174603174604,
-          "exact_match_rate": 0.078125,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.9059341798393367
         },
         "ckv": {
           "samples": 128,
-          "BLEU": 15.068476827959886,
-          "chrF2": 34.00509723995499,
-          "TER": 81.7174515235457,
-          "exact_match_rate": 0.0625,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.9293617021276596
         },
         "dru": {
           "samples": 128,
-          "BLEU": 14.125242167691509,
-          "chrF2": 32.422397474685745,
-          "TER": 77.08757637474541,
-          "exact_match_rate": 0.0390625,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.8651840188939185
         },
         "pwn": {
           "samples": 128,
-          "BLEU": 15.136340404471213,
-          "chrF2": 31.251019351597947,
-          "TER": 84.12438625204582,
-          "exact_match_rate": 0.0390625,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.9996890547263682
         },
         "pyu": {
           "samples": 128,
-          "BLEU": 21.152070411641255,
-          "chrF2": 36.530982538859824,
-          "TER": 76.31578947368422,
-          "exact_match_rate": 0.1171875,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 1.0212474297464016
         },
         "ssf": {
           "samples": 128,
-          "BLEU": 27.0552108633901,
-          "chrF2": 43.026601689706055,
-          "TER": 75.07407407407408,
-          "exact_match_rate": 0.0859375,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.9658896351903069
         },
         "sxr": {
           "samples": 128,
-          "BLEU": 18.850948946483687,
-          "chrF2": 32.63375669578005,
-          "TER": 85.03937007874016,
-          "exact_match_rate": 0.0546875,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.9307644389071816
         },
         "szy": {
           "samples": 128,
-          "BLEU": 24.628613419366584,
-          "chrF2": 39.013905255321895,
-          "TER": 76.93891557995882,
-          "exact_match_rate": 0.1171875,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.9373006134969325
         },
         "tao": {
           "samples": 128,
-          "BLEU": 23.71371562558303,
-          "chrF2": 36.722354771006046,
-          "TER": 77.53240518482957,
-          "exact_match_rate": 0.0703125,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.8812098830563616
         },
         "tay": {
           "samples": 128,
-          "BLEU": 20.62709252771045,
-          "chrF2": 34.23823716027059,
-          "TER": 75.7700205338809,
-          "exact_match_rate": 0.0703125,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.8556210766596919
         },
         "trv": {
           "samples": 128,
-          "BLEU": 17.109829811557756,
-          "chrF2": 35.524546949231734,
-          "TER": 81.50271575135788,
-          "exact_match_rate": 0.0234375,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.856099994748175
         },
         "tsu": {
           "samples": 128,
-          "BLEU": 10.479400664823114,
-          "chrF2": 29.16918823850146,
-          "TER": 92.50645994832041,
-          "exact_match_rate": 0.046875,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.9196694460988684
         },
         "xnb": {
           "samples": 128,
-          "BLEU": 12.416322568561894,
-          "chrF2": 33.35776470058066,
-          "TER": 83.63064008394544,
-          "exact_match_rate": 0.015625,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.9326793614156569
         },
         "xsy": {
           "samples": 128,
-          "BLEU": 19.625253167391385,
-          "chrF2": 34.701191046456756,
-          "TER": 79.07407407407408,
-          "exact_match_rate": 0.0546875,
           "empty_output_rate": 0.0,
-          "character_length_ratio": 0.906812566184257
         }
       }
     },
-    "step": 270000,
     "selection": {
       "metric": "chrF2",
-      "value": 35.32946603492116,
       "improved": true,
-      "best_value_before_eval": 35.08451950904042
     }
   }
 }

 {
+  "model_family": "nllb",
+  "recipe_id": "nllb200-spm8k-directional-v3",
+  "mt_standardization": {
+    "id": "formosan-mt-standard-v3",
+    "sha256": "4bbded87eb4833b2d1cd5a88a1f2b059560416e0e69ecfa55433a77f418c903e",
+    "namespace": "formosan-mt"
+  },
+  "step": 210000,
   "best_metric": "chrF2",
+  "best_value": 29.69675669457501,
   "direction": "f2en",
+  "run_contract_sha256": "aecd06637deb76f5e92f1f74099539741859863bfdb1ee63bdc2bde875205c09",
   "validation": {
+    "mean_token_loss": 3.3026913455802602,
+    "ppl": 27.18570668278096,
     "by_language": {
+      "ami": 2.7741046558310103,
+      "bnn": 2.4203313634080708,
+      "ckv": 3.8672077235665663,
+      "dru": 2.6078648727627933,
+      "pwn": 2.528691228550362,
+      "pyu": 2.6529325262628793,
+      "ssf": 3.959471394623352,
+      "sxr": 3.549196793319494,
+      "szy": 3.1354670804153746,
+      "tao": 5.619721773664961,
+      "tay": 2.977861526899009,
+      "trv": 3.2536238391231334,
+      "tsu": 2.991666843242918,
+      "xnb": 4.993570713988536,
+      "xsy": 3.4992934907904347
     },
     "generation": {
       "samples": 1920,
       "bleu_tokenize": "13a",
       "global": {
+        "BLEU": 9.527384853249911,
+        "chrF2": 29.69675669457501,
+        "TER": 92.68359895283149,
+        "exact_match_rate": 0.00625,
         "empty_output_rate": 0.0,
+        "character_length_ratio": 1.0418471274597054,
+        "signatures": {
+          "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
+          "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
+          "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+        }
       },
       "by_language": {
         "ami": {
           "samples": 128,
+          "BLEU": 9.238455573040238,
+          "chrF2": 31.95266793741873,
+          "TER": 87.55020080321285,
+          "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 0.9711417816813049,
+          "signatures": {
+            "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
+            "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
+            "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          }
         },
         "bnn": {
           "samples": 128,
+          "BLEU": 15.111661288601976,
+          "chrF2": 37.447343289996,
+          "TER": 94.64285714285714,
+          "exact_match_rate": 0.015625,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 1.1645185746777862,
+          "signatures": {
+            "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
+            "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
+            "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          }
         },
         "ckv": {
           "samples": 128,
+          "BLEU": 8.31432371756987,
+          "chrF2": 28.393994850834748,
+          "TER": 90.07751937984496,
+          "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 1.0116120218579234,
+          "signatures": {
+            "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
+            "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
+            "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          }
         },
         "dru": {
           "samples": 128,
+          "BLEU": 7.778519779287307,
+          "chrF2": 25.795375902173102,
+          "TER": 88.82226980728052,
+          "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 0.918150488225158,
+          "signatures": {
+            "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
+            "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
+            "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          }
         },
         "pwn": {
           "samples": 128,
+          "BLEU": 8.658812231685097,
+          "chrF2": 32.22194063034251,
+          "TER": 89.4127732533219,
+          "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 1.0827989963758016,
+          "signatures": {
+            "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
+            "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
+            "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          }
         },
         "pyu": {
           "samples": 128,
+          "BLEU": 16.483653914447636,
+          "chrF2": 37.10931345333428,
+          "TER": 80.96856414613424,
+          "exact_match_rate": 0.0078125,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 1.1018188636789799,
+          "signatures": {
+            "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
+            "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
+            "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          }
         },
         "ssf": {
           "samples": 128,
+          "BLEU": 5.545373975940428,
+          "chrF2": 28.047744106886313,
+          "TER": 109.63855421686748,
+          "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 1.2119829235978212,
+          "signatures": {
+            "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
+            "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
+            "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          }
         },
         "sxr": {
           "samples": 128,
+          "BLEU": 7.707769211359447,
+          "chrF2": 29.688050193906157,
+          "TER": 97.93991416309012,
+          "exact_match_rate": 0.0078125,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 1.0473008616142079,
+          "signatures": {
+            "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
+            "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
+            "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          }
         },
         "szy": {
           "samples": 128,
+          "BLEU": 10.566286844226358,
+          "chrF2": 31.689864087527926,
+          "TER": 89.02517753389283,
+          "exact_match_rate": 0.0078125,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 1.0423351759471484,
+          "signatures": {
+            "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
+            "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
+            "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          }
         },
         "tao": {
           "samples": 128,
+          "BLEU": 1.5669790931362446,
+          "chrF2": 25.267381629423703,
+          "TER": 122.75600505689002,
+          "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 1.2265066394279878,
+          "signatures": {
+            "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
+            "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
+            "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          }
         },
         "tay": {
           "samples": 128,
+          "BLEU": 7.238946215102692,
+          "chrF2": 24.056495084442492,
+          "TER": 87.7221324717286,
+          "exact_match_rate": 0.015625,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 0.827789393651397,
+          "signatures": {
+            "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
+            "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
+            "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          }
         },
         "trv": {
           "samples": 128,
+          "BLEU": 9.010710983806707,
+          "chrF2": 29.99822177170911,
+          "TER": 95.42947202521671,
+          "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 1.1590753424657534,
+          "signatures": {
+            "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
+            "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
+            "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          }
         },
         "tsu": {
           "samples": 128,
+          "BLEU": 14.586628714061831,
+          "chrF2": 30.754865405687898,
+          "TER": 77.6869893148963,
+          "exact_match_rate": 0.0234375,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 0.9626507775032699,
+          "signatures": {
+            "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
+            "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
+            "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          }
         },
         "xnb": {
           "samples": 128,
+          "BLEU": 5.352878317943197,
+          "chrF2": 23.01127510854248,
+          "TER": 109.5948827292111,
+          "exact_match_rate": 0.0,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 1.0608658982288446,
+          "signatures": {
+            "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
+            "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
+            "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          }
         },
         "xsy": {
           "samples": 128,
+          "BLEU": 12.065210750445159,
+          "chrF2": 30.269374379711817,
+          "TER": 89.06942392909897,
+          "exact_match_rate": 0.015625,
           "empty_output_rate": 0.0,
+          "character_length_ratio": 0.9753761969904241,
+          "signatures": {
+            "BLEU": "nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1",
+            "chrF2": "nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1",
+            "TER": "nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1"
+          }
         }
       }
     },
+    "step": 210000,
     "selection": {
       "metric": "chrF2",
+      "value": 29.69675669457501,
       "improved": true,
+      "best_value_before_eval": 29.56158400843854
     }
   }
 }

formosan_mt_inference.py ADDED Viewed

	@@ -0,0 +1,79 @@

+#!/usr/bin/env python3
+"""Apply the corpus MT standardization profile to Formosan inference input."""
+from __future__ import annotations
+import argparse
+import sys
+from pathlib import Path
+SCRIPT_DIR = Path(__file__).resolve().parent
+PROJECT_ROOT = SCRIPT_DIR.parents[1]
+LOCAL_SCRIPTS = PROJECT_ROOT / "scripts" / "local"
+if LOCAL_SCRIPTS.is_dir():
+    sys.path.insert(0, str(LOCAL_SCRIPTS))
+from mt_standardization import (  # noqa: E402
+    StandardizationContext,
+    assert_idempotent,
+    load_profile,
+    standardize_text,
+)
+PACKAGED_PROFILE = SCRIPT_DIR / "mt_standardization_profile.json"
+PROJECT_PROFILE = PROJECT_ROOT / "config" / "mt_standardization.json"
+def default_profile_path() -> Path:
+    if PACKAGED_PROFILE.is_file():
+        return PACKAGED_PROFILE
+    return PROJECT_PROFILE
+def normalize_formosan(
+    text: str,
+    lang_code: str,
+    *,
+    row_type: str = "sentence",
+    source_repository: str = "inference",
+    source_path: str = "interactive",
+    profile_path: Path | None = None,
+) -> str:
+    """Return model-ready Formosan text or reject unresolved notation."""
+    path = (profile_path or default_profile_path()).expanduser().resolve()
+    profile = load_profile(path)
+    context = StandardizationContext(
+        language=lang_code.strip().lower(),
+        row_type=row_type,
+        repository=source_repository,
+        xml_path=source_path,
+    )
+    result = standardize_text(text, context=context, profile=profile)
+    if result.status != "accepted":
+        raise ValueError(
+            "Formosan input is not safe to normalize automatically: "
+            f"{result.reason or result.status}"
+        )
+    assert_idempotent(result, context=context, profile=profile)
+    return result.text
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("text")
+    parser.add_argument("--lang-code", required=True)
+    parser.add_argument("--row-type", default="sentence")
+    parser.add_argument("--profile", type=Path)
+    args = parser.parse_args()
+    print(
+        normalize_formosan(
+            args.text,
+            args.lang_code,
+            row_type=args.row_type,
+            profile_path=args.profile,
+        )
+    )
+if __name__ == "__main__":
+    main()

model.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a17b04a6aa8fb520fac5be7e2c83fd50781ed625502651fc65be6c5e79ee7f2c
-size 2473425248

 version https://git-lfs.github.com/spec/v1
+oid sha256:105bb37a848f1fedac34391abbe11e2caa185edf9b97b7325b871362f7b11f1c
+size 2482530656

mt_standardization.py ADDED Viewed

	@@ -0,0 +1,458 @@

+"""Deterministic, source-aware Formosan standardization for MT corpora."""
+from __future__ import annotations
+import hashlib
+import html
+import json
+import re
+import unicodedata
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Iterable
+PROJECT_ROOT = Path(__file__).resolve().parents[2]
+DEFAULT_PROFILE_PATH = PROJECT_ROOT / "config" / "mt_standardization.json"
+CONTROL_RE = re.compile(r"[\x00-\x08\x0B\x0C\x0E-\x1F\x7F]")
+FORMAT_CHARACTERS = frozenset("\u200b\u200c\u200d\ufeff")
+WHITESPACE_RE = re.compile(r"\s+")
+WIKI_HEADING_RE = re.compile(r"(^|\s)==+\s*([^=\n]+?)\s*==+(?=\s|$)")
+ANGLE_CONTENT_RE = re.compile(r"<([^<>\s]{1,24})>")
+BRACE_CONTENT_RE = re.compile(r"\{([^{}\s]{1,24})}")
+OPTIONAL_CONTENT_RE = re.compile(r"\(([^()\s]{1,16})\)")
+SIMPLE_ALTERNATIVE_RE = re.compile(r"(?<!\S)([^\s/]+)\s*/\s*([^\s/]+)(?!\S)")
+SPACED_TILDE_RE = re.compile(r"(?<!\S)([^\s~]+)\s+~\s+([^\s~]+)(?!\S)")
+SPEAKER_PREFIX_RE = re.compile(r"^([A-Z][A-Za-z0-9_. -]{0,31}):\s+")
+URL_RE = re.compile(r"(?:https?://|www\.)", re.IGNORECASE)
+SUPERSCRIPT_FOOTNOTE_RE = re.compile(r"(?<=\D)[⁰¹²³⁴⁵⁶⁷⁸⁹]+(?=\s|$)")
+EXERCISE_BLANK_RE = re.compile(r"_{2,}")
+@dataclass(frozen=True)
+class StandardizationContext:
+    language: str
+    row_type: str
+    repository: str
+    xml_path: str
+@dataclass(frozen=True)
+class StandardizationResult:
+    text: str
+    status: str
+    confidence: str
+    eval_eligible: bool
+    transformations: tuple[dict[str, Any], ...]
+    unresolved_markers: tuple[str, ...]
+    reason: str = ""
+    speaker_label: str = ""
+def load_profile(path: Path = DEFAULT_PROFILE_PATH) -> dict[str, Any]:
+    try:
+        profile = json.loads(path.read_text(encoding="utf-8"))
+    except (OSError, json.JSONDecodeError) as exc:
+        raise SystemExit(f"Cannot load MT standardization profile {path}: {exc}") from exc
+    if profile.get("schema_version") != 1:
+        raise SystemExit(f"Unsupported MT standardization profile schema: {profile.get('schema_version')}")
+    if not str(profile.get("profile_id") or "").strip():
+        raise SystemExit(f"MT standardization profile has no profile_id: {path}")
+    if profile.get("unicode_normalization") != "NFC":
+        raise SystemExit("MT standardization currently requires NFC")
+    expected_implementation = str(
+        profile.get("implementation_sha256") or ""
+    )
+    actual_implementation = hashlib.sha256(
+        Path(__file__).read_bytes()
+    ).hexdigest()
+    if expected_implementation != actual_implementation:
+        raise SystemExit(
+            "MT standardization implementation hash mismatch: "
+            f"profile={expected_implementation or '<missing>'}, "
+            f"loaded={actual_implementation}"
+        )
+    if not isinstance(profile.get("policy"), dict):
+        raise SystemExit(f"MT standardization profile has no policy mapping: {path}")
+    if not isinstance(profile.get("source_overrides", []), list):
+        raise SystemExit(f"MT standardization source_overrides must be a list: {path}")
+    return profile
+def profile_sha256(path: Path = DEFAULT_PROFILE_PATH) -> str:
+    return hashlib.sha256(path.read_bytes()).hexdigest()
+def is_orthographic_character(character: str) -> bool:
+    if not character:
+        return False
+    return unicodedata.category(character)[0] in {"L", "M", "N"} or character in {"'", "’", "ʼ"}
+def insertion_content_is_safe(value: str) -> bool:
+    return bool(value) and all(is_orthographic_character(character) for character in value)
+def source_matches(selector: dict[str, Any], context: StandardizationContext) -> bool:
+    for key, actual in (
+        ("language", context.language),
+        ("row_type", context.row_type),
+        ("repository", context.repository),
+    ):
+        expected = str(selector.get(key) or "*")
+        if expected not in {"*", actual}:
+            return False
+    path_pattern = str(selector.get("path_regex") or "")
+    return not path_pattern or re.search(path_pattern, context.xml_path) is not None
+def effective_policy(profile: dict[str, Any], context: StandardizationContext) -> tuple[dict[str, Any], bool]:
+    policy = dict(profile["policy"])
+    reviewed_ambiguous = False
+    for override in profile.get("source_overrides", []):
+        if not isinstance(override, dict) or not source_matches(override, context):
+            continue
+        values = override.get("policy", {})
+        if not isinstance(values, dict):
+            raise ValueError(f"Source override policy must be a mapping: {override}")
+        policy.update(values)
+        reviewed_ambiguous = reviewed_ambiguous or bool(override.get("reviewed_ambiguous"))
+    return policy, reviewed_ambiguous
+class _StandardizationRun:
+    def __init__(self, text: str) -> None:
+        self.text = text
+        self.transformations: list[dict[str, Any]] = []
+        self.ambiguous = False
+        self.reason = ""
+        self.speaker_label = ""
+    def replace(self, rule: str, new_text: str, *, count: int = 1, ambiguous: bool = False) -> None:
+        if new_text == self.text:
+            return
+        self.text = new_text
+        self.transformations.append({"rule": rule, "count": int(max(count, 1))})
+        self.ambiguous = self.ambiguous or ambiguous
+    def regex_sub(
+        self,
+        rule: str,
+        pattern: re.Pattern[str],
+        replacement: str | Any,
+        *,
+        ambiguous: bool = False,
+    ) -> None:
+        new_text, count = pattern.subn(replacement, self.text)
+        if count:
+            self.replace(rule, new_text, count=count, ambiguous=ambiguous)
+def unwrap_infixes(run: _StandardizationRun, pattern: re.Pattern[str], rule: str) -> None:
+    count = 0
+    def replacement(match: re.Match[str]) -> str:
+        nonlocal count
+        content = match.group(1)
+        if not insertion_content_is_safe(content):
+            return match.group(0)
+        count += 1
+        return content
+    new_text = pattern.sub(replacement, run.text)
+    if count:
+        run.replace(rule, new_text, count=count)
+def include_optional_segments(run: _StandardizationRun) -> None:
+    original = run.text
+    working = original
+    total = 0
+    while True:
+        count = 0
+        def replacement(match: re.Match[str]) -> str:
+            nonlocal count
+            start, end = match.span()
+            previous = working[start - 1] if start else ""
+            following = working[end] if end < len(working) else ""
+            content = match.group(1)
+            previous_is_boundary = (
+                is_orthographic_character(previous)
+                or previous in {"-", "=", "+", "~"}
+            )
+            following_is_boundary = (
+                is_orthographic_character(following)
+                or following in {"-", "=", "+", "~"}
+            )
+            if not (
+                insertion_content_is_safe(content)
+                and previous_is_boundary
+                and following_is_boundary
+            ):
+                return match.group(0)
+            count += 1
+            return content
+        updated = OPTIONAL_CONTENT_RE.sub(replacement, working)
+        if not count:
+            break
+        total += count
+        working = updated
+    if total:
+        run.replace(
+            "include_optional_intraword_segment",
+            working,
+            count=total,
+            ambiguous=True,
+        )
+def remove_boundary_character(run: _StandardizationRun, marker: str, rule: str) -> None:
+    output: list[str] = []
+    removed = 0
+    index = 0
+    while index < len(run.text):
+        character = run.text[index]
+        if character != marker:
+            output.append(character)
+            index += 1
+            continue
+        end = index + 1
+        while end < len(run.text) and run.text[end] == marker:
+            end += 1
+        previous = run.text[index - 1] if index else ""
+        following = run.text[end] if end < len(run.text) else ""
+        if (is_orthographic_character(previous) or is_orthographic_character(following)) and not (
+            previous.isdigit() and following.isdigit()
+        ):
+            removed += end - index
+            index = end
+            continue
+        output.append(run.text[index:end])
+        index = end
+    if removed:
+        run.replace(rule, "".join(output), count=removed)
+def normalize_morphological_boundaries(
+    run: _StandardizationRun,
+    policy: dict[str, Any],
+) -> None:
+    while True:
+        before = run.text
+        if policy.get("remove_clitic_boundaries") and "=" in run.text:
+            count = run.text.count("=")
+            run.replace(
+                "remove_clitic_boundary",
+                run.text.replace("=", ""),
+                count=count,
+            )
+        if policy.get("remove_intraword_hyphens"):
+            remove_boundary_character(
+                run,
+                "-",
+                "remove_hyphen_boundary",
+            )
+        if policy.get("remove_intraword_tildes"):
+            remove_boundary_character(
+                run,
+                "~",
+                "remove_tilde_boundary",
+            )
+        if policy.get("remove_intraword_pluses"):
+            remove_boundary_character(
+                run,
+                "+",
+                "remove_plus_boundary",
+            )
+        if policy.get("include_optional_intraword_segments"):
+            include_optional_segments(run)
+        if run.text == before:
+            return
+        if len(run.text) >= len(before):
+            raise ValueError(
+                "Morphological normalization did not strictly reduce text"
+            )
+def select_simple_alternatives(run: _StandardizationRun) -> None:
+    if URL_RE.search(run.text):
+        run.reason = "url_in_formosan_standard"
+        return
+    slash_count = run.text.count("/")
+    if not slash_count:
+        return
+    if slash_count > 4:
+        run.reason = "complex_slash_notation"
+        return
+    working = run.text
+    total = 0
+    while True:
+        working, count = SIMPLE_ALTERNATIVE_RE.subn(lambda match: match.group(1), working)
+        total += count
+        if not count:
+            break
+    if total:
+        run.replace("select_first_slash_alternative", working, count=total, ambiguous=True)
+    if "/" in run.text:
+        run.reason = "unresolved_slash_notation"
+def select_tilde_alternatives(run: _StandardizationRun) -> None:
+    working = run.text
+    total = 0
+    while True:
+        working, count = SPACED_TILDE_RE.subn(lambda match: match.group(1), working)
+        total += count
+        if not count:
+            break
+    if total:
+        run.replace("select_first_tilde_alternative", working, count=total, ambiguous=True)
+def unresolved_markers(text: str, configured: Iterable[str]) -> tuple[str, ...]:
+    markers = {marker for marker in configured if marker and marker in text}
+    if EXERCISE_BLANK_RE.search(text):
+        markers.add("exercise_blank")
+    return tuple(sorted(markers))
+def standardize_text(
+    value: object,
+    *,
+    context: StandardizationContext,
+    profile: dict[str, Any],
+    contains_unclear: bool = False,
+) -> StandardizationResult:
+    raw = "" if value is None else str(value)
+    if contains_unclear:
+        return StandardizationResult(
+            text="",
+            status="ineligible",
+            confidence="none",
+            eval_eligible=False,
+            transformations=(),
+            unresolved_markers=(),
+            reason="contains_unclear",
+        )
+    if not raw.strip():
+        return StandardizationResult(
+            text="",
+            status="ineligible",
+            confidence="none",
+            eval_eligible=False,
+            transformations=(),
+            unresolved_markers=(),
+            reason="empty_source_standard",
+        )
+    policy, reviewed_ambiguous = effective_policy(profile, context)
+    run = _StandardizationRun(raw)
+    controls = CONTROL_RE.sub(" ", run.text)
+    controls = "".join("" if character in FORMAT_CHARACTERS else character for character in controls)
+    run.replace("remove_control_and_format_characters", controls)
+    run.replace("html_unescape", html.unescape(run.text))
+    run.replace("unicode_nfc", unicodedata.normalize("NFC", run.text))
+    run.replace(
+        "normalize_whitespace",
+        WHITESPACE_RE.sub(" ", run.text).strip(),
+    )
+    if policy.get("strip_wiki_heading_markup"):
+        run.regex_sub("strip_wiki_heading_markup", WIKI_HEADING_RE, lambda match: f"{match.group(1)}{match.group(2)}")
+    if policy.get("remove_null_morphemes"):
+        for marker in ("Ø-", "∅-", "-Ø", "-∅", "Ø", "∅"):
+            count = run.text.count(marker)
+            if count:
+                run.replace("remove_null_morpheme", run.text.replace(marker, ""), count=count)
+    if policy.get("unwrap_letter_infixes"):
+        unwrap_infixes(run, ANGLE_CONTENT_RE, "unwrap_angle_infix")
+        unwrap_infixes(run, BRACE_CONTENT_RE, "unwrap_braced_morpheme")
+    # Remove annotation characters before boundary and alternative handling so
+    # cleanup cannot expose notation that is processed only on a later pass.
+    if "_" in run.text and not EXERCISE_BLANK_RE.search(run.text):
+        count = run.text.count("_")
+        run.replace("remove_annotation_underscore", run.text.replace("_", ""), count=count, ambiguous=True)
+    if "\\" in run.text:
+        count = run.text.count("\\")
+        run.replace("remove_escape_backslash", run.text.replace("\\", ""), count=count)
+    normalize_morphological_boundaries(run, policy)
+    if policy.get("select_first_simple_alternative"):
+        select_simple_alternatives(run)
+        select_tilde_alternatives(run)
+    if policy.get("strip_trailing_superscript_footnotes"):
+        run.regex_sub("strip_superscript_footnote", SUPERSCRIPT_FOOTNOTE_RE, "")
+    squeezed = WHITESPACE_RE.sub(" ", run.text).strip()
+    run.replace("normalize_whitespace", squeezed)
+    speaker = SPEAKER_PREFIX_RE.match(run.text)
+    if speaker:
+        run.speaker_label = speaker.group(1)
+    markers = unresolved_markers(run.text, profile.get("unresolved_annotation_characters", []))
+    reason = run.reason
+    if markers and not reason:
+        reason = "unresolved_annotation_markers"
+    if not run.text or not any(unicodedata.category(character)[0] in {"L", "M", "N"} for character in run.text):
+        return StandardizationResult(
+            text=run.text,
+            status="ineligible",
+            confidence="none",
+            eval_eligible=False,
+            transformations=tuple(run.transformations),
+            unresolved_markers=markers,
+            reason="empty_or_nonlinguistic_after_standardization",
+            speaker_label=run.speaker_label,
+        )
+    if reason or markers:
+        return StandardizationResult(
+            text=run.text,
+            status="quarantine",
+            confidence="ambiguous",
+            eval_eligible=False,
+            transformations=tuple(run.transformations),
+            unresolved_markers=markers,
+            reason=reason,
+            speaker_label=run.speaker_label,
+        )
+    confidence = "ambiguous" if run.ambiguous else ("safe" if run.transformations else "unchanged")
+    return StandardizationResult(
+        text=run.text,
+        status="accepted",
+        confidence=confidence,
+        eval_eligible=(context.row_type == "sentence" and (not run.ambiguous or reviewed_ambiguous)),
+        transformations=tuple(run.transformations),
+        unresolved_markers=(),
+        speaker_label=run.speaker_label,
+    )
+def assert_idempotent(
+    result: StandardizationResult,
+    *,
+    context: StandardizationContext,
+    profile: dict[str, Any],
+) -> None:
+    if result.status != "accepted":
+        return
+    repeated = standardize_text(result.text, context=context, profile=profile)
+    if (
+        repeated.text != result.text
+        or repeated.status != "accepted"
+        or repeated.transformations
+        or repeated.speaker_label != result.speaker_label
+    ):
+        raise ValueError(
+            f"MT standardization is not idempotent for {context.repository}/{context.xml_path}: "
+            f"{result.text!r} -> {repeated.text!r}"
+        )

mt_standardization_profile.json ADDED Viewed

	@@ -0,0 +1,32 @@

+{
+  "schema_version": 1,
+  "profile_id": "formosan-mt-standard-v3",
+  "implementation_sha256": "c78f092f826aed70468bb3d4276eb27bab482dc5dc36dedf9ed5726596e0dd31",
+  "unicode_normalization": "NFC",
+  "policy": {
+    "unwrap_letter_infixes": true,
+    "remove_clitic_boundaries": true,
+    "remove_null_morphemes": true,
+    "remove_intraword_hyphens": true,
+    "remove_intraword_tildes": true,
+    "remove_intraword_pluses": true,
+    "include_optional_intraword_segments": true,
+    "select_first_simple_alternative": true,
+    "strip_wiki_heading_markup": true,
+    "strip_trailing_superscript_footnotes": true
+  },
+  "source_overrides": [],
+  "unresolved_annotation_characters": [
+    "<",
+    ">",
+    "=",
+    "Ø",
+    "∅",
+    "*",
+    "_",
+    "{",
+    "}",
+    "|",
+    "\\"
+  ]
+}

publication.json ADDED Viewed

	@@ -0,0 +1,30 @@

+{
+  "schema_version": 1,
+  "repo_id": "FormosanBank/nllb200-formosan-en-spm8k",
+  "direction": "f2en",
+  "run_stamp": "20260809-210523",
+  "checkpoint": "best",
+  "checkpoint_step": 210000,
+  "corpus": {
+    "name": "private_no_bible",
+    "target_lang": "english",
+    "rows": 657530,
+    "splits": {
+      "test": 63140,
+      "train": 573657,
+      "validate": 20733
+    },
+    "sha256": "e3feeaf7c3c51b9cd5c4b0537ffe44a370e7d02723f34b467479b6c4f5f0ea77"
+  },
+  "profile_sha256": "34f45832bdedc1b8322b26936dd88667dfecb360d941a7f54f60ae31a1fb4e10",
+  "base_model": {
+    "name": "facebook/nllb-200-distilled-600M",
+    "revision": "f8d333a098d19b4fd9a8b18f94170487ad3f821d"
+  },
+  "mt_standardization": {
+    "id": "formosan-mt-standard-v3",
+    "sha256": "4bbded87eb4833b2d1cd5a88a1f2b059560416e0e69ecfa55433a77f418c903e",
+    "namespace": "formosan-mt"
+  },
+  "run_contract_sha256": "aecd06637deb76f5e92f1f74099539741859863bfdb1ee63bdc2bde875205c09"
+}

sentencepiece.bpe.model CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4fdb786ba0b0f8f103583a044c1c9d232707d2178f94016baebdde9ac5bd60ff
-size 4903201

 version https://git-lfs.github.com/spec/v1
+oid sha256:13e0177da34cfe2c2ee36e49235ffa494edf17a5d62453cca71f6cfee794b8e4
+size 4951530

special_tokens_map.json CHANGED Viewed

@@ -217,8 +217,6 @@
     "tsu_Latn",
     "xnb_Latn",
     "xsy_Latn",
-    "<mask>",
-    "<dae>",
     "<dialect_central>",
     "<dialect_coastal>",
     "<dialect_dawu>",
@@ -265,81 +263,127 @@
     "<dom_culture>",
     "<dom_dictionary>",
     "<dom_essays>",
-    "<dom_ethnography_journal_ritual_song>",
     "<dom_formosan_100_paiwan_texts>",
     "<dom_formosan_academia_sinica_oral_legends>",
     "<dom_formosan_amis_adversative_constructions>",
     "<dom_formosan_amis_kavalan_lin_interrogative_verbs>",
     "<dom_formosan_amis_kuo_sung_comparative_constructions>",
     "<dom_formosan_amis_lifok_dongi_saopo_kimakimad>",
     "<dom_formosan_amis_myths_and_customs>",
     "<dom_formosan_amis_pa_verbs>",
-    "<dom_formosan_amis_serial_verb_constructions>",
-    "<dom_formosan_amis_tung_chiou>",
-    "<dom_formosan_asai_sedik_language>",
     "<dom_formosan_bunun_debusser_dissertation>",
-    "<dom_formosan_bunun_moriguchi_northern_texts>",
     "<dom_formosan_bunun_topic_focus>",
-    "<dom_formosan_chen_fukuda_relabeling_ergative>",
-    "<dom_formosan_chen_fukuda_three_ways_steal>",
     "<dom_formosan_cip_atayal_grammar_overview>",
     "<dom_formosan_dean_johnson_year_clouds_smangus>",
-    "<dom_formosan_dorinda_tsai_hsiu_liu_neutral_imperfect>",
-    "<dom_formosan_egerod_origin_headhunting_atayal_text_v>",
-    "<dom_formosan_egerod_soren_word_order_word_classes_at>",
     "<dom_formosan_elizabeth_zeitoun_chen_huei_wu_overview>",
     "<dom_formosan_epark>",
-    "<dom_formosan_ferrell_taiwan_aboriginal_groups_proble>",
     "<dom_formosan_gitbook_translations>",
     "<dom_formosan_glosbe>",
-    "<dom_formosan_hala_saku_la_videos>",
     "<dom_formosan_holmer_seediq>",
     "<dom_formosan_huang_grammaticalization_squliq_atayal>",
     "<dom_formosan_huang_mei_chin_atayal_reference_grammar>",
     "<dom_formosan_huteson_rukai_survey>",
     "<dom_formosan_ilrdf_42_language_practice_word_lists>",
     "<dom_formosan_ilrdf_tousvusvutu_kita>",
     "<dom_formosan_indigenous_language_literary_awards>",
     "<dom_formosan_kanakanavu_texts>",
     "<dom_formosan_kavalan_zhang_reference_grammar>",
     "<dom_formosan_li_peirong_xu_weisheng_introduction_tru>",
     "<dom_formosan_li_peizhen_wu_xingyu_shouhu_xing>",
     "<dom_formosan_naomi_tsukida_correlative_clauses_seedi>",
     "<dom_formosan_nowbucyang_truku_thesis>",
     "<dom_formosan_old_texts>",
     "<dom_formosan_paiwan_collart_zeitoun_time_reference>",
     "<dom_formosan_paiwan_ho_five_dialects>",
     "<dom_formosan_paiwanstories>",
-    "<dom_formosan_puyuma_chen_raising_to_object>",
     "<dom_formosan_puyuma_katipol_kopfjagdriten>",
-    "<dom_formosan_puyuma_teng_pronominal_systems>",
     "<dom_formosan_puyuma_teng_reference_grammar>",
     "<dom_formosan_rik_bunun>",
-    "<dom_formosan_robert_blust_austronesian_homeland_ling>",
-    "<dom_formosan_rukai_mantauran_stories>",
-    "<dom_formosan_rukai_texts>",
-    "<dom_formosan_rukai_zeitoun_saying>",
     "<dom_formosan_saaroa_pan_grammar>",
     "<dom_formosan_sakizaya_affixes>",
     "<dom_formosan_seals>",
     "<dom_formosan_seediq_zhang_reference_grammar>",
-    "<dom_formosan_shigeru_tsuchida_kanakanavu_texts>",
     "<dom_formosan_shigeru_tsuchida_reconstruction_proto_t>",
     "<dom_formosan_shih_rukai_adverbial>",
     "<dom_formosan_song_limei_introduction_kanakanavu_gram>",
     "<dom_formosan_song_limei_introduction_seediq_grammar>",
-    "<dom_formosan_sung_kuo_descriptive_comparative_constr>",
-    "<dom_formosan_taoshan_elementary_school_atelier_hui_k>",
-    "<dom_formosan_tsukida_naomi_verb_classification_amis>",
-    "<dom_formosan_wei_matri_clan_lineage_system_ami>",
     "<dom_formosan_wilang_yutas_videos>",
     "<dom_formosan_wu_jinglan_introduction_amis_grammar>",
     "<dom_formosan_yedda_palemeq_blog>",
     "<dom_formosan_yeddas_blog>",
     "<dom_formosan_yeh_meili_introduction_saisiyat_grammar>",
     "<dom_formosan_zhang_yongli_pan_jiarong_introduction_t>",
     "<dom_formosan_zheng_acl_2024>",
     "<dom_formosan_zheng_data>",
     "<dom_learning_vocab>",
     "<dom_nine_level>",
     "<dom_ntu>",
@@ -407,8 +451,8 @@
   },
   "mask_token": {
     "content": "<mask>",
-    "lstrip": false,
-    "normalized": false,
     "rstrip": false,
     "single_word": false
   },

     "tsu_Latn",
     "xnb_Latn",
     "xsy_Latn",
     "<dialect_central>",
     "<dialect_coastal>",
     "<dialect_dawu>",
     "<dom_culture>",
     "<dom_dictionary>",
     "<dom_essays>",
     "<dom_formosan_100_paiwan_texts>",
     "<dom_formosan_academia_sinica_oral_legends>",
     "<dom_formosan_amis_adversative_constructions>",
+    "<dom_formosan_amis_huang_english_phonology_grammar>",
     "<dom_formosan_amis_kavalan_lin_interrogative_verbs>",
     "<dom_formosan_amis_kuo_sung_comparative_constructions>",
     "<dom_formosan_amis_lifok_dongi_saopo_kimakimad>",
     "<dom_formosan_amis_myths_and_customs>",
     "<dom_formosan_amis_pa_verbs>",
+    "<dom_formosan_anna_hsiou_chuan_chang_reference_gramma>",
+    "<dom_formosan_anton_quack_puyuma_pulingaw>",
+    "<dom_formosan_atayal_you_nao_savak_raxayal>",
+    "<dom_formosan_boyizhenu_narrative_oral_literature_tap>",
     "<dom_formosan_bunun_debusser_dissertation>",
     "<dom_formosan_bunun_topic_focus>",
+    "<dom_formosan_catherine_tseng_seediq_atayal_foods_the>",
+    "<dom_formosan_chen_yaohan_miss_virginia_fey_played_es>",
+    "<dom_formosan_chun_mei_chen_comparative_phonology_pai>",
+    "<dom_formosan_chunming_wu_two_types_of_noun_incorpora>",
     "<dom_formosan_cip_atayal_grammar_overview>",
+    "<dom_formosan_claire_mcgill_a_brief_tayal_vocabulary>",
+    "<dom_formosan_david_blundell_ed_austronesian_taiwan>",
     "<dom_formosan_dean_johnson_year_clouds_smangus>",
+    "<dom_formosan_deng_fangqing_introduction_puyuma_gramm>",
+    "<dom_formosan_der_hwa_victoria_rau_grammar_atayal>",
+    "<dom_formosan_dong_manv_yami_ode_to_taro>",
+    "<dom_formosan_dong_yijia_tjuwabar_paiwan_ritual_langu>",
     "<dom_formosan_elizabeth_zeitoun_chen_huei_wu_overview>",
+    "<dom_formosan_elizabeth_zeitoun_lillian_huang_marie_y>",
+    "<dom_formosan_elizabeth_zeitoun_pronominal_system_man>",
+    "<dom_formosan_elizabeth_zeitoun_squib_dynamic_vs_stat>",
     "<dom_formosan_epark>",
     "<dom_formosan_gitbook_translations>",
     "<dom_formosan_glosbe>",
+    "<dom_formosan_ho_dan_phonological_system_butonglu_pai>",
     "<dom_formosan_holmer_seediq>",
+    "<dom_formosan_hsiu_chuan_liao_transitivity_ergativity>",
     "<dom_formosan_huang_grammaticalization_squliq_atayal>",
+    "<dom_formosan_huang_hui_chuan_glide_formation_takitud>",
+    "<dom_formosan_huang_huijuan_shi_chaokai_introduction>",
     "<dom_formosan_huang_mei_chin_atayal_reference_grammar>",
     "<dom_formosan_huteson_rukai_survey>",
     "<dom_formosan_ilrdf_42_language_practice_word_lists>",
     "<dom_formosan_ilrdf_tousvusvutu_kita>",
+    "<dom_formosan_indigenous_education_resource_center_ke>",
     "<dom_formosan_indigenous_language_literary_awards>",
+    "<dom_formosan_isabelle_bril_roots_and_stems_lexical_a>",
+    "<dom_formosan_jian_iwan_guo_qingliu_life_history>",
+    "<dom_formosan_jian_shilang_introduction_thao_grammar>",
+    "<dom_formosan_john_u_wolff_the_proto_austronesian_pho>",
+    "<dom_formosan_josiane_cauquelin_aborigines_taiwan_puy>",
     "<dom_formosan_kanakanavu_texts>",
     "<dom_formosan_kavalan_zhang_reference_grammar>",
+    "<dom_formosan_kolas_foting_aboriginal_education_world>",
+    "<dom_formosan_kucapungane>",
+    "<dom_formosan_kumu_tapas_tribal_memory_oral_history_w>",
+    "<dom_formosan_lei_shih_family_system_paiwan_at_su_pai>",
+    "<dom_formosan_li_may_sung_budai_rukai_exclamatives>",
+    "<dom_formosan_li_may_sung_clausal_nominalization_buda>",
     "<dom_formosan_li_peirong_xu_weisheng_introduction_tru>",
     "<dom_formosan_li_peizhen_wu_xingyu_shouhu_xing>",
+    "<dom_formosan_lillian_huang_atayal_participants_cross>",
+    "<dom_formosan_liu_manyi_kaskun_ata_matas_i_ki_quaz>",
+    "<dom_formosan_malcom_ross_proto_austronesian_verbal_m>",
+    "<dom_formosan_maya_yeh_blaq_uv_construction_atayal>",
+    "<dom_formosan_nanang_tadaw_naci_mowna_pgagu>",
     "<dom_formosan_naomi_tsukida_correlative_clauses_seedi>",
     "<dom_formosan_nowbucyang_truku_thesis>",
+    "<dom_formosan_ochiai_izumi_numerals_paran_seediq_rela>",
     "<dom_formosan_old_texts>",
     "<dom_formosan_paiwan_collart_zeitoun_time_reference>",
     "<dom_formosan_paiwan_ho_five_dialects>",
     "<dom_formosan_paiwanstories>",
+    "<dom_formosan_patricia_stanley_morphophonemics_of_ver>",
+    "<dom_formosan_paul_jen_kuei_li_internal_relationships>",
+    "<dom_formosan_paul_jen_kuei_li_rukai_structure>",
+    "<dom_formosan_paul_jen_kuei_li_the_preglottalised_sto>",
+    "<dom_formosan_paul_li_atayalic_final_voiced_stops>",
+    "<dom_formosan_paul_li_position_atayal_austronesian>",
+    "<dom_formosan_pingdong_county_government_kai_na_sepuc>",
     "<dom_formosan_puyuma_katipol_kopfjagdriten>",
     "<dom_formosan_puyuma_teng_reference_grammar>",
+    "<dom_formosan_raleigh_ferrell_construction_markers_fo>",
     "<dom_formosan_rik_bunun>",
+    "<dom_formosan_robert_blust_the_austronesian_languages>",
     "<dom_formosan_saaroa_pan_grammar>",
     "<dom_formosan_sakizaya_affixes>",
     "<dom_formosan_seals>",
     "<dom_formosan_seediq_zhang_reference_grammar>",
+    "<dom_formosan_shen_wenqi_introduction_sakizaya_gramma>",
     "<dom_formosan_shigeru_tsuchida_reconstruction_proto_t>",
     "<dom_formosan_shih_rukai_adverbial>",
+    "<dom_formosan_sing_olam_hong_tinglan_aboriginal_educa>",
     "<dom_formosan_song_limei_introduction_kanakanavu_gram>",
     "<dom_formosan_song_limei_introduction_seediq_grammar>",
+    "<dom_formosan_soren_egerod_statement_atayal_phonology>",
+    "<dom_formosan_stacy_teng_malcom_ross_is_puyuma_primar>",
+    "<dom_formosan_truku_lowking_demonstratives>",
+    "<dom_formosan_tsai_hsui_liu_complementation_three_lan>",
+    "<dom_formosan_tsai_wei_tien_dylan_conjunctive_reducti>",
+    "<dom_formosan_tsou_descriptive_study>",
+    "<dom_formosan_tung_tsuchida_ting_li_pan_saaroa_texts>",
+    "<dom_formosan_wei_huilin_liu_social_structure_yami>",
     "<dom_formosan_wilang_yutas_videos>",
+    "<dom_formosan_wu_chunming_adverbials_in_paiwan>",
     "<dom_formosan_wu_jinglan_introduction_amis_grammar>",
+    "<dom_formosan_xie_fuhui_introduction_kavalan_grammar>",
     "<dom_formosan_yedda_palemeq_blog>",
     "<dom_formosan_yeddas_blog>",
     "<dom_formosan_yeh_meili_introduction_saisiyat_grammar>",
+    "<dom_formosan_yi_yang_cheng_kanakanavu_word_level_pro>",
+    "<dom_formosan_yi_yang_cheng_tense_aspect_agent_markin>",
+    "<dom_formosan_yoshiro_nihira_bunun_vocabulary>",
+    "<dom_formosan_zeng_shifen_reduplication_affixation_pa>",
+    "<dom_formosan_zhan_sujuan_taiwan_indigenous_peoples_h>",
+    "<dom_formosan_zhang_xiujuan_introduction_paiwan_gramm>",
     "<dom_formosan_zhang_yongli_pan_jiarong_introduction_t>",
+    "<dom_formosan_zhao_shanhe_li_taiyuan_zhu_qingyi_abori>",
     "<dom_formosan_zheng_acl_2024>",
     "<dom_formosan_zheng_data>",
+    "<dom_formosan_zheng_yumei_zhu_qingyi_bo_hongming_abor>",
     "<dom_learning_vocab>",
     "<dom_nine_level>",
     "<dom_ntu>",
   },
   "mask_token": {
     "content": "<mask>",
+    "lstrip": true,
+    "normalized": true,
     "rstrip": false,
     "single_word": false
   },

tokenizer_config.json CHANGED Viewed

@@ -1650,13 +1650,13 @@
     },
     "256203": {
       "content": "<mask>",
-      "lstrip": false,
-      "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259217": {
       "content": "ami_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1664,7 +1664,7 @@
       "single_word": false,
       "special": true
     },
-    "259218": {
       "content": "bnn_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1672,7 +1672,7 @@
       "single_word": false,
       "special": true
     },
-    "259219": {
       "content": "ckv_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1680,7 +1680,7 @@
       "single_word": false,
       "special": true
     },
-    "259220": {
       "content": "dru_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1688,7 +1688,7 @@
       "single_word": false,
       "special": true
     },
-    "259221": {
       "content": "pwn_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1696,7 +1696,7 @@
       "single_word": false,
       "special": true
     },
-    "259222": {
       "content": "pyu_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1704,7 +1704,7 @@
       "single_word": false,
       "special": true
     },
-    "259223": {
       "content": "ssf_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1712,7 +1712,7 @@
       "single_word": false,
       "special": true
     },
-    "259224": {
       "content": "sxr_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1720,7 +1720,7 @@
       "single_word": false,
       "special": true
     },
-    "259225": {
       "content": "szy_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1728,7 +1728,7 @@
       "single_word": false,
       "special": true
     },
-    "259226": {
       "content": "tao_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1736,7 +1736,7 @@
       "single_word": false,
       "special": true
     },
-    "259227": {
       "content": "tay_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1744,7 +1744,7 @@
       "single_word": false,
       "special": true
     },
-    "259228": {
       "content": "trv_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1752,7 +1752,7 @@
       "single_word": false,
       "special": true
     },
-    "259229": {
       "content": "tsu_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1760,7 +1760,7 @@
       "single_word": false,
       "special": true
     },
-    "259230": {
       "content": "xnb_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1768,7 +1768,7 @@
       "single_word": false,
       "special": true
     },
-    "259231": {
       "content": "xsy_Latn",
       "lstrip": false,
       "normalized": false,
@@ -1776,15 +1776,7 @@
       "single_word": false,
       "special": true
     },
-    "259232": {
-      "content": "<dae>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "259233": {
       "content": "<dialect_central>",
       "lstrip": false,
       "normalized": false,
@@ -1792,7 +1784,7 @@
       "single_word": false,
       "special": true
     },
-    "259234": {
       "content": "<dialect_coastal>",
       "lstrip": false,
       "normalized": false,
@@ -1800,7 +1792,7 @@
       "single_word": false,
       "special": true
     },
-    "259235": {
       "content": "<dialect_dawu>",
       "lstrip": false,
       "normalized": false,
@@ -1808,7 +1800,7 @@
       "single_word": false,
       "special": true
     },
-    "259236": {
       "content": "<dialect_default>",
       "lstrip": false,
       "normalized": false,
@@ -1816,7 +1808,7 @@
       "single_word": false,
       "special": true
     },
-    "259237": {
       "content": "<dialect_deluvalley>",
       "lstrip": false,
       "normalized": false,
@@ -1824,7 +1816,7 @@
       "single_word": false,
       "special": true
     },
-    "259238": {
       "content": "<dialect_dona>",
       "lstrip": false,
       "normalized": false,
@@ -1832,7 +1824,7 @@
       "single_word": false,
       "special": true
     },
-    "259239": {
       "content": "<dialect_duda>",
       "lstrip": false,
       "normalized": false,
@@ -1840,7 +1832,7 @@
       "single_word": false,
       "special": true
     },
-    "259240": {
       "content": "<dialect_eastern>",
       "lstrip": false,
       "normalized": false,
@@ -1848,7 +1840,7 @@
       "single_word": false,
       "special": true
     },
-    "259241": {
       "content": "<dialect_fourseasons>",
       "lstrip": false,
       "normalized": false,
@@ -1856,7 +1848,7 @@
       "single_word": false,
       "special": true
     },
-    "259242": {
       "content": "<dialect_hengchun>",
       "lstrip": false,
       "normalized": false,
@@ -1864,7 +1856,7 @@
       "single_word": false,
       "special": true
     },
-    "259243": {
       "content": "<dialect_jianhe>",
       "lstrip": false,
       "normalized": false,
@@ -1872,7 +1864,7 @@
       "single_word": false,
       "special": true
     },
-    "259244": {
       "content": "<dialect_junqun>",
       "lstrip": false,
       "normalized": false,
@@ -1880,7 +1872,7 @@
       "single_word": false,
       "special": true
     },
-    "259245": {
       "content": "<dialect_kanakanavu>",
       "lstrip": false,
       "normalized": false,
@@ -1888,7 +1880,7 @@
       "single_word": false,
       "special": true
     },
-    "259246": {
       "content": "<dialect_kaqun>",
       "lstrip": false,
       "normalized": false,
@@ -1896,7 +1888,7 @@
       "single_word": false,
       "special": true
     },
-    "259247": {
       "content": "<dialect_kavalan>",
       "lstrip": false,
       "normalized": false,
@@ -1904,7 +1896,7 @@
       "single_word": false,
       "special": true
     },
-    "259248": {
       "content": "<dialect_luanqun>",
       "lstrip": false,
       "normalized": false,
@@ -1912,7 +1904,7 @@
       "single_word": false,
       "special": true
     },
-    "259249": {
       "content": "<dialect_malan>",
       "lstrip": false,
       "normalized": false,
@@ -1920,7 +1912,7 @@
       "single_word": false,
       "special": true
     },
-    "259250": {
       "content": "<dialect_maolin>",
       "lstrip": false,
       "normalized": false,
@@ -1928,7 +1920,7 @@
       "single_word": false,
       "special": true
     },
-    "259251": {
       "content": "<dialect_nanwang>",
       "lstrip": false,
       "normalized": false,
@@ -1936,7 +1928,7 @@
       "single_word": false,
       "special": true
     },
-    "259252": {
       "content": "<dialect_northern>",
       "lstrip": false,
       "normalized": false,
@@ -1944,7 +1936,7 @@
       "single_word": false,
       "special": true
     },
-    "259253": {
       "content": "<dialect_saaroa>",
       "lstrip": false,
       "normalized": false,
@@ -1952,7 +1944,7 @@
       "single_word": false,
       "special": true
     },
-    "259254": {
       "content": "<dialect_saisiyat>",
       "lstrip": false,
       "normalized": false,
@@ -1960,7 +1952,7 @@
       "single_word": false,
       "special": true
     },
-    "259255": {
       "content": "<dialect_sakizaya>",
       "lstrip": false,
       "normalized": false,
@@ -1968,7 +1960,7 @@
       "single_word": false,
       "special": true
     },
-    "259256": {
       "content": "<dialect_sekolik>",
       "lstrip": false,
       "normalized": false,
@@ -1976,7 +1968,7 @@
       "single_word": false,
       "special": true
     },
-    "259257": {
       "content": "<dialect_southern>",
       "lstrip": false,
       "normalized": false,
@@ -1984,7 +1976,7 @@
       "single_word": false,
       "special": true
     },
-    "259258": {
       "content": "<dialect_tanqun>",
       "lstrip": false,
       "normalized": false,
@@ -1992,7 +1984,7 @@
       "single_word": false,
       "special": true
     },
-    "259259": {
       "content": "<dialect_tegudaya>",
       "lstrip": false,
       "normalized": false,
@@ -2000,7 +1992,7 @@
       "single_word": false,
       "special": true
     },
-    "259260": {
       "content": "<dialect_thao>",
       "lstrip": false,
       "normalized": false,
@@ -2008,7 +2000,7 @@
       "single_word": false,
       "special": true
     },
-    "259261": {
       "content": "<dialect_truku>",
       "lstrip": false,
       "normalized": false,
@@ -2016,7 +2008,7 @@
       "single_word": false,
       "special": true
     },
-    "259262": {
       "content": "<dialect_tsou>",
       "lstrip": false,
       "normalized": false,
@@ -2024,7 +2016,7 @@
       "single_word": false,
       "special": true
     },
-    "259263": {
       "content": "<dialect_unknown>",
       "lstrip": false,
       "normalized": false,
@@ -2032,7 +2024,7 @@
       "single_word": false,
       "special": true
     },
-    "259264": {
       "content": "<dialect_wanda>",
       "lstrip": false,
       "normalized": false,
@@ -2040,7 +2032,7 @@
       "single_word": false,
       "special": true
     },
-    "259265": {
       "content": "<dialect_wanshan>",
       "lstrip": false,
       "normalized": false,
@@ -2048,7 +2040,7 @@
       "single_word": false,
       "special": true
     },
-    "259266": {
       "content": "<dialect_wenshui>",
       "lstrip": false,
       "normalized": false,
@@ -2056,7 +2048,7 @@
       "single_word": false,
       "special": true
     },
-    "259267": {
       "content": "<dialect_wutai>",
       "lstrip": false,
       "normalized": false,
@@ -2064,7 +2056,7 @@
       "single_word": false,
       "special": true
     },
-    "259268": {
       "content": "<dialect_xiqun>",
       "lstrip": false,
       "normalized": false,
@@ -2072,7 +2064,7 @@
       "single_word": false,
       "special": true
     },
-    "259269": {
       "content": "<dialect_xiuguluan>",
       "lstrip": false,
       "normalized": false,
@@ -2080,7 +2072,7 @@
       "single_word": false,
       "special": true
     },
-    "259270": {
       "content": "<dialect_yami>",
       "lstrip": false,
       "normalized": false,
@@ -2088,7 +2080,7 @@
       "single_word": false,
       "special": true
     },
-    "259271": {
       "content": "<dialect_yilanzeaol>",
       "lstrip": false,
       "normalized": false,
@@ -2096,7 +2088,7 @@
       "single_word": false,
       "special": true
     },
-    "259272": {
       "content": "<dialect_zeaol>",
       "lstrip": false,
       "normalized": false,
@@ -2104,7 +2096,7 @@
       "single_word": false,
       "special": true
     },
-    "259273": {
       "content": "<dialect_zhiben>",
       "lstrip": false,
       "normalized": false,
@@ -2112,7 +2104,7 @@
       "single_word": false,
       "special": true
     },
-    "259274": {
       "content": "<dialect_zhuoqun>",
       "lstrip": false,
       "normalized": false,
@@ -2120,7 +2112,7 @@
       "single_word": false,
       "special": true
     },
-    "259275": {
       "content": "<dom_classroom_context>",
       "lstrip": false,
       "normalized": false,
@@ -2128,7 +2120,7 @@
       "single_word": false,
       "special": true
     },
-    "259276": {
       "content": "<dom_culture>",
       "lstrip": false,
       "normalized": false,
@@ -2136,7 +2128,7 @@
       "single_word": false,
       "special": true
     },
-    "259277": {
       "content": "<dom_dictionary>",
       "lstrip": false,
       "normalized": false,
@@ -2144,7 +2136,7 @@
       "single_word": false,
       "special": true
     },
-    "259278": {
       "content": "<dom_essays>",
       "lstrip": false,
       "normalized": false,
@@ -2152,39 +2144,39 @@
       "single_word": false,
       "special": true
     },
-    "259279": {
-      "content": "<dom_ethnography_journal_ritual_song>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259280": {
-      "content": "<dom_formosan_100_paiwan_texts>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259281": {
-      "content": "<dom_formosan_academia_sinica_oral_legends>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259282": {
-      "content": "<dom_formosan_amis_adversative_constructions>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259283": {
       "content": "<dom_formosan_amis_kavalan_lin_interrogative_verbs>",
       "lstrip": false,
       "normalized": false,
@@ -2192,7 +2184,7 @@
       "single_word": false,
       "special": true
     },
-    "259284": {
       "content": "<dom_formosan_amis_kuo_sung_comparative_constructions>",
       "lstrip": false,
       "normalized": false,
@@ -2200,7 +2192,7 @@
       "single_word": false,
       "special": true
     },
-    "259285": {
       "content": "<dom_formosan_amis_lifok_dongi_saopo_kimakimad>",
       "lstrip": false,
       "normalized": false,
@@ -2208,7 +2200,7 @@
       "single_word": false,
       "special": true
     },
-    "259286": {
       "content": "<dom_formosan_amis_myths_and_customs>",
       "lstrip": false,
       "normalized": false,
@@ -2216,7 +2208,7 @@
       "single_word": false,
       "special": true
     },
-    "259287": {
       "content": "<dom_formosan_amis_pa_verbs>",
       "lstrip": false,
       "normalized": false,
@@ -2224,47 +2216,47 @@
       "single_word": false,
       "special": true
     },
-    "259288": {
-      "content": "<dom_formosan_amis_serial_verb_constructions>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259289": {
-      "content": "<dom_formosan_amis_tung_chiou>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259290": {
-      "content": "<dom_formosan_asai_sedik_language>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259291": {
-      "content": "<dom_formosan_bunun_debusser_dissertation>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259292": {
-      "content": "<dom_formosan_bunun_moriguchi_northern_texts>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259293": {
       "content": "<dom_formosan_bunun_topic_focus>",
       "lstrip": false,
       "normalized": false,
@@ -2272,23 +2264,39 @@
       "single_word": false,
       "special": true
     },
-    "259294": {
-      "content": "<dom_formosan_chen_fukuda_relabeling_ergative>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259295": {
-      "content": "<dom_formosan_chen_fukuda_three_ways_steal>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259296": {
       "content": "<dom_formosan_cip_atayal_grammar_overview>",
       "lstrip": false,
       "normalized": false,
@@ -2296,7 +2304,23 @@
       "single_word": false,
       "special": true
     },
-    "259297": {
       "content": "<dom_formosan_dean_johnson_year_clouds_smangus>",
       "lstrip": false,
       "normalized": false,
@@ -2304,31 +2328,39 @@
       "single_word": false,
       "special": true
     },
-    "259298": {
-      "content": "<dom_formosan_dorinda_tsai_hsiu_liu_neutral_imperfect>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259299": {
-      "content": "<dom_formosan_egerod_origin_headhunting_atayal_text_v>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259300": {
-      "content": "<dom_formosan_egerod_soren_word_order_word_classes_at>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259301": {
       "content": "<dom_formosan_elizabeth_zeitoun_chen_huei_wu_overview>",
       "lstrip": false,
       "normalized": false,
@@ -2336,23 +2368,39 @@
       "single_word": false,
       "special": true
     },
-    "259302": {
-      "content": "<dom_formosan_epark>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259303": {
-      "content": "<dom_formosan_ferrell_taiwan_aboriginal_groups_proble>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259304": {
       "content": "<dom_formosan_gitbook_translations>",
       "lstrip": false,
       "normalized": false,
@@ -2360,7 +2408,7 @@
       "single_word": false,
       "special": true
     },
-    "259305": {
       "content": "<dom_formosan_glosbe>",
       "lstrip": false,
       "normalized": false,
@@ -2368,15 +2416,15 @@
       "single_word": false,
       "special": true
     },
-    "259306": {
-      "content": "<dom_formosan_hala_saku_la_videos>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259307": {
       "content": "<dom_formosan_holmer_seediq>",
       "lstrip": false,
       "normalized": false,
@@ -2384,7 +2432,15 @@
       "single_word": false,
       "special": true
     },
-    "259308": {
       "content": "<dom_formosan_huang_grammaticalization_squliq_atayal>",
       "lstrip": false,
       "normalized": false,
@@ -2392,7 +2448,23 @@
       "single_word": false,
       "special": true
     },
-    "259309": {
       "content": "<dom_formosan_huang_mei_chin_atayal_reference_grammar>",
       "lstrip": false,
       "normalized": false,
@@ -2400,7 +2472,7 @@
       "single_word": false,
       "special": true
     },
-    "259310": {
       "content": "<dom_formosan_huteson_rukai_survey>",
       "lstrip": false,
       "normalized": false,
@@ -2408,7 +2480,7 @@
       "single_word": false,
       "special": true
     },
-    "259311": {
       "content": "<dom_formosan_ilrdf_42_language_practice_word_lists>",
       "lstrip": false,
       "normalized": false,
@@ -2416,7 +2488,7 @@
       "single_word": false,
       "special": true
     },
-    "259312": {
       "content": "<dom_formosan_ilrdf_tousvusvutu_kita>",
       "lstrip": false,
       "normalized": false,
@@ -2424,7 +2496,15 @@
       "single_word": false,
       "special": true
     },
-    "259313": {
       "content": "<dom_formosan_indigenous_language_literary_awards>",
       "lstrip": false,
       "normalized": false,
@@ -2432,7 +2512,47 @@
       "single_word": false,
       "special": true
     },
-    "259314": {
       "content": "<dom_formosan_kanakanavu_texts>",
       "lstrip": false,
       "normalized": false,
@@ -2440,7 +2560,7 @@
       "single_word": false,
       "special": true
     },
-    "259315": {
       "content": "<dom_formosan_kavalan_zhang_reference_grammar>",
       "lstrip": false,
       "normalized": false,
@@ -2448,7 +2568,55 @@
       "single_word": false,
       "special": true
     },
-    "259316": {
       "content": "<dom_formosan_li_peirong_xu_weisheng_introduction_tru>",
       "lstrip": false,
       "normalized": false,
@@ -2456,7 +2624,7 @@
       "single_word": false,
       "special": true
     },
-    "259317": {
       "content": "<dom_formosan_li_peizhen_wu_xingyu_shouhu_xing>",
       "lstrip": false,
       "normalized": false,
@@ -2464,7 +2632,47 @@
       "single_word": false,
       "special": true
     },
-    "259318": {
       "content": "<dom_formosan_naomi_tsukida_correlative_clauses_seedi>",
       "lstrip": false,
       "normalized": false,
@@ -2472,7 +2680,7 @@
       "single_word": false,
       "special": true
     },
-    "259319": {
       "content": "<dom_formosan_nowbucyang_truku_thesis>",
       "lstrip": false,
       "normalized": false,
@@ -2480,7 +2688,15 @@
       "single_word": false,
       "special": true
     },
-    "259320": {
       "content": "<dom_formosan_old_texts>",
       "lstrip": false,
       "normalized": false,
@@ -2488,7 +2704,7 @@
       "single_word": false,
       "special": true
     },
-    "259321": {
       "content": "<dom_formosan_paiwan_collart_zeitoun_time_reference>",
       "lstrip": false,
       "normalized": false,
@@ -2496,7 +2712,7 @@
       "single_word": false,
       "special": true
     },
-    "259322": {
       "content": "<dom_formosan_paiwan_ho_five_dialects>",
       "lstrip": false,
       "normalized": false,
@@ -2504,7 +2720,7 @@
       "single_word": false,
       "special": true
     },
-    "259323": {
       "content": "<dom_formosan_paiwanstories>",
       "lstrip": false,
       "normalized": false,
@@ -2512,79 +2728,103 @@
       "single_word": false,
       "special": true
     },
-    "259324": {
-      "content": "<dom_formosan_puyuma_chen_raising_to_object>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259325": {
-      "content": "<dom_formosan_puyuma_katipol_kopfjagdriten>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259326": {
-      "content": "<dom_formosan_puyuma_teng_pronominal_systems>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259327": {
-      "content": "<dom_formosan_puyuma_teng_reference_grammar>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259328": {
-      "content": "<dom_formosan_rik_bunun>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259329": {
-      "content": "<dom_formosan_robert_blust_austronesian_homeland_ling>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259330": {
-      "content": "<dom_formosan_rukai_mantauran_stories>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259331": {
-      "content": "<dom_formosan_rukai_texts>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259332": {
-      "content": "<dom_formosan_rukai_zeitoun_saying>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259333": {
       "content": "<dom_formosan_saaroa_pan_grammar>",
       "lstrip": false,
       "normalized": false,
@@ -2592,7 +2832,7 @@
       "single_word": false,
       "special": true
     },
-    "259334": {
       "content": "<dom_formosan_sakizaya_affixes>",
       "lstrip": false,
       "normalized": false,
@@ -2600,7 +2840,7 @@
       "single_word": false,
       "special": true
     },
-    "259335": {
       "content": "<dom_formosan_seals>",
       "lstrip": false,
       "normalized": false,
@@ -2608,7 +2848,7 @@
       "single_word": false,
       "special": true
     },
-    "259336": {
       "content": "<dom_formosan_seediq_zhang_reference_grammar>",
       "lstrip": false,
       "normalized": false,
@@ -2616,15 +2856,15 @@
       "single_word": false,
       "special": true
     },
-    "259337": {
-      "content": "<dom_formosan_shigeru_tsuchida_kanakanavu_texts>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259338": {
       "content": "<dom_formosan_shigeru_tsuchida_reconstruction_proto_t>",
       "lstrip": false,
       "normalized": false,
@@ -2632,7 +2872,7 @@
       "single_word": false,
       "special": true
     },
-    "259339": {
       "content": "<dom_formosan_shih_rukai_adverbial>",
       "lstrip": false,
       "normalized": false,
@@ -2640,7 +2880,15 @@
       "single_word": false,
       "special": true
     },
-    "259340": {
       "content": "<dom_formosan_song_limei_introduction_kanakanavu_gram>",
       "lstrip": false,
       "normalized": false,
@@ -2648,7 +2896,7 @@
       "single_word": false,
       "special": true
     },
-    "259341": {
       "content": "<dom_formosan_song_limei_introduction_seediq_grammar>",
       "lstrip": false,
       "normalized": false,
@@ -2656,39 +2904,71 @@
       "single_word": false,
       "special": true
     },
-    "259342": {
-      "content": "<dom_formosan_sung_kuo_descriptive_comparative_constr>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259343": {
-      "content": "<dom_formosan_taoshan_elementary_school_atelier_hui_k>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259344": {
-      "content": "<dom_formosan_tsukida_naomi_verb_classification_amis>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259345": {
-      "content": "<dom_formosan_wei_matri_clan_lineage_system_ami>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
-    "259346": {
       "content": "<dom_formosan_wilang_yutas_videos>",
       "lstrip": false,
       "normalized": false,
@@ -2696,7 +2976,15 @@
       "single_word": false,
       "special": true
     },
-    "259347": {
       "content": "<dom_formosan_wu_jinglan_introduction_amis_grammar>",
       "lstrip": false,
       "normalized": false,
@@ -2704,7 +2992,15 @@
       "single_word": false,
       "special": true
     },
-    "259348": {
       "content": "<dom_formosan_yedda_palemeq_blog>",
       "lstrip": false,
       "normalized": false,
@@ -2712,7 +3008,7 @@
       "single_word": false,
       "special": true
     },
-    "259349": {
       "content": "<dom_formosan_yeddas_blog>",
       "lstrip": false,
       "normalized": false,
@@ -2720,7 +3016,7 @@
       "single_word": false,
       "special": true
     },
-    "259350": {
       "content": "<dom_formosan_yeh_meili_introduction_saisiyat_grammar>",
       "lstrip": false,
       "normalized": false,
@@ -2728,7 +3024,55 @@
       "single_word": false,
       "special": true
     },
-    "259351": {
       "content": "<dom_formosan_zhang_yongli_pan_jiarong_introduction_t>",
       "lstrip": false,
       "normalized": false,
@@ -2736,7 +3080,15 @@
       "single_word": false,
       "special": true
     },
-    "259352": {
       "content": "<dom_formosan_zheng_acl_2024>",
       "lstrip": false,
       "normalized": false,
@@ -2744,7 +3096,7 @@
       "single_word": false,
       "special": true
     },
-    "259353": {
       "content": "<dom_formosan_zheng_data>",
       "lstrip": false,
       "normalized": false,
@@ -2752,7 +3104,15 @@
       "single_word": false,
       "special": true
     },
-    "259354": {
       "content": "<dom_learning_vocab>",
       "lstrip": false,
       "normalized": false,
@@ -2760,7 +3120,7 @@
       "single_word": false,
       "special": true
     },
-    "259355": {
       "content": "<dom_nine_level>",
       "lstrip": false,
       "normalized": false,
@@ -2768,7 +3128,7 @@
       "single_word": false,
       "special": true
     },
-    "259356": {
       "content": "<dom_ntu>",
       "lstrip": false,
       "normalized": false,
@@ -2776,7 +3136,7 @@
       "single_word": false,
       "special": true
     },
-    "259357": {
       "content": "<dom_picture_book>",
       "lstrip": false,
       "normalized": false,
@@ -2784,7 +3144,7 @@
       "single_word": false,
       "special": true
     },
-    "259358": {
       "content": "<dom_picture_story>",
       "lstrip": false,
       "normalized": false,
@@ -2792,7 +3152,7 @@
       "single_word": false,
       "special": true
     },
-    "259359": {
       "content": "<dom_presidential_apology>",
       "lstrip": false,
       "normalized": false,
@@ -2800,7 +3160,7 @@
       "single_word": false,
       "special": true
     },
-    "259360": {
       "content": "<dom_reading_writing>",
       "lstrip": false,
       "normalized": false,
@@ -2808,7 +3168,7 @@
       "single_word": false,
       "special": true
     },
-    "259361": {
       "content": "<dom_unknown>",
       "lstrip": false,
       "normalized": false,
@@ -2816,7 +3176,7 @@
       "single_word": false,
       "special": true
     },
-    "259362": {
       "content": "<dom_youtube>",
       "lstrip": false,
       "normalized": false,
@@ -2824,7 +3184,7 @@
       "single_word": false,
       "special": true
     },
-    "259363": {
       "content": "<src_ami>",
       "lstrip": false,
       "normalized": false,
@@ -2832,7 +3192,7 @@
       "single_word": false,
       "special": true
     },
-    "259364": {
       "content": "<src_bnn>",
       "lstrip": false,
       "normalized": false,
@@ -2840,7 +3200,7 @@
       "single_word": false,
       "special": true
     },
-    "259365": {
       "content": "<src_ckv>",
       "lstrip": false,
       "normalized": false,
@@ -2848,7 +3208,7 @@
       "single_word": false,
       "special": true
     },
-    "259366": {
       "content": "<src_dru>",
       "lstrip": false,
       "normalized": false,
@@ -2856,7 +3216,7 @@
       "single_word": false,
       "special": true
     },
-    "259367": {
       "content": "<src_eng>",
       "lstrip": false,
       "normalized": false,
@@ -2864,7 +3224,7 @@
       "single_word": false,
       "special": true
     },
-    "259368": {
       "content": "<src_pwn>",
       "lstrip": false,
       "normalized": false,
@@ -2872,7 +3232,7 @@
       "single_word": false,
       "special": true
     },
-    "259369": {
       "content": "<src_pyu>",
       "lstrip": false,
       "normalized": false,
@@ -2880,7 +3240,7 @@
       "single_word": false,
       "special": true
     },
-    "259370": {
       "content": "<src_ssf>",
       "lstrip": false,
       "normalized": false,
@@ -2888,7 +3248,7 @@
       "single_word": false,
       "special": true
     },
-    "259371": {
       "content": "<src_sxr>",
       "lstrip": false,
       "normalized": false,
@@ -2896,7 +3256,7 @@
       "single_word": false,
       "special": true
     },
-    "259372": {
       "content": "<src_szy>",
       "lstrip": false,
       "normalized": false,
@@ -2904,7 +3264,7 @@
       "single_word": false,
       "special": true
     },
-    "259373": {
       "content": "<src_tao>",
       "lstrip": false,
       "normalized": false,
@@ -2912,7 +3272,7 @@
       "single_word": false,
       "special": true
     },
-    "259374": {
       "content": "<src_tay>",
       "lstrip": false,
       "normalized": false,
@@ -2920,7 +3280,7 @@
       "single_word": false,
       "special": true
     },
-    "259375": {
       "content": "<src_trv>",
       "lstrip": false,
       "normalized": false,
@@ -2928,7 +3288,7 @@
       "single_word": false,
       "special": true
     },
-    "259376": {
       "content": "<src_tsu>",
       "lstrip": false,
       "normalized": false,
@@ -2936,7 +3296,7 @@
       "single_word": false,
       "special": true
     },
-    "259377": {
       "content": "<src_xnb>",
       "lstrip": false,
       "normalized": false,
@@ -2944,7 +3304,7 @@
       "single_word": false,
       "special": true
     },
-    "259378": {
       "content": "<src_xsy>",
       "lstrip": false,
       "normalized": false,
@@ -2952,7 +3312,7 @@
       "single_word": false,
       "special": true
     },
-    "259379": {
       "content": "<src_zh>",
       "lstrip": false,
       "normalized": false,
@@ -2960,7 +3320,7 @@
       "single_word": false,
       "special": true
     },
-    "259380": {
       "content": "<to_ami>",
       "lstrip": false,
       "normalized": false,
@@ -2968,7 +3328,7 @@
       "single_word": false,
       "special": true
     },
-    "259381": {
       "content": "<to_bnn>",
       "lstrip": false,
       "normalized": false,
@@ -2976,7 +3336,7 @@
       "single_word": false,
       "special": true
     },
-    "259382": {
       "content": "<to_ckv>",
       "lstrip": false,
       "normalized": false,
@@ -2984,7 +3344,7 @@
       "single_word": false,
       "special": true
     },
-    "259383": {
       "content": "<to_dru>",
       "lstrip": false,
       "normalized": false,
@@ -2992,7 +3352,7 @@
       "single_word": false,
       "special": true
     },
-    "259384": {
       "content": "<to_eng>",
       "lstrip": false,
       "normalized": false,
@@ -3000,7 +3360,7 @@
       "single_word": false,
       "special": true
     },
-    "259385": {
       "content": "<to_pwn>",
       "lstrip": false,
       "normalized": false,
@@ -3008,7 +3368,7 @@
       "single_word": false,
       "special": true
     },
-    "259386": {
       "content": "<to_pyu>",
       "lstrip": false,
       "normalized": false,
@@ -3016,7 +3376,7 @@
       "single_word": false,
       "special": true
     },
-    "259387": {
       "content": "<to_ssf>",
       "lstrip": false,
       "normalized": false,
@@ -3024,7 +3384,7 @@
       "single_word": false,
       "special": true
     },
-    "259388": {
       "content": "<to_sxr>",
       "lstrip": false,
       "normalized": false,
@@ -3032,7 +3392,7 @@
       "single_word": false,
       "special": true
     },
-    "259389": {
       "content": "<to_szy>",
       "lstrip": false,
       "normalized": false,
@@ -3040,7 +3400,7 @@
       "single_word": false,
       "special": true
     },
-    "259390": {
       "content": "<to_tao>",
       "lstrip": false,
       "normalized": false,
@@ -3048,7 +3408,7 @@
       "single_word": false,
       "special": true
     },
-    "259391": {
       "content": "<to_tay>",
       "lstrip": false,
       "normalized": false,
@@ -3056,7 +3416,7 @@
       "single_word": false,
       "special": true
     },
-    "259392": {
       "content": "<to_trv>",
       "lstrip": false,
       "normalized": false,
@@ -3064,7 +3424,7 @@
       "single_word": false,
       "special": true
     },
-    "259393": {
       "content": "<to_tsu>",
       "lstrip": false,
       "normalized": false,
@@ -3072,7 +3432,7 @@
       "single_word": false,
       "special": true
     },
-    "259394": {
       "content": "<to_xnb>",
       "lstrip": false,
       "normalized": false,
@@ -3080,7 +3440,7 @@
       "single_word": false,
       "special": true
     },
-    "259395": {
       "content": "<to_xsy>",
       "lstrip": false,
       "normalized": false,
@@ -3088,7 +3448,7 @@
       "single_word": false,
       "special": true
     },
-    "259396": {
       "content": "<to_zh>",
       "lstrip": false,
       "normalized": false,
@@ -3315,8 +3675,6 @@
     "tsu_Latn",
     "xnb_Latn",
     "xsy_Latn",
-    "<mask>",
-    "<dae>",
     "<dialect_central>",
     "<dialect_coastal>",
     "<dialect_dawu>",
@@ -3363,81 +3721,127 @@
     "<dom_culture>",
     "<dom_dictionary>",
     "<dom_essays>",
-    "<dom_ethnography_journal_ritual_song>",
     "<dom_formosan_100_paiwan_texts>",
     "<dom_formosan_academia_sinica_oral_legends>",
     "<dom_formosan_amis_adversative_constructions>",
     "<dom_formosan_amis_kavalan_lin_interrogative_verbs>",
     "<dom_formosan_amis_kuo_sung_comparative_constructions>",
     "<dom_formosan_amis_lifok_dongi_saopo_kimakimad>",
     "<dom_formosan_amis_myths_and_customs>",
     "<dom_formosan_amis_pa_verbs>",
-    "<dom_formosan_amis_serial_verb_constructions>",
-    "<dom_formosan_amis_tung_chiou>",
-    "<dom_formosan_asai_sedik_language>",
     "<dom_formosan_bunun_debusser_dissertation>",
-    "<dom_formosan_bunun_moriguchi_northern_texts>",
     "<dom_formosan_bunun_topic_focus>",
-    "<dom_formosan_chen_fukuda_relabeling_ergative>",
-    "<dom_formosan_chen_fukuda_three_ways_steal>",
     "<dom_formosan_cip_atayal_grammar_overview>",
     "<dom_formosan_dean_johnson_year_clouds_smangus>",
-    "<dom_formosan_dorinda_tsai_hsiu_liu_neutral_imperfect>",
-    "<dom_formosan_egerod_origin_headhunting_atayal_text_v>",
-    "<dom_formosan_egerod_soren_word_order_word_classes_at>",
     "<dom_formosan_elizabeth_zeitoun_chen_huei_wu_overview>",
     "<dom_formosan_epark>",
-    "<dom_formosan_ferrell_taiwan_aboriginal_groups_proble>",
     "<dom_formosan_gitbook_translations>",
     "<dom_formosan_glosbe>",
-    "<dom_formosan_hala_saku_la_videos>",
     "<dom_formosan_holmer_seediq>",
     "<dom_formosan_huang_grammaticalization_squliq_atayal>",
     "<dom_formosan_huang_mei_chin_atayal_reference_grammar>",
     "<dom_formosan_huteson_rukai_survey>",
     "<dom_formosan_ilrdf_42_language_practice_word_lists>",
     "<dom_formosan_ilrdf_tousvusvutu_kita>",
     "<dom_formosan_indigenous_language_literary_awards>",
     "<dom_formosan_kanakanavu_texts>",
     "<dom_formosan_kavalan_zhang_reference_grammar>",
     "<dom_formosan_li_peirong_xu_weisheng_introduction_tru>",
     "<dom_formosan_li_peizhen_wu_xingyu_shouhu_xing>",
     "<dom_formosan_naomi_tsukida_correlative_clauses_seedi>",
     "<dom_formosan_nowbucyang_truku_thesis>",
     "<dom_formosan_old_texts>",
     "<dom_formosan_paiwan_collart_zeitoun_time_reference>",
     "<dom_formosan_paiwan_ho_five_dialects>",
     "<dom_formosan_paiwanstories>",
-    "<dom_formosan_puyuma_chen_raising_to_object>",
     "<dom_formosan_puyuma_katipol_kopfjagdriten>",
-    "<dom_formosan_puyuma_teng_pronominal_systems>",
     "<dom_formosan_puyuma_teng_reference_grammar>",
     "<dom_formosan_rik_bunun>",
-    "<dom_formosan_robert_blust_austronesian_homeland_ling>",
-    "<dom_formosan_rukai_mantauran_stories>",
-    "<dom_formosan_rukai_texts>",
-    "<dom_formosan_rukai_zeitoun_saying>",
     "<dom_formosan_saaroa_pan_grammar>",
     "<dom_formosan_sakizaya_affixes>",
     "<dom_formosan_seals>",
     "<dom_formosan_seediq_zhang_reference_grammar>",
-    "<dom_formosan_shigeru_tsuchida_kanakanavu_texts>",
     "<dom_formosan_shigeru_tsuchida_reconstruction_proto_t>",
     "<dom_formosan_shih_rukai_adverbial>",
     "<dom_formosan_song_limei_introduction_kanakanavu_gram>",
     "<dom_formosan_song_limei_introduction_seediq_grammar>",
-    "<dom_formosan_sung_kuo_descriptive_comparative_constr>",
-    "<dom_formosan_taoshan_elementary_school_atelier_hui_k>",
-    "<dom_formosan_tsukida_naomi_verb_classification_amis>",
-    "<dom_formosan_wei_matri_clan_lineage_system_ami>",
     "<dom_formosan_wilang_yutas_videos>",
     "<dom_formosan_wu_jinglan_introduction_amis_grammar>",
     "<dom_formosan_yedda_palemeq_blog>",
     "<dom_formosan_yeddas_blog>",
     "<dom_formosan_yeh_meili_introduction_saisiyat_grammar>",
     "<dom_formosan_zhang_yongli_pan_jiarong_introduction_t>",
     "<dom_formosan_zheng_acl_2024>",
     "<dom_formosan_zheng_data>",
     "<dom_learning_vocab>",
     "<dom_nine_level>",
     "<dom_ntu>",

     },
     "256203": {
       "content": "<mask>",
+      "lstrip": true,
+      "normalized": true,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261395": {
       "content": "ami_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261396": {
       "content": "bnn_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261397": {
       "content": "ckv_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261398": {
       "content": "dru_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261399": {
       "content": "pwn_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261400": {
       "content": "pyu_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261401": {
       "content": "ssf_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261402": {
       "content": "sxr_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261403": {
       "content": "szy_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261404": {
       "content": "tao_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261405": {
       "content": "tay_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261406": {
       "content": "trv_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261407": {
       "content": "tsu_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261408": {
       "content": "xnb_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261409": {
       "content": "xsy_Latn",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261410": {
       "content": "<dialect_central>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261411": {
       "content": "<dialect_coastal>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261412": {
       "content": "<dialect_dawu>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261413": {
       "content": "<dialect_default>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261414": {
       "content": "<dialect_deluvalley>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261415": {
       "content": "<dialect_dona>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261416": {
       "content": "<dialect_duda>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261417": {
       "content": "<dialect_eastern>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261418": {
       "content": "<dialect_fourseasons>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261419": {
       "content": "<dialect_hengchun>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261420": {
       "content": "<dialect_jianhe>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261421": {
       "content": "<dialect_junqun>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261422": {
       "content": "<dialect_kanakanavu>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261423": {
       "content": "<dialect_kaqun>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261424": {
       "content": "<dialect_kavalan>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261425": {
       "content": "<dialect_luanqun>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261426": {
       "content": "<dialect_malan>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261427": {
       "content": "<dialect_maolin>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261428": {
       "content": "<dialect_nanwang>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261429": {
       "content": "<dialect_northern>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261430": {
       "content": "<dialect_saaroa>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261431": {
       "content": "<dialect_saisiyat>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261432": {
       "content": "<dialect_sakizaya>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261433": {
       "content": "<dialect_sekolik>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261434": {
       "content": "<dialect_southern>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261435": {
       "content": "<dialect_tanqun>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261436": {
       "content": "<dialect_tegudaya>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261437": {
       "content": "<dialect_thao>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261438": {
       "content": "<dialect_truku>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261439": {
       "content": "<dialect_tsou>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261440": {
       "content": "<dialect_unknown>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261441": {
       "content": "<dialect_wanda>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261442": {
       "content": "<dialect_wanshan>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261443": {
       "content": "<dialect_wenshui>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261444": {
       "content": "<dialect_wutai>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261445": {
       "content": "<dialect_xiqun>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261446": {
       "content": "<dialect_xiuguluan>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261447": {
       "content": "<dialect_yami>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261448": {
       "content": "<dialect_yilanzeaol>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261449": {
       "content": "<dialect_zeaol>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261450": {
       "content": "<dialect_zhiben>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261451": {
       "content": "<dialect_zhuoqun>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261452": {
       "content": "<dom_classroom_context>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261453": {
       "content": "<dom_culture>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261454": {
       "content": "<dom_dictionary>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261455": {
       "content": "<dom_essays>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261456": {
+      "content": "<dom_formosan_100_paiwan_texts>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261457": {
+      "content": "<dom_formosan_academia_sinica_oral_legends>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261458": {
+      "content": "<dom_formosan_amis_adversative_constructions>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261459": {
+      "content": "<dom_formosan_amis_huang_english_phonology_grammar>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261460": {
       "content": "<dom_formosan_amis_kavalan_lin_interrogative_verbs>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261461": {
       "content": "<dom_formosan_amis_kuo_sung_comparative_constructions>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261462": {
       "content": "<dom_formosan_amis_lifok_dongi_saopo_kimakimad>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261463": {
       "content": "<dom_formosan_amis_myths_and_customs>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261464": {
       "content": "<dom_formosan_amis_pa_verbs>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261465": {
+      "content": "<dom_formosan_anna_hsiou_chuan_chang_reference_gramma>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261466": {
+      "content": "<dom_formosan_anton_quack_puyuma_pulingaw>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261467": {
+      "content": "<dom_formosan_atayal_you_nao_savak_raxayal>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261468": {
+      "content": "<dom_formosan_boyizhenu_narrative_oral_literature_tap>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261469": {
+      "content": "<dom_formosan_bunun_debusser_dissertation>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261470": {
       "content": "<dom_formosan_bunun_topic_focus>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261471": {
+      "content": "<dom_formosan_catherine_tseng_seediq_atayal_foods_the>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261472": {
+      "content": "<dom_formosan_chen_yaohan_miss_virginia_fey_played_es>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261473": {
+      "content": "<dom_formosan_chun_mei_chen_comparative_phonology_pai>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261474": {
+      "content": "<dom_formosan_chunming_wu_two_types_of_noun_incorpora>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261475": {
       "content": "<dom_formosan_cip_atayal_grammar_overview>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261476": {
+      "content": "<dom_formosan_claire_mcgill_a_brief_tayal_vocabulary>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261477": {
+      "content": "<dom_formosan_david_blundell_ed_austronesian_taiwan>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261478": {
       "content": "<dom_formosan_dean_johnson_year_clouds_smangus>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261479": {
+      "content": "<dom_formosan_deng_fangqing_introduction_puyuma_gramm>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261480": {
+      "content": "<dom_formosan_der_hwa_victoria_rau_grammar_atayal>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261481": {
+      "content": "<dom_formosan_dong_manv_yami_ode_to_taro>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261482": {
+      "content": "<dom_formosan_dong_yijia_tjuwabar_paiwan_ritual_langu>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261483": {
       "content": "<dom_formosan_elizabeth_zeitoun_chen_huei_wu_overview>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261484": {
+      "content": "<dom_formosan_elizabeth_zeitoun_lillian_huang_marie_y>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261485": {
+      "content": "<dom_formosan_elizabeth_zeitoun_pronominal_system_man>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261486": {
+      "content": "<dom_formosan_elizabeth_zeitoun_squib_dynamic_vs_stat>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261487": {
+      "content": "<dom_formosan_epark>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261488": {
       "content": "<dom_formosan_gitbook_translations>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261489": {
       "content": "<dom_formosan_glosbe>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261490": {
+      "content": "<dom_formosan_ho_dan_phonological_system_butonglu_pai>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261491": {
       "content": "<dom_formosan_holmer_seediq>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261492": {
+      "content": "<dom_formosan_hsiu_chuan_liao_transitivity_ergativity>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261493": {
       "content": "<dom_formosan_huang_grammaticalization_squliq_atayal>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261494": {
+      "content": "<dom_formosan_huang_hui_chuan_glide_formation_takitud>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261495": {
+      "content": "<dom_formosan_huang_huijuan_shi_chaokai_introduction>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261496": {
       "content": "<dom_formosan_huang_mei_chin_atayal_reference_grammar>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261497": {
       "content": "<dom_formosan_huteson_rukai_survey>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261498": {
       "content": "<dom_formosan_ilrdf_42_language_practice_word_lists>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261499": {
       "content": "<dom_formosan_ilrdf_tousvusvutu_kita>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261500": {
+      "content": "<dom_formosan_indigenous_education_resource_center_ke>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261501": {
       "content": "<dom_formosan_indigenous_language_literary_awards>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261502": {
+      "content": "<dom_formosan_isabelle_bril_roots_and_stems_lexical_a>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261503": {
+      "content": "<dom_formosan_jian_iwan_guo_qingliu_life_history>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261504": {
+      "content": "<dom_formosan_jian_shilang_introduction_thao_grammar>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261505": {
+      "content": "<dom_formosan_john_u_wolff_the_proto_austronesian_pho>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261506": {
+      "content": "<dom_formosan_josiane_cauquelin_aborigines_taiwan_puy>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261507": {
       "content": "<dom_formosan_kanakanavu_texts>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261508": {
       "content": "<dom_formosan_kavalan_zhang_reference_grammar>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261509": {
+      "content": "<dom_formosan_kolas_foting_aboriginal_education_world>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261510": {
+      "content": "<dom_formosan_kucapungane>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261511": {
+      "content": "<dom_formosan_kumu_tapas_tribal_memory_oral_history_w>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261512": {
+      "content": "<dom_formosan_lei_shih_family_system_paiwan_at_su_pai>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261513": {
+      "content": "<dom_formosan_li_may_sung_budai_rukai_exclamatives>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261514": {
+      "content": "<dom_formosan_li_may_sung_clausal_nominalization_buda>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261515": {
       "content": "<dom_formosan_li_peirong_xu_weisheng_introduction_tru>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261516": {
       "content": "<dom_formosan_li_peizhen_wu_xingyu_shouhu_xing>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261517": {
+      "content": "<dom_formosan_lillian_huang_atayal_participants_cross>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261518": {
+      "content": "<dom_formosan_liu_manyi_kaskun_ata_matas_i_ki_quaz>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261519": {
+      "content": "<dom_formosan_malcom_ross_proto_austronesian_verbal_m>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261520": {
+      "content": "<dom_formosan_maya_yeh_blaq_uv_construction_atayal>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261521": {
+      "content": "<dom_formosan_nanang_tadaw_naci_mowna_pgagu>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261522": {
       "content": "<dom_formosan_naomi_tsukida_correlative_clauses_seedi>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261523": {
       "content": "<dom_formosan_nowbucyang_truku_thesis>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261524": {
+      "content": "<dom_formosan_ochiai_izumi_numerals_paran_seediq_rela>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261525": {
       "content": "<dom_formosan_old_texts>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261526": {
       "content": "<dom_formosan_paiwan_collart_zeitoun_time_reference>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261527": {
       "content": "<dom_formosan_paiwan_ho_five_dialects>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261528": {
       "content": "<dom_formosan_paiwanstories>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261529": {
+      "content": "<dom_formosan_patricia_stanley_morphophonemics_of_ver>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261530": {
+      "content": "<dom_formosan_paul_jen_kuei_li_internal_relationships>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261531": {
+      "content": "<dom_formosan_paul_jen_kuei_li_rukai_structure>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261532": {
+      "content": "<dom_formosan_paul_jen_kuei_li_the_preglottalised_sto>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261533": {
+      "content": "<dom_formosan_paul_li_atayalic_final_voiced_stops>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261534": {
+      "content": "<dom_formosan_paul_li_position_atayal_austronesian>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261535": {
+      "content": "<dom_formosan_pingdong_county_government_kai_na_sepuc>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261536": {
+      "content": "<dom_formosan_puyuma_katipol_kopfjagdriten>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261537": {
+      "content": "<dom_formosan_puyuma_teng_reference_grammar>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261538": {
+      "content": "<dom_formosan_raleigh_ferrell_construction_markers_fo>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261539": {
+      "content": "<dom_formosan_rik_bunun>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261540": {
+      "content": "<dom_formosan_robert_blust_the_austronesian_languages>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261541": {
       "content": "<dom_formosan_saaroa_pan_grammar>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261542": {
       "content": "<dom_formosan_sakizaya_affixes>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261543": {
       "content": "<dom_formosan_seals>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261544": {
       "content": "<dom_formosan_seediq_zhang_reference_grammar>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261545": {
+      "content": "<dom_formosan_shen_wenqi_introduction_sakizaya_gramma>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261546": {
       "content": "<dom_formosan_shigeru_tsuchida_reconstruction_proto_t>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261547": {
       "content": "<dom_formosan_shih_rukai_adverbial>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261548": {
+      "content": "<dom_formosan_sing_olam_hong_tinglan_aboriginal_educa>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261549": {
       "content": "<dom_formosan_song_limei_introduction_kanakanavu_gram>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261550": {
       "content": "<dom_formosan_song_limei_introduction_seediq_grammar>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261551": {
+      "content": "<dom_formosan_soren_egerod_statement_atayal_phonology>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261552": {
+      "content": "<dom_formosan_stacy_teng_malcom_ross_is_puyuma_primar>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261553": {
+      "content": "<dom_formosan_truku_lowking_demonstratives>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261554": {
+      "content": "<dom_formosan_tsai_hsui_liu_complementation_three_lan>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261555": {
+      "content": "<dom_formosan_tsai_wei_tien_dylan_conjunctive_reducti>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261556": {
+      "content": "<dom_formosan_tsou_descriptive_study>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261557": {
+      "content": "<dom_formosan_tung_tsuchida_ting_li_pan_saaroa_texts>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
     },
+    "261558": {
+      "content": "<dom_formosan_wei_huilin_liu_social_structure_yami>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261559": {
       "content": "<dom_formosan_wilang_yutas_videos>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261560": {
+      "content": "<dom_formosan_wu_chunming_adverbials_in_paiwan>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261561": {
       "content": "<dom_formosan_wu_jinglan_introduction_amis_grammar>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261562": {
+      "content": "<dom_formosan_xie_fuhui_introduction_kavalan_grammar>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261563": {
       "content": "<dom_formosan_yedda_palemeq_blog>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261564": {
       "content": "<dom_formosan_yeddas_blog>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261565": {
       "content": "<dom_formosan_yeh_meili_introduction_saisiyat_grammar>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261566": {
+      "content": "<dom_formosan_yi_yang_cheng_kanakanavu_word_level_pro>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261567": {
+      "content": "<dom_formosan_yi_yang_cheng_tense_aspect_agent_markin>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261568": {
+      "content": "<dom_formosan_yoshiro_nihira_bunun_vocabulary>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261569": {
+      "content": "<dom_formosan_zeng_shifen_reduplication_affixation_pa>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261570": {
+      "content": "<dom_formosan_zhan_sujuan_taiwan_indigenous_peoples_h>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261571": {
+      "content": "<dom_formosan_zhang_xiujuan_introduction_paiwan_gramm>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261572": {
       "content": "<dom_formosan_zhang_yongli_pan_jiarong_introduction_t>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261573": {
+      "content": "<dom_formosan_zhao_shanhe_li_taiyuan_zhu_qingyi_abori>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261574": {
       "content": "<dom_formosan_zheng_acl_2024>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261575": {
       "content": "<dom_formosan_zheng_data>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261576": {
+      "content": "<dom_formosan_zheng_yumei_zhu_qingyi_bo_hongming_abor>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261577": {
       "content": "<dom_learning_vocab>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261578": {
       "content": "<dom_nine_level>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261579": {
       "content": "<dom_ntu>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261580": {
       "content": "<dom_picture_book>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261581": {
       "content": "<dom_picture_story>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261582": {
       "content": "<dom_presidential_apology>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261583": {
       "content": "<dom_reading_writing>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261584": {
       "content": "<dom_unknown>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261585": {
       "content": "<dom_youtube>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261586": {
       "content": "<src_ami>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261587": {
       "content": "<src_bnn>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261588": {
       "content": "<src_ckv>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261589": {
       "content": "<src_dru>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261590": {
       "content": "<src_eng>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261591": {
       "content": "<src_pwn>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261592": {
       "content": "<src_pyu>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261593": {
       "content": "<src_ssf>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261594": {
       "content": "<src_sxr>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261595": {
       "content": "<src_szy>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261596": {
       "content": "<src_tao>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261597": {
       "content": "<src_tay>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261598": {
       "content": "<src_trv>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261599": {
       "content": "<src_tsu>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261600": {
       "content": "<src_xnb>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261601": {
       "content": "<src_xsy>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261602": {
       "content": "<src_zh>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261603": {
       "content": "<to_ami>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261604": {
       "content": "<to_bnn>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261605": {
       "content": "<to_ckv>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261606": {
       "content": "<to_dru>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261607": {
       "content": "<to_eng>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261608": {
       "content": "<to_pwn>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261609": {
       "content": "<to_pyu>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261610": {
       "content": "<to_ssf>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261611": {
       "content": "<to_sxr>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261612": {
       "content": "<to_szy>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261613": {
       "content": "<to_tao>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261614": {
       "content": "<to_tay>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261615": {
       "content": "<to_trv>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261616": {
       "content": "<to_tsu>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261617": {
       "content": "<to_xnb>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261618": {
       "content": "<to_xsy>",
       "lstrip": false,
       "normalized": false,
       "single_word": false,
       "special": true
     },
+    "261619": {
       "content": "<to_zh>",
       "lstrip": false,
       "normalized": false,
     "tsu_Latn",
     "xnb_Latn",
     "xsy_Latn",
     "<dialect_central>",
     "<dialect_coastal>",
     "<dialect_dawu>",
     "<dom_culture>",
     "<dom_dictionary>",
     "<dom_essays>",
     "<dom_formosan_100_paiwan_texts>",
     "<dom_formosan_academia_sinica_oral_legends>",
     "<dom_formosan_amis_adversative_constructions>",
+    "<dom_formosan_amis_huang_english_phonology_grammar>",
     "<dom_formosan_amis_kavalan_lin_interrogative_verbs>",
     "<dom_formosan_amis_kuo_sung_comparative_constructions>",
     "<dom_formosan_amis_lifok_dongi_saopo_kimakimad>",
     "<dom_formosan_amis_myths_and_customs>",
     "<dom_formosan_amis_pa_verbs>",
+    "<dom_formosan_anna_hsiou_chuan_chang_reference_gramma>",
+    "<dom_formosan_anton_quack_puyuma_pulingaw>",
+    "<dom_formosan_atayal_you_nao_savak_raxayal>",
+    "<dom_formosan_boyizhenu_narrative_oral_literature_tap>",
     "<dom_formosan_bunun_debusser_dissertation>",
     "<dom_formosan_bunun_topic_focus>",
+    "<dom_formosan_catherine_tseng_seediq_atayal_foods_the>",
+    "<dom_formosan_chen_yaohan_miss_virginia_fey_played_es>",
+    "<dom_formosan_chun_mei_chen_comparative_phonology_pai>",
+    "<dom_formosan_chunming_wu_two_types_of_noun_incorpora>",
     "<dom_formosan_cip_atayal_grammar_overview>",
+    "<dom_formosan_claire_mcgill_a_brief_tayal_vocabulary>",
+    "<dom_formosan_david_blundell_ed_austronesian_taiwan>",
     "<dom_formosan_dean_johnson_year_clouds_smangus>",
+    "<dom_formosan_deng_fangqing_introduction_puyuma_gramm>",
+    "<dom_formosan_der_hwa_victoria_rau_grammar_atayal>",
+    "<dom_formosan_dong_manv_yami_ode_to_taro>",
+    "<dom_formosan_dong_yijia_tjuwabar_paiwan_ritual_langu>",
     "<dom_formosan_elizabeth_zeitoun_chen_huei_wu_overview>",
+    "<dom_formosan_elizabeth_zeitoun_lillian_huang_marie_y>",
+    "<dom_formosan_elizabeth_zeitoun_pronominal_system_man>",
+    "<dom_formosan_elizabeth_zeitoun_squib_dynamic_vs_stat>",
     "<dom_formosan_epark>",
     "<dom_formosan_gitbook_translations>",
     "<dom_formosan_glosbe>",
+    "<dom_formosan_ho_dan_phonological_system_butonglu_pai>",
     "<dom_formosan_holmer_seediq>",
+    "<dom_formosan_hsiu_chuan_liao_transitivity_ergativity>",
     "<dom_formosan_huang_grammaticalization_squliq_atayal>",
+    "<dom_formosan_huang_hui_chuan_glide_formation_takitud>",
+    "<dom_formosan_huang_huijuan_shi_chaokai_introduction>",
     "<dom_formosan_huang_mei_chin_atayal_reference_grammar>",
     "<dom_formosan_huteson_rukai_survey>",
     "<dom_formosan_ilrdf_42_language_practice_word_lists>",
     "<dom_formosan_ilrdf_tousvusvutu_kita>",
+    "<dom_formosan_indigenous_education_resource_center_ke>",
     "<dom_formosan_indigenous_language_literary_awards>",
+    "<dom_formosan_isabelle_bril_roots_and_stems_lexical_a>",
+    "<dom_formosan_jian_iwan_guo_qingliu_life_history>",
+    "<dom_formosan_jian_shilang_introduction_thao_grammar>",
+    "<dom_formosan_john_u_wolff_the_proto_austronesian_pho>",
+    "<dom_formosan_josiane_cauquelin_aborigines_taiwan_puy>",
     "<dom_formosan_kanakanavu_texts>",
     "<dom_formosan_kavalan_zhang_reference_grammar>",
+    "<dom_formosan_kolas_foting_aboriginal_education_world>",
+    "<dom_formosan_kucapungane>",
+    "<dom_formosan_kumu_tapas_tribal_memory_oral_history_w>",
+    "<dom_formosan_lei_shih_family_system_paiwan_at_su_pai>",
+    "<dom_formosan_li_may_sung_budai_rukai_exclamatives>",
+    "<dom_formosan_li_may_sung_clausal_nominalization_buda>",
     "<dom_formosan_li_peirong_xu_weisheng_introduction_tru>",
     "<dom_formosan_li_peizhen_wu_xingyu_shouhu_xing>",
+    "<dom_formosan_lillian_huang_atayal_participants_cross>",
+    "<dom_formosan_liu_manyi_kaskun_ata_matas_i_ki_quaz>",
+    "<dom_formosan_malcom_ross_proto_austronesian_verbal_m>",
+    "<dom_formosan_maya_yeh_blaq_uv_construction_atayal>",
+    "<dom_formosan_nanang_tadaw_naci_mowna_pgagu>",
     "<dom_formosan_naomi_tsukida_correlative_clauses_seedi>",
     "<dom_formosan_nowbucyang_truku_thesis>",
+    "<dom_formosan_ochiai_izumi_numerals_paran_seediq_rela>",
     "<dom_formosan_old_texts>",
     "<dom_formosan_paiwan_collart_zeitoun_time_reference>",
     "<dom_formosan_paiwan_ho_five_dialects>",
     "<dom_formosan_paiwanstories>",
+    "<dom_formosan_patricia_stanley_morphophonemics_of_ver>",
+    "<dom_formosan_paul_jen_kuei_li_internal_relationships>",
+    "<dom_formosan_paul_jen_kuei_li_rukai_structure>",
+    "<dom_formosan_paul_jen_kuei_li_the_preglottalised_sto>",
+    "<dom_formosan_paul_li_atayalic_final_voiced_stops>",
+    "<dom_formosan_paul_li_position_atayal_austronesian>",
+    "<dom_formosan_pingdong_county_government_kai_na_sepuc>",
     "<dom_formosan_puyuma_katipol_kopfjagdriten>",
     "<dom_formosan_puyuma_teng_reference_grammar>",
+    "<dom_formosan_raleigh_ferrell_construction_markers_fo>",
     "<dom_formosan_rik_bunun>",
+    "<dom_formosan_robert_blust_the_austronesian_languages>",
     "<dom_formosan_saaroa_pan_grammar>",
     "<dom_formosan_sakizaya_affixes>",
     "<dom_formosan_seals>",
     "<dom_formosan_seediq_zhang_reference_grammar>",
+    "<dom_formosan_shen_wenqi_introduction_sakizaya_gramma>",
     "<dom_formosan_shigeru_tsuchida_reconstruction_proto_t>",
     "<dom_formosan_shih_rukai_adverbial>",
+    "<dom_formosan_sing_olam_hong_tinglan_aboriginal_educa>",
     "<dom_formosan_song_limei_introduction_kanakanavu_gram>",
     "<dom_formosan_song_limei_introduction_seediq_grammar>",
+    "<dom_formosan_soren_egerod_statement_atayal_phonology>",
+    "<dom_formosan_stacy_teng_malcom_ross_is_puyuma_primar>",
+    "<dom_formosan_truku_lowking_demonstratives>",
+    "<dom_formosan_tsai_hsui_liu_complementation_three_lan>",
+    "<dom_formosan_tsai_wei_tien_dylan_conjunctive_reducti>",
+    "<dom_formosan_tsou_descriptive_study>",
+    "<dom_formosan_tung_tsuchida_ting_li_pan_saaroa_texts>",
+    "<dom_formosan_wei_huilin_liu_social_structure_yami>",
     "<dom_formosan_wilang_yutas_videos>",
+    "<dom_formosan_wu_chunming_adverbials_in_paiwan>",
     "<dom_formosan_wu_jinglan_introduction_amis_grammar>",
+    "<dom_formosan_xie_fuhui_introduction_kavalan_grammar>",
     "<dom_formosan_yedda_palemeq_blog>",
     "<dom_formosan_yeddas_blog>",
     "<dom_formosan_yeh_meili_introduction_saisiyat_grammar>",
+    "<dom_formosan_yi_yang_cheng_kanakanavu_word_level_pro>",
+    "<dom_formosan_yi_yang_cheng_tense_aspect_agent_markin>",
+    "<dom_formosan_yoshiro_nihira_bunun_vocabulary>",
+    "<dom_formosan_zeng_shifen_reduplication_affixation_pa>",
+    "<dom_formosan_zhan_sujuan_taiwan_indigenous_peoples_h>",
+    "<dom_formosan_zhang_xiujuan_introduction_paiwan_gramm>",
     "<dom_formosan_zhang_yongli_pan_jiarong_introduction_t>",
+    "<dom_formosan_zhao_shanhe_li_taiyuan_zhu_qingyi_abori>",
     "<dom_formosan_zheng_acl_2024>",
     "<dom_formosan_zheng_data>",
+    "<dom_formosan_zheng_yumei_zhu_qingyi_bo_hongming_abor>",
     "<dom_learning_vocab>",
     "<dom_nine_level>",
     "<dom_ntu>",

training_profile.json ADDED Viewed

	@@ -0,0 +1,94 @@

+{
+  "schema_version": 3,
+  "recipe_id": "nllb200-spm8k-directional-v3",
+  "model_family": "nllb",
+  "corpus_pipeline_version": "formosan-mt-corpus-v3",
+  "mt_standardization": {
+    "id": "formosan-mt-standard-v3",
+    "sha256": "4bbded87eb4833b2d1cd5a88a1f2b059560416e0e69ecfa55433a77f418c903e",
+    "namespace": "formosan-mt"
+  },
+  "project_root": ".",
+  "corpus_variants": ["public_no_bible", "private_no_bible"],
+  "base_model": {
+    "name": "facebook/nllb-200-distilled-600M",
+    "revision": "f8d333a098d19b4fd9a8b18f94170487ad3f821d"
+  },
+  "primary_input_pattern": "corpus_builds/{corpus_name}/pivot_corpora_final/big_corpus_{target}_in_domain_hard.csv",
+  "targets": {
+    "english": {
+      "input_pattern": "corpus_builds/{corpus_name}/pivot_corpora_final/big_corpus_en_in_domain_hard.csv",
+      "target_col": "english_sentence",
+      "directions": ["f2en", "en2f"],
+      "selected_recipe": "NLLB-200 SPM8k directional"
+    },
+    "chinese": {
+      "input_pattern": "corpus_builds/{corpus_name}/pivot_corpora_final/big_corpus_zh_in_domain_hard.csv",
+      "target_col": "chinese_sentence",
+      "directions": ["f2zh", "zh2f"],
+      "selected_recipe": "NLLB-200 SPM8k directional"
+    }
+  },
+  "splits": {
+    "english_output_dir": "formosan_mt_experiments/data/splits_en_v1",
+    "chinese_output_dir": "formosan_mt_experiments/data/splits_zh_v1",
+    "train_ratio": 0.9,
+    "validate_ratio": 0.025,
+    "test_ratio": 0.075,
+    "min_test_rows": 500,
+    "min_validate_rows": 150,
+    "character_ngram_jaccard_threshold": 0.82,
+    "tiers": ["in_domain_hard"],
+    "headline_tier": "in_domain_hard"
+  },
+  "tokenizer": {
+    "mode": "spm",
+    "spm_vocab_sweep": [8192],
+    "default_spm_vocab": 8192,
+    "chinese_default_spm_vocab": 8192,
+    "min_char_frequency": 3,
+    "max_dialect_tags": 200,
+    "min_dialect_frequency": 3,
+    "training_columns": ["formosan_sentence"],
+    "setup_splits": ["train"]
+  },
+  "training_defaults": {
+    "steps": 300000,
+    "batch_size": 16,
+    "grad_accum_steps": 4,
+    "effective_batch_size": 64,
+    "max_length": 384,
+    "learning_rate": 0.00002,
+    "warmup_steps": 4000,
+    "weight_decay": 0.001,
+    "alpha": 0.5,
+    "precision": "bf16",
+    "label_smoothing": 0.1,
+    "save_interval": 0,
+    "log_interval": 500,
+    "generation_eval_interval": 10000,
+    "generation_eval_samples_per_language": 128,
+    "generation_eval_batch_size": 16,
+    "validation_beam": 2,
+    "best_metric": "chrF2",
+    "early_stopping_patience": 5,
+    "early_stopping_min_delta": 0.05,
+    "early_stopping_start_step": 30000,
+    "f2en_easy_source_weight": 0.05,
+    "en2f_easy_source_weight": 0.15,
+    "f2zh_easy_source_weight": 0.05,
+    "zh2f_easy_source_weight": 0.15
+  },
+  "generation_defaults": {
+    "beam": 4,
+    "max_length": 384,
+    "max_new_tokens": 256,
+    "min_new_tokens": 1,
+    "no_repeat_ngram_size": 0,
+    "repetition_penalty": 1.0,
+    "length_penalty": 1.0,
+    "metadata_modes": ["default", "oracle"],
+    "bootstrap_samples": 200,
+    "bootstrap_seed": 42
+  }
+}