--- license: cc-by-nc-sa-4.0 language: - bo tags: - seq2seq - normalisation - classical-tibetan - historical-text - encoder-decoder - transformer - low-resource - digital-humanities pipeline_tag: text-generation --- # Tibetan Normalisation - S2S Model (Non-Tokenised) A character-level sequence-to-sequence (S2) encoder-decoder transformer model for the normalisation of Old/Classical Tibetan, converting diplomatic (non-standard, abbreviated) Tibetan manuscript text into Standard Classical Tibetan. This is the **non-tokenised** variant of the model — input and output are raw Tibetan Unicode character sequences without prior word segmentation. This model is part of the [PaganTibet](https://www.pagantibet.com/) project and accompanies the paper: Meelen, M. & Griffiths, R.M. (2026) 'Historical Tibetan Normalisation: rule-based vs neural & n-gram LM methods for extremely low-resource languages' in *Proceedings of the AI4CHIEF conference*, Springer. Please cite the paper and the [code repository](https://github.com/pagantibet/normalisation) when using this model. --- ## Model Overview Old/Classical Tibetan manuscripts present major normalisation challenges: extensive abbreviations, non-standard orthography, scribal variation, and a near-complete absence of gold-standard parallel data. This model addresses these challenges using a hybrid approach combining a neural sequence-to-sequence transformer with optional rule-based pre-/post-processing and KenLM n-gram language model ranking (the latter applied at inference time; see the [Inference scripts](https://github.com/pagantibet/normalisation/tree/main/Inference)). The model operates at the character level on non-tokenised input, meaning it processes raw Tibetan Unicode strings (syllables and punctuation) without word segmentation. Results from Meelen & Griffiths (2026) show that, for non-tokenised text, the neural model alone performs reasonably well on standard Buddhist texts, while more challenging diplomatic corpora benefit from the addition of rule-based processing and n-gram ranking. ### Architecture - **Type**: Character-level encoder-decoder transformer (Seq2Seq) - **Layers**: 4 - **Attention heads**: 8 - **Optimiser**: Adam (lr = 0.0005, β1 = 0.9, β2 = 0.997) - **Label smoothing**: 0.1 - **Framework**: PyTorch - **Training hardware**: RTX ADA 6000 GPU (~5–6 hours training time) Full hyperparameter settings are reported in the Appendix of Meelen & Griffiths (2026). --- ## Training Data The model was trained on the dataset [`pagantibet/normalisation-S2S-training`](https://huggingface.co/datasets/pagantibet/normalisation-S2S-training) (~2 million rows), which combines: 1. **Gold-standard data**: 7,421 manually normalised parallel sentence pairs from the PaganTibet corpus. 2. **Augmented data**: The gold data was substantially expanded using four data augmentation strategies, each designed to simulate the kinds of variation found in historical Tibetan manuscripts: - **Random noise injection**: Probabilistic character substitutions, diacritic variations, and orthographic inconsistencies calibrated to realistic manuscript variation frequencies (following [Huang et al. 2023](https://www.isca-archive.org/sigul_2023/huang23_sigul.html)). - **OCR-based noise simulation**: OCR-realistic noise patterns generated using the [nlpaug](https://github.com/makcedward/nlpaug) library. - **Rule-based diplomatic transformations**: Stochastic application of character replacements reflecting common scribal conventions in historical Tibetan manuscripts. - **Dictionary-based augmentation**: Insertion of entries from a [custom Tibetan abbreviation dictionary](https://huggingface.co/datasets/pagantibet/Tibetan-abbreviation-dictionary) (~10,000 abbreviation–expansion pairs) to help the model learn abbreviation resolution. Additional training data was derived from the Standard Classical Tibetan ACTib corpus (>180 million words; [Meelen & Roux 2020](https://zenodo.org/records/3951503)), processed into manuscript-length lines using the `createTiblines.py` script. Full details of the data preparation and augmentation pipeline are described in the [GitHub repository](https://github.com/pagantibet/normalisation). --- ## Intended Use This model is intended for: - **Normalisation of diplomatic Classical Tibetan texts** into Standard Classical Tibetan, as a preprocessing step for downstream NLP tasks (e.g. tokenisation, tagging, translation). - **Digital humanities** workflows for processing historical Tibetan manuscripts, particularly texts with heavy abbreviation or non-standard orthography. - **Research** on low-resource historical text normalisation. **Note on pipeline order**: Per the results in Meelen & Griffiths (2026), it is recommended to apply normalisation *before* tokenisation in the processing pipeline. For challenging diplomatic texts, combining this model with the KenLM n-gram ranker and rule-based pre/post-processing (see [Inference](https://github.com/pagantibet/normalisation/tree/main/Inference)) yields the best results. --- ## How to Use The model can be used with the inference scripts provided in the [PaganTibet normalisation repository](https://github.com/pagantibet/normalisation/tree/main/Inference). Six inference modes are available, ranging from rule-based only to combined neural + n-gram + rule-based pipelines: ```bash # Run on a GPU cluster via Slurm sbatch tibetan-inference-flexible.sh # Or run directly python3 tibetan-inference-flexible.py ``` See the [Inference ReadMe](https://github.com/pagantibet/normalisation/blob/main/Inference/FlexibleInference_ReadMe.md) for full usage details and configuration options. --- ## Evaluation The training script includes a built-in beam search evaluation. Separate evaluation is available via the evaluation scripts, which reports: - **CER** (Character Error Rate) - **Precision, Recall, F1** - **Correction Precision (CP) and Correction Recall (CR)** (following [Huang et al. 2023](https://www.isca-archive.org/sigul_2023/huang23_sigul.html)) for a more accurate picture of normalisation effectiveness - **Bootstrapped Confidence Intervals** (1,000 iterations) for small test sets (optional) Two versions of the evaluation script are available: - evaluate_model.py — the standard script - evaluate-model-withCIs.py — an extended version that additionally computes 95% bootstrap confidence intervals (CI) for all metrics ```bash sbatch evaluate-model.sh # or python3 evaluate_model.py # or python3 evaluate-model-withCIs.py ``` Full evaluation results including confidence intervals and example predictions are available in the [non-tokenised Evaluations directory](https://github.com/pagantibet/normalisation/tree/main/Evaluations/Gold-nontokenised-CI) of the repository. --- ## Related Models and Resources All models and datasets from the PaganTibet normalisation project are collected in the [Normalisation collection on Hugging Face](https://huggingface.co/collections/pagantibet/normalisation). | Resource | Link | |---|---| | Training dataset | [`pagantibet/normalisation-S2S-training`](https://huggingface.co/datasets/pagantibet/normalisation-S2S-training) | | Abbreviation dictionary | [`pagantibet/Tibetan-abbreviation-dictionary`](https://huggingface.co/datasets/pagantibet/Tibetan-abbreviation-dictionary) | | Training & inference code | [github.com/pagantibet/normalisation](https://github.com/pagantibet/normalisation) | | ACTib corpus | [Zenodo (Meelen & Roux 2020)](https://zenodo.org/records/3951503) | | PaganTibet project | [pagantibet.com](https://www.pagantibet.com/) | --- ## License This model is released under [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/). It may be used freely for non-commercial research and educational purposes, with attribution and under the same licence terms. --- ## Funding This work was partially funded by the European Union (ERC, Pagan Tibet, grant no. 101097364). Views and opinions expressed are those of the authors only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency.