Upload folder using huggingface_hub

README.md

@@ -0,0 +1,224 @@
+---
+language:
+- la
+tags:
+- text2text-generation
+- punctuation-restoration
+- seq2seq
+- latin
+- historical-text
+license: mit
+library_name: transformers
+pipeline_tag: text2text-generation
+base_model: google/mt5-large
+widget:
+- text: "punctuate: gallia est omnis divisa in partes tres quarum unam incolunt belgae aliam aquitani tertiam qui ipsorum lingua celtae nostra galli appellantur"
+  example_title: "Caesar (Classical)"
+- text: "punctuate: consilium ad conceptionem mondini bononiensis de leuciis quia ex relatis principio sterilitatis causam fore duplicem horum coniugatorum unam ocultam scilicet disproportionem eorum nobis ignotam cuius remotioni altissimus provideat quia nostrum non est nec posse alicuius artis et aliam manifestam existimo esse complectionem eorum declinare ad frigidum et declinabilem in processu etatis etiam ad frigidum et humidum humiditatem accidentali corporibus eorum est subveniendum triplici instrumento medici declinante ad calidum et siccum quo ad medicinas et ad calidum et humidum quantum ad dictam"
+  example_title: "Mondino de' Liuzzi (Medieval Medical)"
+---
+
+# mT5 Latin Punctuator (mt5-large)
+
+## Model Description
+
+This model is a fine-tuned version of **[google/mt5-large](https://huggingface.co/google/mt5-large)** designed for **automatic punctuation restoration in Latin text**.
+
+It takes unpunctuated Latin text as input and generates a fully punctuated version of the same text, adding sentence boundaries and internal punctuation. The model was optimized for **Classical and Medieval Latin prose** (approx. 400–1500 AD), making it suitable for digital editions and post-processing OCR/ATR outputs.
+
+* **Task:** Sequence-to-Sequence Punctuation Restoration
+* **Base Model:** mT5-Large
+* **Language:** Latin (`la`)
+* **Developer:** Michael Schonhardt (TU Darmstadt)
+
+> **Note:** This model operates on continuous text streams. While it restores sentence boundaries (`.`, `?`, `!`) and capitalization, it is not designed to predict paragraph breaks or layout features.
+
+---
+
+## Intended Use
+
+### Primary Use Cases
+* **OCR Post-processing:** Restoring punctuation to raw OCR output from early printed books or manuscripts.
+* **NLP Pipeline Preparation:** Pre-formatting Latin text for downstream tasks like Machine Translation or Named Entity Recognition.
+* **Pedagogical Aid:** Helping students or readers navigate raw Latin texts by providing suggested sentence structures.
+
+### Out-of-Scope / Limitations
+* **Not a Critical Editor:** The model makes probabilistic guesses based on training data. In ambiguous cases, it may insert punctuation that disagrees with specific editorial conventions.
+* **Diplomatic Transcription:** The model expects expanded spelling based on normalisation conventions and may struggle with raw diplomatic transcriptions containing heavy abbreviations.
+
+---
+
+## Usage
+
+**Important:** The model was trained with the prefix `"punctuate: "` and expects lowercased input. The helper function below handles this automatically.
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
+
+device = "cuda" if torch.cuda.is_available() else "cpu"
+
+model_name = "mschonhardt/mt5-latin-punctuator-large" 
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForSeq2SeqLM.from_pretrained(model_name).to(device)
+
+def punctuate(text: str) -> str:
+    # Preprocessing: Add prefix and lowercase as per training script
+    input_text = "punctuate: " + text.lower()
+    
+    inputs = tokenizer(
+        input_text,
+        return_tensors="pt",
+        truncation=True,
+        max_length=1024,
+    ).to(device)
+
+    with torch.no_grad():
+        output_ids = model.generate(
+            **inputs,
+            max_length=1024,
+            num_beams=4,
+            early_stopping=True,
+        )
+    return tokenizer.decode(output_ids[0], skip_special_tokens=True)
+
+# Example usage
+text = "gallia est omnis divisa in partes tres"
+print(punctuate(text))
+# Output: Gallia est omnis divisa in partes tres.
+```
+
+---
+
+## Training Data
+
+The model was trained on a custom dataset derived from the **[Latin Text Archive (LTA)](https://lta.bbaw.de)** as well as data created as part of the **[Burchards Dekret Digital Project](https://burchards-dekret-digital.de)**.
+
+### Data Preparation & Cleaning
+The dataset was processed using the following pipeline:
+1.  **Artifact Removal:** Line number references (e.g., "667 - 668") were removed via regex.
+2.  **Normalization:** Whitespace was normalized to single spaces.
+3.  **Source/Target Formatting:**
+    * **Source:** Text was **lowercased**, punctuation (`.,;?!`) was removed, and the prefix `"punctuate: "` was added.
+    * **Target:** Original text with case and punctuation preserved.
+4.  **Splitting:** The data was split into **Train**, **Validation**, and **Test** sets using a random seed of 42.
+
+### Data Source & Acknowledgements
+We gratefully acknowledge that the training data originates from the **[Latin Text Archive (LTA)](http://lta.bbaw.de)** (**Prof. Dr. Bernhard Jussen**, **Dr. Tim Geelhaar**) including data from  Monumenta Germaniae Historica, Corpus Corporum and IRHT. 
+
+---
+
+## Training Procedure
+
+The model was fine-tuned on a high-performance compute node using the `transformers` Seq2SeqTrainer.
+
+### Hardware Configuration
+* **GPU:** NVIDIA L40S (45 GB VRAM)
+* **Precision:** `bfloat16` (BF16) mixed precision
+* **Optimizer:** Adafactor
+
+### Hyperparameters
+
+| Parameter | Value |
+| :--- | :--- |
+| **Batch Size** | 4 |
+| **Grad Accumulation** | 16 steps |
+| **Effective Batch Size** | 64 |
+| **Learning Rate** | 5e-5 (Linear Decay) |
+| **Max Sequence Length** | 1024 (Source and Target) |
+| **Gradient Checkpointing** | True |
+
+### Training Dynamics
+
+**Early Stopping:** Training was stopped at **~1.6 epochs** as validation gains diminished.
+
+**Validation Loss Progression:**
+
+| Epoch | Validation Loss |
+| :--- | :--- |
+| 0.2 | 0.1448 |
+| 0.6 | 0.1137 |
+| 1.0 | 0.1043 |
+| 1.4 | 0.0982 |
+| **1.6** | **0.0973 (Best)** |
+
+---
+
+## Example Outputs
+
+**1. Classical Latin (Caesar)**
+> **Input:** gallia est omnis divisa in partes tres quarum unam incolunt belgae aliam aquitani tertiam qui ipsorum lingua celtae nostra galli appellantur
+> 
+> **Output:** Gallia est omnis divisa in partes tres, quarum unam incolunt Belgae, aliam Aquitani, tertiam qui ipsorum lingua Celtae, nostra Galli appellantur.
+
+**2. Medieval Medical Latin (Mondino de' Liuzzi)**
+> **Input:** Consilium ad conceptionem mondini bononiensis de leuciis quia ex relatis principio sterilitatis causam fore duplicem horum coniugatorum unam ocultam scilicet disproportionem eorum nobis ignotam cuius remotioni altissimus provideat quia nostrum non est nec posse alicuius artis et aliam manifestam existimo esse complectionem eorum declinare ad frigidum et declinabilem in processu etatis etiam ad frigidum et humidum humiditatem accidentali corporibus eorum est subveniendum triplici instrumento medici declinante ad calidum et siccum quo ad medicinas et ad calidum et humidum quantum ad dictam
+>
+> **Output:** Consilium ad conceptionem Mondini Bononiensis de leuciis. Quia ex relatis principio sterilitatis causam fore duplicem horum coniugatorum, unam ocultam, scilicet disproportionem eorum nobis ignotam, cuius remotioni Altissimus provideat, quia nostrum non est nec posse alicuius artis, et aliam manifestam existimo esse complectionem eorum declinare ad frigidum et declinabilem in processu etatis etiam ad frigidum et humidum. Humiditatem accidentali corporibus eorum est subveniendum triplici instrumento medici declinante ad calidum et siccum quo ad medicinas et ad calidum et humidum quantum ad dictam.
+
+---
+
+## Model Development and Funding
+
+This model was developed by Michael Schonhardt as part of the **Digital Editing Toolkit**.
+
+**Affiliation:**
+[Fachgebiet Digitale Editorik und Kulturgeschichte des Mittelalters](https://www.geschichte.tu-darmstadt.de/)  
+(Digital Editing and Cultural History of the Middle Ages)  
+**Technische Universität Darmstadt** *In cooperation with the Academy of Sciences and Literature | Mainz*
+
+**Funding:**
+The development was funded by the **Hessian Ministry of Higher Education, Research, Science and the Arts (HMWK)** within the **LOEWE Exploration** project: *"Embedding the Past"*.
+
+The project focuses on developing transparent and responsible AI methods for the semantic analysis and editorial processing of historical documents.
+
+---
+
+## Citation
+
+If you use this model in your research, please cite it as follows:
+
+```bibtex
+@misc{schonhardt-2025-latin-punctuator,
+  author       = {Schonhardt, Michael},
+  title        = {mT5 Latin Punctuator (mt5-large)},
+  year         = {2025},
+  publisher    = {Hugging Face},
+  howpublished = {\url{[https://huggingface.co/YOUR_USERNAME/mt5-latin-punctuator-large](https://huggingface.co/YOUR_USERNAME/mt5-latin-punctuator-large)}},
+  note         = {Part of the LOEWE Exploration 'Embedding the Past'. Data provided by LTA}
+}
+```
+
+Please also cite the original mT5 paper:
+
+```bibtex
+@inproceedings{xue-etal-2021-mt5,
+    title = "m{T}5: A Massively Multilingual Pre-trained Text-to-Text Transformer",
+    author = "Xue, Linting  and
+      Constant, Noah  and
+      Roberts, Adam  and
+      Kale, Mihir  and
+      Al-Rfou, Rami  and
+      Siddhant, Aditya  and
+      Barua, Aditya  and
+      Raffel, Colin",
+    editor = "Toutanova, Kristina  and
+      Rumshisky, Anna  and
+      Zettlemoyer, Luke  and
+      Hakkani-Tur, Dilek  and
+      Beltagy, Iz  and
+      Bethard, Steven  and
+      Cotterell, Ryan  and
+      Chakraborty, Tanmoy  and
+      Zhou, Yichao",
+    booktitle = "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
+    month = jun,
+    year = "2021",
+    address = "Online",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2021.naacl-main.41/",
+    doi = "10.18653/v1/2021.naacl-main.41",
+    pages = "483--498"
+    
+}
+```

config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "MT5ForConditionalGeneration"
+  ],
+  "classifier_dropout": 0.0,
+  "d_ff": 2816,
+  "d_kv": 64,
+  "d_model": 1024,
+  "decoder_start_token_id": 0,
+  "dense_act_fn": "gelu_new",
+  "dropout_rate": 0.1,
+  "dtype": "float32",
+  "eos_token_id": 1,
+  "feed_forward_proj": "gated-gelu",
+  "initializer_factor": 1.0,
+  "is_encoder_decoder": true,
+  "is_gated_act": true,
+  "layer_norm_epsilon": 1e-06,
+  "model_type": "mt5",
+  "num_decoder_layers": 24,
+  "num_heads": 16,
+  "num_layers": 24,
+  "output_past": true,
+  "pad_token_id": 0,
+  "relative_attention_max_distance": 128,
+  "relative_attention_num_buckets": 32,
+  "tie_word_embeddings": false,
+  "tokenizer_class": "T5Tokenizer",
+  "transformers_version": "4.57.3",
+  "use_cache": true,
+  "vocab_size": 250112
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "decoder_start_token_id": 0,
+  "eos_token_id": [
+    1
+  ],
+  "pad_token_id": 0,
+  "transformers_version": "4.57.3"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b705a9a1866adee4c6cfb64188963ce71d95beca0efed65dc0d0b4188e1f61fa
+size 4918393736

optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9d8283ad3820fe2debd6a0d0d38dc8cefcc57c9ad80c6c789badc9a4d90627e9
+size 7521587

rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:723ee72669283e1fa704467f831a4cfbf3f4a1a570dcc070291e8cf4ee042a12
+size 14645

scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e356711d012b62f57d5b6cac0c8d9d7f255705a52bc3309afc8bd788c08088fc
+size 1465

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

spiece.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ef78f86560d809067d12bac6c09f19a462cb3af3f54d2b8acbba26e1433125d6
+size 4309802

tokenizer_config.json

@@ -0,0 +1,40 @@
+{
+  "add_prefix_space": true,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [],
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "extra_ids": 0,
+  "extra_special_tokens": {},
+  "legacy": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "T5Tokenizer",
+  "unk_token": "<unk>"
+}

trainer_state.json

@@ -0,0 +1,658 @@
+{
+  "best_global_step": 8000,
+  "best_metric": 0.09730422496795654,
+  "best_model_checkpoint": "./mt5_latin_punctuator_large/checkpoint-8000",
+  "epoch": 1.6045835213880948,
+  "eval_steps": 1000,
+  "global_step": 8000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.020059174564966652,
+      "grad_norm": 13.09343147277832,
+      "learning_rate": 4.9669073405535504e-05,
+      "loss": 15.4848,
+      "step": 100
+    },
+    {
+      "epoch": 0.040118349129933305,
+      "grad_norm": 0.4526735842227936,
+      "learning_rate": 4.9334804118197625e-05,
+      "loss": 0.66,
+      "step": 200
+    },
+    {
+      "epoch": 0.060177523694899954,
+      "grad_norm": 0.4390902817249298,
+      "learning_rate": 4.900053483085975e-05,
+      "loss": 0.385,
+      "step": 300
+    },
+    {
+      "epoch": 0.08023669825986661,
+      "grad_norm": 0.3624674677848816,
+      "learning_rate": 4.866626554352187e-05,
+      "loss": 0.3017,
+      "step": 400
+    },
+    {
+      "epoch": 0.10029587282483325,
+      "grad_norm": 0.2848065197467804,
+      "learning_rate": 4.833199625618399e-05,
+      "loss": 0.2648,
+      "step": 500
+    },
+    {
+      "epoch": 0.12035504738979991,
+      "grad_norm": 0.2836224138736725,
+      "learning_rate": 4.7997726968846105e-05,
+      "loss": 0.2419,
+      "step": 600
+    },
+    {
+      "epoch": 0.14041422195476655,
+      "grad_norm": 0.3000536561012268,
+      "learning_rate": 4.7663457681508226e-05,
+      "loss": 0.2238,
+      "step": 700
+    },
+    {
+      "epoch": 0.16047339651973322,
+      "grad_norm": 0.6995307803153992,
+      "learning_rate": 4.732918839417035e-05,
+      "loss": 0.2157,
+      "step": 800
+    },
+    {
+      "epoch": 0.18053257108469986,
+      "grad_norm": 0.1916164755821228,
+      "learning_rate": 4.699491910683247e-05,
+      "loss": 0.2061,
+      "step": 900
+    },
+    {
+      "epoch": 0.2005917456496665,
+      "grad_norm": 0.3005019426345825,
+      "learning_rate": 4.666064981949459e-05,
+      "loss": 0.2001,
+      "step": 1000
+    },
+    {
+      "epoch": 0.2005917456496665,
+      "eval_loss": 0.1447821706533432,
+      "eval_runtime": 687.5913,
+      "eval_samples_per_second": 25.778,
+      "eval_steps_per_second": 6.446,
+      "step": 1000
+    },
+    {
+      "epoch": 0.22065092021463317,
+      "grad_norm": 0.34013351798057556,
+      "learning_rate": 4.632638053215671e-05,
+      "loss": 0.1911,
+      "step": 1100
+    },
+    {
+      "epoch": 0.24071009477959981,
+      "grad_norm": 0.19066131114959717,
+      "learning_rate": 4.5992111244818834e-05,
+      "loss": 0.1868,
+      "step": 1200
+    },
+    {
+      "epoch": 0.26076926934456646,
+      "grad_norm": 0.1935589760541916,
+      "learning_rate": 4.565784195748095e-05,
+      "loss": 0.1828,
+      "step": 1300
+    },
+    {
+      "epoch": 0.2808284439095331,
+      "grad_norm": 0.31047841906547546,
+      "learning_rate": 4.532357267014307e-05,
+      "loss": 0.1798,
+      "step": 1400
+    },
+    {
+      "epoch": 0.3008876184744998,
+      "grad_norm": 0.24223153293132782,
+      "learning_rate": 4.498930338280519e-05,
+      "loss": 0.1751,
+      "step": 1500
+    },
+    {
+      "epoch": 0.32094679303946644,
+      "grad_norm": 1.5393741130828857,
+      "learning_rate": 4.465503409546731e-05,
+      "loss": 0.1704,
+      "step": 1600
+    },
+    {
+      "epoch": 0.3410059676044331,
+      "grad_norm": 0.21514640748500824,
+      "learning_rate": 4.4320764808129435e-05,
+      "loss": 0.1689,
+      "step": 1700
+    },
+    {
+      "epoch": 0.3610651421693997,
+      "grad_norm": 0.17480678856372833,
+      "learning_rate": 4.3986495520791556e-05,
+      "loss": 0.1664,
+      "step": 1800
+    },
+    {
+      "epoch": 0.38112431673436636,
+      "grad_norm": 0.1928788721561432,
+      "learning_rate": 4.365222623345368e-05,
+      "loss": 0.1646,
+      "step": 1900
+    },
+    {
+      "epoch": 0.401183491299333,
+      "grad_norm": 0.20150336623191833,
+      "learning_rate": 4.331795694611579e-05,
+      "loss": 0.1616,
+      "step": 2000
+    },
+    {
+      "epoch": 0.401183491299333,
+      "eval_loss": 0.11908746510744095,
+      "eval_runtime": 687.2579,
+      "eval_samples_per_second": 25.791,
+      "eval_steps_per_second": 6.449,
+      "step": 2000
+    },
+    {
+      "epoch": 0.4212426658642997,
+      "grad_norm": 0.17183753848075867,
+      "learning_rate": 4.2983687658777914e-05,
+      "loss": 0.1632,
+      "step": 2100
+    },
+    {
+      "epoch": 0.44130184042926635,
+      "grad_norm": 0.2742299735546112,
+      "learning_rate": 4.2649418371440035e-05,
+      "loss": 0.1578,
+      "step": 2200
+    },
+    {
+      "epoch": 0.461361014994233,
+      "grad_norm": 0.22588330507278442,
+      "learning_rate": 4.231514908410216e-05,
+      "loss": 0.1563,
+      "step": 2300
+    },
+    {
+      "epoch": 0.48142018955919963,
+      "grad_norm": 0.25857874751091003,
+      "learning_rate": 4.198087979676428e-05,
+      "loss": 0.1536,
+      "step": 2400
+    },
+    {
+      "epoch": 0.5014793641241663,
+      "grad_norm": 0.2725016474723816,
+      "learning_rate": 4.16466105094264e-05,
+      "loss": 0.155,
+      "step": 2500
+    },
+    {
+      "epoch": 0.5215385386891329,
+      "grad_norm": 0.14976142346858978,
+      "learning_rate": 4.131234122208852e-05,
+      "loss": 0.1507,
+      "step": 2600
+    },
+    {
+      "epoch": 0.5415977132540996,
+      "grad_norm": 0.16467268764972687,
+      "learning_rate": 4.0978071934750636e-05,
+      "loss": 0.1505,
+      "step": 2700
+    },
+    {
+      "epoch": 0.5616568878190662,
+      "grad_norm": 0.16063468158245087,
+      "learning_rate": 4.064380264741276e-05,
+      "loss": 0.147,
+      "step": 2800
+    },
+    {
+      "epoch": 0.581716062384033,
+      "grad_norm": 0.23703046143054962,
+      "learning_rate": 4.030953336007488e-05,
+      "loss": 0.1445,
+      "step": 2900
+    },
+    {
+      "epoch": 0.6017752369489996,
+      "grad_norm": 0.194139301776886,
+      "learning_rate": 3.9975264072737e-05,
+      "loss": 0.1444,
+      "step": 3000
+    },
+    {
+      "epoch": 0.6017752369489996,
+      "eval_loss": 0.11365870386362076,
+      "eval_runtime": 687.3052,
+      "eval_samples_per_second": 25.789,
+      "eval_steps_per_second": 6.448,
+      "step": 3000
+    },
+    {
+      "epoch": 0.6218344115139662,
+      "grad_norm": 0.2936673164367676,
+      "learning_rate": 3.964099478539912e-05,
+      "loss": 0.1443,
+      "step": 3100
+    },
+    {
+      "epoch": 0.6418935860789329,
+      "grad_norm": 0.15139873325824738,
+      "learning_rate": 3.9306725498061244e-05,
+      "loss": 0.1415,
+      "step": 3200
+    },
+    {
+      "epoch": 0.6619527606438995,
+      "grad_norm": 0.22842614352703094,
+      "learning_rate": 3.8972456210723365e-05,
+      "loss": 0.141,
+      "step": 3300
+    },
+    {
+      "epoch": 0.6820119352088662,
+      "grad_norm": 0.17998695373535156,
+      "learning_rate": 3.863818692338548e-05,
+      "loss": 0.1423,
+      "step": 3400
+    },
+    {
+      "epoch": 0.7020711097738328,
+      "grad_norm": 0.18952590227127075,
+      "learning_rate": 3.83039176360476e-05,
+      "loss": 0.1388,
+      "step": 3500
+    },
+    {
+      "epoch": 0.7221302843387994,
+      "grad_norm": 0.11938633769750595,
+      "learning_rate": 3.796964834870972e-05,
+      "loss": 0.1378,
+      "step": 3600
+    },
+    {
+      "epoch": 0.7421894589037661,
+      "grad_norm": 0.1671166867017746,
+      "learning_rate": 3.7635379061371845e-05,
+      "loss": 0.1389,
+      "step": 3700
+    },
+    {
+      "epoch": 0.7622486334687327,
+      "grad_norm": 0.1272394061088562,
+      "learning_rate": 3.7301109774033966e-05,
+      "loss": 0.1362,
+      "step": 3800
+    },
+    {
+      "epoch": 0.7823078080336994,
+      "grad_norm": 0.19626741111278534,
+      "learning_rate": 3.696684048669609e-05,
+      "loss": 0.1377,
+      "step": 3900
+    },
+    {
+      "epoch": 0.802366982598666,
+      "grad_norm": 0.3008900284767151,
+      "learning_rate": 3.663257119935821e-05,
+      "loss": 0.1347,
+      "step": 4000
+    },
+    {
+      "epoch": 0.802366982598666,
+      "eval_loss": 0.10825913399457932,
+      "eval_runtime": 687.2417,
+      "eval_samples_per_second": 25.792,
+      "eval_steps_per_second": 6.449,
+      "step": 4000
+    },
+    {
+      "epoch": 0.8224261571636328,
+      "grad_norm": 0.45567601919174194,
+      "learning_rate": 3.6298301912020324e-05,
+      "loss": 0.1363,
+      "step": 4100
+    },
+    {
+      "epoch": 0.8424853317285994,
+      "grad_norm": 0.17759402096271515,
+      "learning_rate": 3.5964032624682446e-05,
+      "loss": 0.1358,
+      "step": 4200
+    },
+    {
+      "epoch": 0.862544506293566,
+      "grad_norm": 0.15186358988285065,
+      "learning_rate": 3.562976333734457e-05,
+      "loss": 0.1365,
+      "step": 4300
+    },
+    {
+      "epoch": 0.8826036808585327,
+      "grad_norm": 0.20942702889442444,
+      "learning_rate": 3.529549405000669e-05,
+      "loss": 0.1341,
+      "step": 4400
+    },
+    {
+      "epoch": 0.9026628554234993,
+      "grad_norm": 0.37031808495521545,
+      "learning_rate": 3.496122476266881e-05,
+      "loss": 0.1325,
+      "step": 4500
+    },
+    {
+      "epoch": 0.922722029988466,
+      "grad_norm": 0.13902556896209717,
+      "learning_rate": 3.462695547533093e-05,
+      "loss": 0.1305,
+      "step": 4600
+    },
+    {
+      "epoch": 0.9427812045534326,
+      "grad_norm": 0.48744386434555054,
+      "learning_rate": 3.429268618799305e-05,
+      "loss": 0.1315,
+      "step": 4700
+    },
+    {
+      "epoch": 0.9628403791183993,
+      "grad_norm": 0.164585679769516,
+      "learning_rate": 3.395841690065517e-05,
+      "loss": 0.1307,
+      "step": 4800
+    },
+    {
+      "epoch": 0.9828995536833659,
+      "grad_norm": 0.2665926218032837,
+      "learning_rate": 3.362414761331729e-05,
+      "loss": 0.1295,
+      "step": 4900
+    },
+    {
+      "epoch": 1.0028082844390953,
+      "grad_norm": 0.16074395179748535,
+      "learning_rate": 3.328987832597941e-05,
+      "loss": 0.1285,
+      "step": 5000
+    },
+    {
+      "epoch": 1.0028082844390953,
+      "eval_loss": 0.1043267473578453,
+      "eval_runtime": 687.1542,
+      "eval_samples_per_second": 25.795,
+      "eval_steps_per_second": 6.45,
+      "step": 5000
+    },
+    {
+      "epoch": 1.022867459004062,
+      "grad_norm": 0.20015545189380646,
+      "learning_rate": 3.295560903864153e-05,
+      "loss": 0.1275,
+      "step": 5100
+    },
+    {
+      "epoch": 1.0429266335690286,
+      "grad_norm": 0.13576115667819977,
+      "learning_rate": 3.2621339751303654e-05,
+      "loss": 0.1269,
+      "step": 5200
+    },
+    {
+      "epoch": 1.0629858081339953,
+      "grad_norm": 0.12735939025878906,
+      "learning_rate": 3.2287070463965776e-05,
+      "loss": 0.1265,
+      "step": 5300
+    },
+    {
+      "epoch": 1.0830449826989619,
+      "grad_norm": 0.13942281901836395,
+      "learning_rate": 3.19528011766279e-05,
+      "loss": 0.1264,
+      "step": 5400
+    },
+    {
+      "epoch": 1.1031041572639286,
+      "grad_norm": 0.15262341499328613,
+      "learning_rate": 3.161853188929001e-05,
+      "loss": 0.1253,
+      "step": 5500
+    },
+    {
+      "epoch": 1.1231633318288952,
+      "grad_norm": 0.1289774477481842,
+      "learning_rate": 3.1284262601952133e-05,
+      "loss": 0.1252,
+      "step": 5600
+    },
+    {
+      "epoch": 1.143222506393862,
+      "grad_norm": 0.13571390509605408,
+      "learning_rate": 3.0949993314614255e-05,
+      "loss": 0.1239,
+      "step": 5700
+    },
+    {
+      "epoch": 1.1632816809588284,
+      "grad_norm": 0.13661526143550873,
+      "learning_rate": 3.0615724027276377e-05,
+      "loss": 0.125,
+      "step": 5800
+    },
+    {
+      "epoch": 1.1833408555237952,
+      "grad_norm": 0.20611687004566193,
+      "learning_rate": 3.0281454739938498e-05,
+      "loss": 0.1242,
+      "step": 5900
+    },
+    {
+      "epoch": 1.203400030088762,
+      "grad_norm": 0.12380609661340714,
+      "learning_rate": 2.994718545260062e-05,
+      "loss": 0.1231,
+      "step": 6000
+    },
+    {
+      "epoch": 1.203400030088762,
+      "eval_loss": 0.10073110461235046,
+      "eval_runtime": 687.1346,
+      "eval_samples_per_second": 25.796,
+      "eval_steps_per_second": 6.45,
+      "step": 6000
+    },
+    {
+      "epoch": 1.2234592046537285,
+      "grad_norm": 0.1408674120903015,
+      "learning_rate": 2.9612916165262738e-05,
+      "loss": 0.1245,
+      "step": 6100
+    },
+    {
+      "epoch": 1.2435183792186952,
+      "grad_norm": 0.14142949879169464,
+      "learning_rate": 2.927864687792486e-05,
+      "loss": 0.1218,
+      "step": 6200
+    },
+    {
+      "epoch": 1.2635775537836618,
+      "grad_norm": 0.13047949969768524,
+      "learning_rate": 2.894437759058698e-05,
+      "loss": 0.121,
+      "step": 6300
+    },
+    {
+      "epoch": 1.2836367283486285,
+      "grad_norm": 5.7425217628479,
+      "learning_rate": 2.86101083032491e-05,
+      "loss": 0.1207,
+      "step": 6400
+    },
+    {
+      "epoch": 1.303695902913595,
+      "grad_norm": 0.28052738308906555,
+      "learning_rate": 2.827583901591122e-05,
+      "loss": 0.12,
+      "step": 6500
+    },
+    {
+      "epoch": 1.3237550774785618,
+      "grad_norm": 0.13514311611652374,
+      "learning_rate": 2.7941569728573342e-05,
+      "loss": 0.1199,
+      "step": 6600
+    },
+    {
+      "epoch": 1.3438142520435283,
+      "grad_norm": 0.14283640682697296,
+      "learning_rate": 2.7607300441235463e-05,
+      "loss": 0.1197,
+      "step": 6700
+    },
+    {
+      "epoch": 1.363873426608495,
+      "grad_norm": 0.14133544266223907,
+      "learning_rate": 2.727303115389758e-05,
+      "loss": 0.1201,
+      "step": 6800
+    },
+    {
+      "epoch": 1.3839326011734618,
+      "grad_norm": 0.20595553517341614,
+      "learning_rate": 2.6938761866559703e-05,
+      "loss": 0.1213,
+      "step": 6900
+    },
+    {
+      "epoch": 1.4039917757384284,
+      "grad_norm": 0.133372962474823,
+      "learning_rate": 2.6604492579221825e-05,
+      "loss": 0.118,
+      "step": 7000
+    },
+    {
+      "epoch": 1.4039917757384284,
+      "eval_loss": 0.09819206595420837,
+      "eval_runtime": 687.1022,
+      "eval_samples_per_second": 25.797,
+      "eval_steps_per_second": 6.45,
+      "step": 7000
+    },
+    {
+      "epoch": 1.424050950303395,
+      "grad_norm": 0.21542303264141083,
+      "learning_rate": 2.6270223291883943e-05,
+      "loss": 0.1215,
+      "step": 7100
+    },
+    {
+      "epoch": 1.4441101248683617,
+      "grad_norm": 0.24275696277618408,
+      "learning_rate": 2.5935954004546064e-05,
+      "loss": 0.1187,
+      "step": 7200
+    },
+    {
+      "epoch": 1.4641692994333284,
+      "grad_norm": 0.14183971285820007,
+      "learning_rate": 2.5601684717208186e-05,
+      "loss": 0.1173,
+      "step": 7300
+    },
+    {
+      "epoch": 1.484228473998295,
+      "grad_norm": 0.11123021692037582,
+      "learning_rate": 2.5267415429870307e-05,
+      "loss": 0.1167,
+      "step": 7400
+    },
+    {
+      "epoch": 1.5042876485632615,
+      "grad_norm": 0.1129390299320221,
+      "learning_rate": 2.4933146142532426e-05,
+      "loss": 0.1196,
+      "step": 7500
+    },
+    {
+      "epoch": 1.5243468231282282,
+      "grad_norm": 0.1165638417005539,
+      "learning_rate": 2.4598876855194547e-05,
+      "loss": 0.1186,
+      "step": 7600
+    },
+    {
+      "epoch": 1.544405997693195,
+      "grad_norm": 0.3278215527534485,
+      "learning_rate": 2.426460756785667e-05,
+      "loss": 0.1161,
+      "step": 7700
+    },
+    {
+      "epoch": 1.5644651722581617,
+      "grad_norm": 0.2158752679824829,
+      "learning_rate": 2.3930338280518787e-05,
+      "loss": 0.1184,
+      "step": 7800
+    },
+    {
+      "epoch": 1.5845243468231283,
+      "grad_norm": 0.16784250736236572,
+      "learning_rate": 2.3596068993180908e-05,
+      "loss": 0.118,
+      "step": 7900
+    },
+    {
+      "epoch": 1.6045835213880948,
+      "grad_norm": 0.16192170977592468,
+      "learning_rate": 2.326179970584303e-05,
+      "loss": 0.1154,
+      "step": 8000
+    },
+    {
+      "epoch": 1.6045835213880948,
+      "eval_loss": 0.09730422496795654,
+      "eval_runtime": 687.1621,
+      "eval_samples_per_second": 25.794,
+      "eval_steps_per_second": 6.45,
+      "step": 8000
+    }
+  ],
+  "logging_steps": 100,
+  "max_steps": 14958,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 1000,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.5259044426479616e+18,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6f6822618a8e3b7538b9ad49376197658aa70f04f45dd2330d4aa34463a6e5fb
+size 5969