| --- |
| language: |
| - fr |
| - de |
| - en |
| - lb |
| base_model: jhu-clsp/mmBERT-base |
| library_name: transformers |
| license: mit |
| tags: |
| - modernbert |
| - mmbert |
| - masked-language-modeling |
| - impresso |
| --- |
| |
| # multilingualmodernimpressoBERT |
|
|
| Source card for the future Hugging Face model repository: |
|
|
| ```text |
| impresso-project/mmbert-multilingual-impresso-continued-mlm |
| ``` |
|
|
| This model is a continued-MLM adaptation of `jhu-clsp/mmBERT-base` on multilingual Impresso newspaper text. |
|
|
| The checkpoint is intended as a domain-adapted base model for downstream media-agency token classification. |
|
|
| ## License |
|
|
| This continued-pretraining checkpoint is released under the MIT license, matching the license of the base model `jhu-clsp/mmBERT-base`. |
|
|
| The model is derived from `jhu-clsp/mmBERT-base`; users should cite and comply with the base model terms. The continued-MLM training corpus consists of Impresso newspaper text samples used for domain adaptation. The model weights are published separately from the source text; users remain responsible for checking whether their downstream use of Impresso-derived models and outputs is compatible with their institutional, corpus, and application-specific requirements. |
|
|
| ## Continued MLM Run |
|
|
| The first completed workbench run used: |
|
|
| - source compiled Impresso files: `fr`, `de`, `en`, `lb` |
| - source filtering: OCR quality at least `0.90`, minimum text length `200` characters |
| - source sampling policy: up to 300,000 texts per language, with smaller languages exhausted |
| - sampled corpus: 872,889 train rows and 8,818 validation rows after split |
| - training subset: 100,000 train rows |
| - validation subset: 2,000 validation rows |
| - base model: `jhu-clsp/mmBERT-base` |
| - objective: masked language modeling |
| - MLM probability: `0.15` |
| - max sequence length: `256` |
| - padding: fixed max-length padding |
| - epochs: `1` |
| - per-device train batch size: `1` |
| - gradient accumulation steps: `8` |
| - effective train batch size per device: `8` |
| - gradient checkpointing: enabled |
| - learning rate: `2e-5` |
| - weight decay: `0.01` |
| - warmup: 750 steps, computed as 6 percent of the capped optimizer steps |
| - intermediate checkpoint saving: disabled; final model only |
| - random seed: `42` |
|
|
| Training was run locally on Apple Silicon using the PyTorch MPS backend. The completed run took about 8 hours and 6 minutes. Observed memory use stayed below roughly 30 GB. |
|
|
| ## Metrics |
|
|
| Final run metrics: |
|
|
| | metric | value | |
| | --- | ---: | |
| | train loss | 1.7775 | |
| | eval loss | 1.7172 | |
| | train runtime | 29,134 seconds | |
| | train samples / second | 3.432 | |
| | train steps / second | 0.429 | |
| | eval samples / second | 14.885 | |
|
|
| The validation loss stayed close to the train loss and the run completed without divergence, so this checkpoint is suitable for downstream comparison against the original `jhu-clsp/mmBERT-base`. |
|
|
