mmBERT: A Modern Multilingual Encoder with Annealed Language Learning
Paper • 2509.06888 • Published • 15
mmBERT: a modern multilingual encoder
mmBERT is trained on 3T tokens from over 1800 languages, showing SoTA scores on benchmarks and exceptional low-resource performance
Note ICML 2026
-
jhu-clsp/mmBERT-base
Fill-Mask • Updated • 530k • • 233
jhu-clsp/mmBERT-small
Fill-Mask • Updated • 47.3k • • 82Note Intermediate checkpoints for continued pre-training (MosiacML Composer format)
jhu-clsp/mmBERT-checkpoints
Updated • 4Note Pre-training Data
-
orionweller/mmBERT-pretrain-p1-fineweb2-langs
Updated • 255 • 7 -
orionweller/mmBERT-pretrain-p2-fineweb2-remaining
Updated • 430 -
orionweller/mmBERT-pretrain-p3-others
Updated • 1.81k -
jhu-clsp/mmBERT-midtraining-data
Updated • 7.08k • 1
jhu-clsp/mmBERT-decay-data
Updated • 18.8k • 6Note Randomized data for training (not recommended unless you are using the same data mix)
-
orionweller/mmBERT-data-decay-all
Updated • 198 -
orionweller/mmBERT-data-decay-cont
Updated • 130 -
orionweller/mmBERT-data-decay-eng
Updated • 244 -
orionweller/mmBERT-data-midtraining
Updated • 341 -
orionweller/mmBERT-pretraining-data-chunk0
Preview • Updated • 269 -
orionweller/mmBERT-pretraining-data-chunk1
Preview • Updated • 398 -
orionweller/mmBERT-pretraining-data-chunk2
Preview • Updated • 411 -
orionweller/mmBERT-pretraining-data-chunk3
Preview • Updated • 2.5k