library_name: transformers
language:
- bm
base_model:
- FacebookAI/xlm-roberta-base
pipeline_tag: fill-mask
tags:
- xlm-roberta
- fill-mask
- masked-lm
- bambara
- bamanankan
- mali
bm-xlm-roberta-base
XLM-RoBERTa base adapted to Bambara (Bamanankan, bm): continued masked-language-model
pretraining with a vocabulary-extended tokenizer that adds Bambara words and the characters
Ɛ Ɔ Ɲ Ŋ.
This is a masked-LM checkpoint, not a task model. It predicts <mask> and produces
contextual representations; use it as the initialisation for fine-tuning on classification,
token tagging, QA, or as a retrieval encoder.
Usage
The mask token is <mask>, not [MASK].
from transformers import pipeline
fill = pipeline("fill-mask", model="djelia/bm-xlm-roberta-base")
for p in fill("Mɔgɔw bɛɛ bɛ bange hɔrɔnya <mask> danbe la."):
print(f"{p['score']:.4f} {p['token_str']!r}")
Always load the tokenizer from this repo — token ids are not interchangeable with stock XLM-R.
Architecture
| Class | XLMRobertaForMaskedLM (encoder-only) |
| Parameters | 280,924,397 (F32) |
| Layers / hidden / heads / FFN | 12 / 768 / 12 / 3072 |
| Max sequence length | 512 |
vocab_size |
253,421 |
| Special tokens | <s>=0, <pad>=1, </s>=2, <unk>=3, <mask>=253420 |
Notes
The checkpoint carries no pooler weights, so pooler_output from AutoModel is randomly
initialised. Take last_hidden_state and pool it yourself:
from transformers import AutoModel, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("djelia/bm-xlm-roberta-base")
encoder = AutoModel.from_pretrained("djelia/bm-xlm-roberta-base").eval()
hidden = encoder(**tokenizer(["Aw ni ce."], return_tensors="pt")).last_hidden_state
Mean-pooled token vectors are not sentence embeddings — no contrastive objective was applied here. Fine-tune with a sentence-similarity objective if you need retrieval.