README.md

---
library_name: transformers
language:
- bm
base_model:
- FacebookAI/xlm-roberta-base
pipeline_tag: fill-mask
tags:
- xlm-roberta
- fill-mask
- masked-lm
- bambara
- bamanankan
- mali
---

# bm-xlm-roberta-base

XLM-RoBERTa base adapted to Bambara (Bamanankan, `bm`): continued masked-language-model
pretraining with a vocabulary-extended tokenizer that adds Bambara words and the characters
`Ɛ Ɔ Ɲ Ŋ`.

This is a masked-LM checkpoint, not a task model. It predicts `<mask>` and produces
contextual representations; use it as the initialisation for fine-tuning on classification,
token tagging, QA, or as a retrieval encoder.

## Usage

The mask token is `<mask>`, not `[MASK]`.

```python
from transformers import pipeline

fill = pipeline("fill-mask", model="djelia/bm-xlm-roberta-base")

for p in fill("Mɔgɔw bɛɛ bɛ bange hɔrɔnya <mask> danbe la."):
    print(f"{p['score']:.4f}  {p['token_str']!r}")
```

Always load the tokenizer from this repo — token ids are not interchangeable with stock
XLM-R.

## Architecture

| | |
| --- | --- |
| Class | `XLMRobertaForMaskedLM` (encoder-only) |
| Parameters | 280,924,397 (`F32`) |
| Layers / hidden / heads / FFN | 12 / 768 / 12 / 3072 |
| Max sequence length | 512 |
| `vocab_size` | 253,421 |
| Special tokens | `<s>`=0, `<pad>`=1, `</s>`=2, `<unk>`=3, `<mask>`=253420 |

## Notes

The checkpoint carries no pooler weights, so `pooler_output` from `AutoModel` is randomly
initialised. Take `last_hidden_state` and pool it yourself:

```python
from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("djelia/bm-xlm-roberta-base")
encoder = AutoModel.from_pretrained("djelia/bm-xlm-roberta-base").eval()

hidden = encoder(**tokenizer(["Aw ni ce."], return_tensors="pt")).last_hidden_state
```

Mean-pooled token vectors are not sentence embeddings — no contrastive objective was applied
here. Fine-tune with a sentence-similarity objective if you need retrieval.