README.md

---
language:
- ar
license: apache-2.0
tags:
- bert
- masked-language-modeling
- arabic
- social-media
- fill-mask
- pilot
datasets:
- thejosango/nuha-dataset
---

# nuha-jo-mlm

## Model Summary

`nuha-jo-mlm` is an Arabic BERT model pre-trained from scratch on Jordanian social media text using masked language modelling (MLM). It serves as the domain-adapted base for the NUHA classifier family — [`nuha-jo-binary`](https://huggingface.co/thejosango/nuha-jo-binary) and [`nuha-jo-trinary-full`](https://huggingface.co/thejosango/nuha-jo-trinary-full) — and was developed as part of a **pilot proof-of-concept** for the NUHA project by the [Jordan Open Source Association (JOSA)](https://josa.ngo).

Starting from a fresh vocabulary trained on the NUHA corpus rather than fine-tuning an existing Arabic BERT, the model is adapted to the vocabulary and linguistic patterns of colloquial Jordanian Arabic social media text.

## Uses

### Direct Use

`nuha-jo-mlm` can be used for masked token prediction on Arabic social media text, or as a base model for fine-tuning downstream Arabic NLP tasks — particularly those involving informal Jordanian or Levantine Arabic.

```python
from transformers import pipeline

fill_mask = pipeline("fill-mask", model="thejosango/nuha-jo-mlm")
results = fill_mask("هذه المرأة [MASK] جداً")
for r in results:
    print(r["token_str"], r["score"])
```

### Downstream Use

The primary intended downstream use is fine-tuning for hate speech and gender-based violence detection using the NUHA dataset, as demonstrated by [`nuha-jo-binary`](https://huggingface.co/thejosango/nuha-jo-binary) and [`nuha-jo-trinary-full`](https://huggingface.co/thejosango/nuha-jo-trinary-full).

### Out-of-Scope Use

This model was pre-trained on a relatively small, domain-specific corpus of Jordanian social media comments. It is not suitable as a general-purpose Arabic language model and should not be used as a replacement for models trained on broader Arabic corpora (e.g. for MSA tasks, translation, or summarisation).

## Bias, Risks, and Limitations

- **Corpus bias**: The pre-training corpus consists entirely of social media comments, many of which contain offensive or harmful language. The model may reflect biases present in that content.
- **Dialect coverage**: The corpus is primarily Jordanian Arabic. Performance on other Arabic dialects or Modern Standard Arabic is not guaranteed.
- **Pilot scale**: As part of an initial proof-of-concept effort, the pre-training corpus size and training duration are modest. A larger corpus and longer training would likely yield a stronger base model.
- **Vocabulary size**: The tokenizer vocabulary (17,513 tokens) was trained specifically on this corpus and is much smaller than general Arabic BERT vocabularies.

## Training Details

### Training Data

Pre-trained on the text portion of the [`thejosango/nuha-dataset`](https://huggingface.co/datasets/thejosango/nuha-dataset) — a corpus of Arabic social media comments collected from Jordanian platforms. The `binary` dataset split was used for pre-training (text only, labels ignored).

### Training Procedure

- **Architecture**: BERT (`BertForMaskedLM`), 12 layers, 12 attention heads, 768 hidden size
- **Vocabulary**: Custom BPE tokenizer trained from scratch on the corpus (17,513 tokens), with `[URL]` as an added special token
- **MLM probability**: 15%
- **Optimizer**: Adam (β₁=0.9, β₂=0.999, ε=1e-8)
- **Learning rate**: 1e-4 with linear schedule, 5,000 warmup steps
- **Batch size**: 128
- **Epochs**: 25
- **Framework**: Transformers 4.32.1, PyTorch 2.0.1

### Training Results

| Epoch | Validation Loss |
|---|---|
| 1 | 8.435 |
| 5 | 7.296 |
| 10 | 6.499 |
| 15 | 6.118 |
| 20 | 5.863 |
| 25 | 5.788 |

---

*This model was developed as part of an initial pilot study. It is intended as a stepping stone for the downstream NUHA classifiers rather than as a general-purpose Arabic language model.*