# NeoAraBERT
NeoAraBERT is a state-of-the-art open-source Arabic text-embedding model built on the NeoBERT architecture. We pretrain NeoAraBERT on diverse open-source and internal datasets covering modern standard, classical, and dialectal Arabic. We guided our design choices with Arabic tailored ablation studies including text normalization, light stemming, and diacritics-aware tokenization handling. We also performed POS-aware token masking and learning-rate scheduling ablation studies. We benchmarked NeoAraBERT against five top-performing Arabic models on 23 tasks, including a synonym-based task, [Muradif](https://huggingface.co/datasets/U4RASD/Muradif), that directly assesses embedding quality with no additional fine-tuning. NeoAraBERT variants rank first in 18 tasks and improve average performance across the full benchmark suite.

This is the NeoAraBERT_Mix checkpoint, our best-performing checkpoint overall. This model was introduced at the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026). For more information, visit our website: https://www.acrps.ai/neoarabert.

### How to Use
Install these libraries:
```
pip install fast-disambig torch==2.5.1 transformers xformers==0.0.28.post3
```
Load the model and use it to generate embeddings:
```python
from transformers import AutoModel, AutoTokenizer

model_name = "U4RASD/NeoAraBERT"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)

# Tokenize input text
text = "المركز العربيّ للأبحاث ودراسة السياسات هو مؤسّسة بحثيّة فكريّة مستقلّة للعلوم الاجتماعية والإنسانية؛ النظرية والتطبيقية."
inputs = tokenizer(text, return_tensors="pt")

# Generate embeddings
outputs = model(**inputs)
embedding = outputs.last_hidden_state[:, 0, :]
print(embedding.shape)
```

### Citation
If you use the code, model, or the Muradif benchmark, please reference this work in your paper:
```bibtex
The citation will be added here soon.
```

### License
This model is licensed under the CC BY-SA 4.0 license. The text of the license can be found [here](https://creativecommons.org/licenses/by-sa/4.0/).