🇸🇩 Sudanizer-72k: The Foundation for Sudanese Arabic NLP

Welcome to Sudanizer-72k, the first purpose-built Byte-Pair Encoding (BPE) tokenizer engineered exclusively for the Sudanese Arabic dialect.

If you've ever tried passing Sudanese colloquial text through models like GPT-4o, AraBERT, or Qwen, you've likely noticed severe subword fragmentation. Generalist tokenizers chop our dialect into tiny, meaningless pieces, inflating token counts and eating up your context window. Sudanizer fixes this. By treating Sudanese Arabic as a first-class language, we achieve up to a 64.6% improvement in compression over existing multilingual baselines.

🌍 Developed by Sudaverse

Sudanizer is a core infrastructural component developed by the Sudaverse team (www.sudaverse.com), an open-source applied AI initiative. Our mission is to build a comprehensive ecosystem—providing shared corpora, custom tokenizers, and model evaluation layers—dedicated to the digital preservation of the Sudanese dialect and the development of highly localized, applied AI solutions.

We believe that native dialect processing shouldn't be an afterthought. Sudanizer is our first step in giving Sudanese NLP researchers, developers, and creators the native tooling they need to build the next generation of dialect-aware applications.

📊 Comprehensive Performance Benchmarks

We rigorously evaluated Sudanizer against nine leading baselines on a strictly held-out test set of 31,199 Sudanese sentences (6,523,776 characters).

Main Tokenizer Comparison

Sudanizer outperforms both generalist Arabic tokenizers and frontier multilingual LLM tokenizers across all metrics, ensuring that roughly 87% of Sudanese words are emitted as single tokens.

Tokenizer	Vocab Size	Characters / Token (95% CI) 📈	Compression Ratio 📉	Fertility (Tokens/Word) 📉	UNK %
Sudanizer-72k (Ours)	72,000	4.511 [4.503, 4.518]	0.2217	1.148	0.0001
Aranizer-SP-86k	86,000	3.736 [3.725, 3.748]	0.2676	1.386	0.000
CAMeLBERT-Mix	30,000	3.624 [3.615, 3.631]	0.2760	1.429	0.014
Aranizer-SP-64k	64,000	3.607 [3.595, 3.619]	0.2772	1.436	0.000
Aranizer-PBE-86k	86,000	3.466 [3.455, 3.479]	0.2885	1.494	0.000
Aranizer-PBE-64k	64,000	3.319 [3.308, 3.331]	0.3013	1.561	0.000
AraBERTv2	64,000	3.181 [3.177, 3.186]	0.3143	1.628	0.244
XLM-RoBERTa	250,002	2.869 [2.863, 2.875]	0.3485	1.805	0.009
GPT-4o (o200k)	200,019	2.741 [2.735, 2.746]	0.3649	1.890	0.000*
Qwen2.5	151,643	2.329 [2.324, 2.334]	0.4293	2.224	0.000*

*Zero by construction due to byte-level fallback.[cite: 2]

Vocabulary-Size Ablation Study

To prove our gains come from true dialect specialization rather than simply having a larger vocabulary, we ablated Sudanizer at smaller sizes[cite: 2]. Even our 32k variant outperforms the strongest 86k and 200k baselines[cite: 2]:

Variant	Vocab Size	Characters / Token 📈	Fertility 📉	Improvement vs. Best Baseline
Sudanizer-72k	72,000	4.511	1.148	+20.7%
Sudanizer-64k	64,000	4.467	1.159	+19.6%
Sudanizer-32k	32,000	4.173	1.241	+11.7%

(Note: In a 4,096-token context window, an LLM using Sudanizer-72k can ingest approximately 18,480 characters of Sudanese text, compared to just 11,230 characters with GPT-4o's o200k tokenizer).

💡 Under the Hood: Built for Dialect Fidelity

Sudanizer doesn't just compress well; it respects the language[cite: 2].

• Lossless Orthography: Standard byte-level normalizers often destroy the crucial distinction between the final-position ya (ي) and alef maqsura (ى). Sudanizer uses a character-level Metaspace pre-tokenizer that preserves these dialect-bearing lexical distinctions perfectly.
• Preserved Suffixes: We deliberately avoided rewriting dialectal variants (e.g., forcing كده to become كدا), allowing downstream models to learn rich, sub-regional signals naturally.
• Rigorous Data Hygiene: Trained on a massive 24.4-million-word corpus (126.6M characters) of Sudanese social media, storytelling, Q&A forums, and YouTube transcripts via the Sudata platform. Evaluation splits were strictly segregated using exact-line deduplication and MinHash LSH near-duplicate detection.

💻 Quick Start

Loading Sudanizer is plug-and-play with the HuggingFace transformers library:

from transformers import PreTrainedTokenizerFast

# 1. Load the Sudaverse tokenizer
tokenizer = PreTrainedTokenizerFast.from_pretrained("O96a/sudanizer-72k")

# 2. Try it on some real Sudanese text
text = "البنات دايرين يمشوا المدرسة بكرة الصباح"
encoded = tokenizer.encode(text)
decoded = tokenizer.decode(encoded)

print(f"Tokens: {encoded}")
# Notice how 'دايرين' stays intact as a single token!
print(f"Decoded: {decoded}")


📚 Paper & Authors
This model is the official artifact for the paper: "Sudanizer: A Dialect-Specific Byte-Pair Encoding Tokenizer for Sudanese Arabic" (Accepted at FITAT 2026).


📂 Need the Raw Training Data?
To respect user privacy and platform Terms of Service, the raw training corpus is not hosted publicly in this repository.
If you are a researcher or developer requiring access to the raw corpus for downstream model training or reproducibility within the Sudaverse ecosystem, please submit a request, detailing your affiliation and intended use case.