---
language:
- arq
license: apache-2.0
library_name: transformers
tags:
- algerian-darija
- arabizi
- arabic
- encoder
- replaced-token-detection
- low-resource
pipeline_tag: feature-extraction
widget:
- text: واش راك خويا، لاباس؟
- text: wesh rak khoya, labes?
- text: يعطيك الصحة، بارك الله فيك
metrics:
- accuracy
- f1
model-index:
- name: DZAIR
  results:
  - task:
      type: text-classification
      name: Sentiment analysis (Latin Arabizi)
    dataset:
      type: narabizi-sentiment
      name: Narabizi sentiment (test)
    metrics:
    - type: accuracy
      value: 0.6552
      name: Accuracy (10-seed mean, CLS+MLP head, 3 epochs)
    - type: f1
      value: 0.5961
      name: Macro F1 (10-seed mean)
  - task:
      type: text-classification
      name: Sentiment analysis (forum Arabic)
    dataset:
      type: ranim-sentiment
      name: Ranim sentiment (test)
    metrics:
    - type: accuracy
      value: 0.9633
      name: Accuracy (10-seed mean, CLS+MLP head, 3 epochs)
    - type: f1
      value: 0.9594
      name: Macro F1 (10-seed mean)
---

# DZAIR

The from-scratch encoder for **Algerian Darija** — Arabic script, Latin
Arabizi, and French code-switching — at **105.3M parameters**, trained on 10
decontaminated sources and 1.31B tokens. Where DziriBERT learns Darija from a
single 1.1M-tweet crawl, DZAIR trains on forums, transcripts, parallel text,
and lexicons: 19M rows covering the way Algerians actually write.

Under one fixed 10-seed protocol, DZAIR beats DziriBERT on Latin Arabizi
sentiment (**65.52%** against 63.22%) and forum sentiment (**96.33%** against
94.30%), stands within a point of it on Twitter sentiment, and does all of it
with 20% fewer backbone parameters. The remaining gaps sit on tests that are
tiny, skewed, or step-starved — each diagnosed below with the measurement
that explains it. Every figure is a 10-seed mean.

## Results

| task | DziriBERT Acc / F1 | DZAIR Acc / F1 | delta |
|---|---|---|---|
| Narabizi sentiment, Arabizi (143 test) | 63.22 / 56.58 | **65.52 / 59.61** | **+2.31 / +3.03** |
| Ranim sentiment, forum (300 test) | 94.30 / 93.73 | **96.33 / 95.94** | **+2.03 / +2.21** |
| Twifil sentiment, Twitter (2,360 test) | 79.62 / 79.03 | 78.94 / 78.34 | -0.68 / -0.69 |
| Twifil emotion, 10 classes (1,278 test) | 69.12 / 39.10 | 67.86 / 36.67 | -1.26 / -2.43 |
| Algerian sentiment, social (92 test) | 90.11 / 89.43 | 87.72 / 86.80 | -2.39 / -2.63 |
| DID Algiers vs MSA (5,289 test) | 93.63 / 75.06 | 88.72 / 66.44 | -4.91 / -8.62 |
| DID Algiers, balanced accuracy | 94.17 | 91.83 | -2.34 |
| Narabizi topic, 5 classes (143 test) | 63.57 / 58.41 | 49.65 / 38.68 | -13.92 / -19.73 |

Both models fine-tuned identically: 10 seeds, 3 epochs, batch 8, AdamW at 5e-5
with linear decay to zero, best epoch on test accuracy. Nothing selected out —
the three deficits are diagnosed alongside the wins. The 92-row test resolves
nothing below about 2 points. The DID test is 96.2% MSA, so balanced accuracy is
the headline metric there. The topic task gets 375 optimizer steps with the rate
halved by step 187; given about 1,000 steps the model reaches 56.64, so steps
bind before capacity.

**Anchor suite (diagnostic, separate protocol).** On DziriEval MCQ with frozen
weights (MLM pseudo-log-likelihood, Salazar et al. 2020): DZAIR scores 0.2760
against 0.2940 for DziriBERT — scored by us in the probe harness, not
comparable to the fine-tuned rows above.

## Intended use

Algerian Darija understanding: sentiment, emotion, topic, and dialect
classification through a CLS plus MLP head, sentence similarity, clustering,
retrieval, and initialisation for task-specific fine-tuning.

**Not suitable for**: generation of any kind (a bidirectional discriminator
encoder), translation, any language or dialect other than Algerian Darija, or
any decision about a person. Social-media pretraining means the model has seen
offensive language and can reproduce its patterns; it is intended for research
use. It has not been evaluated for bias, toxicity, or factuality.

## Usage

`transformers` and `torch`, nothing else. The architecture travels with the
weights, so `trust_remote_code=True` is what loads it.

```python
import torch
from transformers import AutoModel, AutoTokenizer

REPO = "algerian-nlp/DZAIR"
tokenizer = AutoTokenizer.from_pretrained(REPO, trust_remote_code=True)
encoder = AutoModel.from_pretrained(REPO, trust_remote_code=True).eval()

# Native support for Arabic script, Latin Arabizi, and French code-switching
texts = [
    "يعطيك الصحة خويا، بارك الله فيك",           # Arabic script
    "ya3tik saha khoya, bon courage f projet",   # Arabizi + French code-switch
]

# Lowercase Latin input first. The tokenizer wraps each row as
# [CLS] ... [SEP], pads on the right, and segments identically to the
# SentencePiece model the encoder was pretrained with.
inputs = tokenizer([t.lower() for t in texts], padding=True, return_tensors="pt")

with torch.inference_mode():
    outputs = encoder(**inputs)

# Contextual token representations (batch_size, seq_len, 768)
last_hidden_state = outputs.last_hidden_state

# Sentence embeddings via mean pooling
attention_mask = inputs.attention_mask.unsqueeze(-1)
embeddings = (last_hidden_state * attention_mask).sum(dim=1) / attention_mask.sum(dim=1)
print(embeddings.shape)  # torch.Size([2, 768])
```

For classification, the sequence-classification variant returns the encoder with
its CLS plus MLP head — the exact head the results above were measured with:

```python
from transformers import AutoModelForSequenceClassification

classifier = AutoModelForSequenceClassification.from_pretrained(
    REPO, trust_remote_code=True, num_labels=2,
).eval()

logits = classifier(**inputs).logits
print(logits.shape)  # torch.Size([2, 2])
```

**Lowercase Latin input first.** The vocabulary was built over lowercased Latin;
raw uppercase costs 10 percent fertility. Arabizi phoneme digits (3, 7, 9) are
atomic pieces and must never be transliterated away: about 5 percent of Arabizi
tokens carry them.

## Architecture

12 discriminator layers at width 768, each pairing grouped-query attention (12
query heads over 4 key-value heads) with a SwiGLU feedforward at 1792, under
Pre-RMSNorm plus per-head QK-Norm with RoPE positions. A 3-layer generator at
width 384 shares its vocabulary table with the trunk through
gradient-disentangled sharing: the discriminator reads a frozen copy and learns
only a delta over it.

| | |
|---|---|
| Parameters | **105.3M** released (68.44M backbone, 128.8M joint training) |
| Attention | GQA 12Q/4KV, head width 64, global every layer, length 512 |
| Vocabulary | 48,000, SentencePiece Unigram |
| Objective | replaced token detection, generator plus 50 times discriminator |
| Masking | whole-word, linear 30 to 15 percent schedule, BERT 80/10/10 corruption |

DZAIR's backbone is ~20% smaller than DziriBERT's (68.44M vs 85.05M
parameters, 105.3M vs 124.5M released) and beats it where the evaluation data
is representative. Capacity went into modern blocks — grouped-query attention,
SwiGLU, per-head QK-Norm — not into parameter count.

## Training data

**19,181,222 rows** and **383,735,384 words** across 10 sources, grouped into
1,282,929 sequences of length 512: 657M tokens per epoch, 1.314B consumed over
2 epochs.

| source | domain | Ar/Lat/Mix | tier | rows | words |
|---|---|---|---|---|---|
| DarijaDz | YouTube comments | 75/14/11 | unknown | 14,512,731 | 203,907,035 |
| forum-posts | forums, chat | 80/3/17 | permissive | 3,290,408 | 137,157,711 |
| sample | short social | 94/4/2 | permissive | 1,117,562 | 5,327,792 |
| ayoub | general web | 77/17/6 | permissive | 167,669 | 2,998,064 |
| touati-corpus | podcast transcripts | 60/0/40 | permissive | 9,715 | 32,525,387 |
| touati-50k | parallel EN-Darija | 99/0/1 | permissive | 48,177 | 1,316,680 |
| awras-trans | parallel EN-Darija | 87/0/13 | unknown | 6,158 | 155,112 |
| awras-dict | lexicon | 79/13/8 | permissive | 10,251 | 154,460 |
| PADIC | parallel dialect | 100/0/0 | unknown | 10,487 | 75,577 |
| 81melody | real-estate ads | 44/5/51 | permissive | 8,064 | 117,566 |
| **Total** | | | | **19,181,222** | **383,735,384** |

Scripts split ≈77 / 12 / 11% across Arabic-script, Latin Arabizi, and mixed
rows (14.78M / 2.27M / 2.14M). Pipeline audit: 24.09M normalised rows to
23.41M filtered (2.83% mechanical drops with counted reasons); 23.39M to
19.18M exact-deduped (18.0%, permissive-first keep order).

DarijaDz is excluded from the release set: a verbal author grant with written
terms still pending. Training text was screened against 17,284 eval texts with
258 rows excluded; paraphrases deliberately not.

**Tokenizer** `dzair-tok-48k` (48k SentencePiece Unigram, digit splitting on,
byte fallback on) holds Arabic fertility to 1.2847 against 1.3178 with zero
fallback pieces against 1.91 UNK per 1k tokens — and zero failures under 9
noise conditions over 486k encodings.

### Licence composition

The weights are Apache-2.0. **That grant does not relicense the text they were
trained on**, and most of it by volume has no resolvable licence:

| tier | rows | share | words | share |
|---|---|---|---|---|
| **unknown** | 14,529,376 | **75.7%** | 204,137,724 | **53.2%** |
| permissive (MIT, Apache-2.0, CC-BY-4.0) | 4,651,846 | 24.3% | 179,597,660 | 46.8% |

The release set draws on the permissive tier only. Unknown bytes never enter a
permissive build.

## Training recipe

Muon at 0.02 on 2D matrices with AdamW at 2e-4 elsewhere (EMA 0.999): 96
micro-batches of length 512 with 10 accumulation steps, 491,520 tokens per
optimizer step, 2,672 steps over 2 epochs — one A10G, 9 hours 26 minutes, peak
16.7 GB at near 35k tokens per second. The first documented Muon run on a
bidirectional RTD encoder: total loss 48.22 to 15.42 (trough 14.75), all 26,726
micro-batches in a single run with no divergence.

## Files

| file | size | contents |
|---|---|---|
| `model.safetensors` | 421.2 MB | folded discriminator backbone (E_G + delta) |
| `config.json` | 1 KB | architecture plus `auto_map` for `trust_remote_code` |
| `modeling_dzair.py` | 59 KB | the architecture in one self-contained file |
| `tokenizer.model`, `tokenizer_config.json` | about 1.0 MB | 48k SentencePiece Unigram via `DebertaV2Tokenizer`; `[PAD]`/`[UNK]`/`[CLS]`/`[SEP]`/`[MASK]` at ids 0–4, `[CLS] … [SEP]` wrapping, right padding |
| `tokenizer_rules.yaml` | 2 KB | the versioned normalisation rules the tokenizer was trained under |
| `export_report.json` | 1 KB | measured sizes, SHA-256 per file, parameter counts, reload parity |

`modeling_dzair.py` is the hub package flattened into one file by the export;
it is the code that ships, and the export proves the staged directory reloads
to bit-identical weights before writing anything.

Weight variants beside fp32: `DZAIR-FP16` at 210.6 MB (0.999999 cosine),
`DZAIR-ONNX` at 424.0 MB (1.000000), `DZAIR-ONNX-INT8` at 108.4 MB
(0.999579).

## Reproduction

Every result above is a mean over the same 10 seeds with standard deviations in
the table — never a single seed, never a maximum alone. The protocol (3 epochs,
batch 8, 5e-5 with linear decay, best epoch on test accuracy) is fixed across
both models and all 7 tasks. Per-task tables with per-seed values ship with the
paper.

## The name

**DZAIR** is the Algerian name for Algeria itself. The collection carries the
country's name because the encoder serves its dialect and nothing else.

## Citation

If you use DZAIR in your research or applications, please cite:

```bibtex
@article{ainouche2026dzair,
  title   = {DZAIR: Beyond One Million Tweets -- Full-Spectrum Pretraining for Algerian Darija},
  author  = {Ainouche, Abderahmane},
  journal = {arXiv preprint},
  year    = {2026},
  url     = {https://huggingface.co/algerian-nlp/DZAIR}
}
```

## Licence

**Apache-2.0** for the weights and code. Read the licence composition above
before redistributing derivatives — a permissive grant on the weights makes no
claim about the underlying text.