50M-Darija-MSA-v1 — Bidirectional Darija ↔ MSA

A 51.8M-parameter small language model that translates both ways between Moroccan Darija (الدارجة المغربية) and Modern Standard Arabic (الفصحى). A single set of weights serves both directions; a direction-specific Arabic system prompt selects which way to translate.

Finetuned from oddadmix/50M-2048-Emhotob, a tiny Arabic base model trained from scratch.

Evaluation

Evaluated on a deterministic held-out set of ~3,000 pairs (seed=42; 2,961 scored after dropping rows with an empty MSA reference), decoded greedily (do_sample=False, no repetition penalty), scored with sacreBLEU:

Direction	sacreBLEU	chrF
Darija → MSA	31.73	51.02
MSA → Darija	39.88	50.79

MSA→Darija scores higher on BLEU, but both directions are genuinely hard: Darija has no single standardized orthography, is heavily borrowed/code-switched, and appears in the data in both Arabic script and Latin (Arabizi) script. The saved weights are the best checkpoint by validation loss (eval_loss=1.234, epoch 2 of 3).

Decoding note: use plain greedy. A repetition penalty (1.2) was tested across these 50M translation models and lowered BLEU by 5–12 points.

Example translations

Real greedy-decoded outputs from the held-out set:

Darija → MSA

Darija input	Model output (MSA)
لا، عندنا تذاكر يا حبيبة	لا، لدينا تذاكر يا حبيبتي.
نتحداك تعتارد على هاد الموضوع	أنا أتحداك أن تعترض على هذا الموضوع.
walakin hadi awal tsafira ftyyara lia o ana khayfa chwiya	لكن هذه هي أول رحلة لي إلى أستراليا، أنا خائف قليلاً.

(The last row shows the model handling Arabizi — Latin-script Darija — input.)

MSA → Darija

MSA input	Model output (Darija)
لا، لدينا تذاكر يا حبيبتي.	لا، عندنا تاداكير أحبي
لقد غطى وجهه وبكى.	راه غطّي وجهو و بقا
أنا أنتظر منك أن تستحق هذا.	كانتسنّا فيك ت ستاهل هادشي

A larger set of 20 examples per direction (with references) is in eval_bidirectional.json.

Usage

ChatML format. Pick the system prompt for the direction you want:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "oddadmix/50M-Darija-MSA-v1"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

SYS_TO_MSA = "أنت مترجم محترف. ترجم النص من الدارجة المغربية إلى اللغة العربية الفصحى."
SYS_TO_DAR = "أنت مترجم محترف. ترجم النص من اللغة العربية الفصحى إلى الدارجة المغربية."

def translate(text: str, system: str) -> str:
    prompt = (
        f"<|im_start|>system\n{system}<|im_end|>\n"
        f"<|im_start|>user\n{text.strip()}<|im_end|>\n"
        f"<|im_start|>assistant\n"
    )
    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
    if tok.bos_token_id is not None:  # training prepends BOS
        bos = torch.tensor([[tok.bos_token_id]], device=model.device)
        ids["input_ids"] = torch.cat([bos, ids["input_ids"]], dim=1)
        ids["attention_mask"] = torch.cat([torch.ones_like(bos), ids["attention_mask"]], dim=1)
    out = model.generate(**ids, max_new_tokens=256, do_sample=False,
                         eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)
    return tok.decode(out[0, ids["input_ids"].size(1):], skip_special_tokens=True).strip()

print(translate("نتحداك تعتارد على هاد الموضوع", SYS_TO_MSA))
# → أنا أتحداك أن تعترض على هذا الموضوع.
print(translate("لقد غطى وجهه وبكى.", SYS_TO_DAR))
# → راه غطّي وجهو و بقا

Training

• Base model: oddadmix/50M-2048-Emhotob (Llama arch, ~51.8M params)
• Dataset: oddadmix/darija_english_msa_parallel_dataset (90,104 rows; this model uses the darija and msa columns; rows with an empty msa are skipped). Sources: DoDA, HANTIFARAH combined, and ArabML/Skiredj parallel data.
• Method: HuggingFace Trainer, ChatML, prompt-masked cross-entropy (loss only on the assistant turn). Each row is exploded into two training examples (one per direction, ~170.6K total). Two ChatML special tokens (<|im_start|>, <|im_end|>) were added and embeddings resized.
• Hyperparameters: 3 epochs · effective batch 64 · LR 3e-4 (cosine, 5% warmup) · bf16 · max length 1024 · load_best_model_at_end on eval_loss.
• Split: 87,104 train / 3,000 deterministic held-out (seed=42), scored both directions.

Limitations

• A 50M model: expect errors on rare / technical vocabulary, proper nouns, and long or noisy inputs. Everyday conversational text is handled best.
• Darija has no standard orthography and mixes Arabic and Latin (Arabizi) script plus French/Amazigh loanwords — outputs may vary in spelling and occasionally hallucinate on out-of-domain input. The training data also contains some crawled/wiki-formatting artifacts.
• Gender is disambiguated only from context; ambiguous inputs may default one way.
• For Egyptian dialect or English pairs, see the sibling models oddadmix/50M-Darija-English-v1, oddadmix/50M-English-MSA-v1, oddadmix/50M-MSA-Egyptian-v1.

License

Apache-2.0 (model weights, inherited from the base model). Note the training dataset aggregates several community corpora — check their individual licenses for downstream use.