Emhotob-25M-Egyptian-English-v2 — Bidirectional Egyptian Arabic ↔ English (~25.3M params)

A 25.3M-parameter model that translates both ways between Egyptian colloquial Arabic (المصرية العامية) and English. A single set of weights serves both directions; a direction-specific system prompt selects which way to translate.

Finetuned from oddadmix/Emhotob-25M-v2, a tiny Llama-architecture base (hidden 384, 8 layers, 6 heads, vocab 32000, tied embeddings).

Scaling study. This is one rung of a from-scratch Arabic scaling study that runs an identical SFT + eval recipe across bases from 0.5M to 50M parameters to locate where translation emerges. On the headline MSA↔Egyptian pair, output is degenerate at ≤1M, becomes real-but-rough at 5M, and usable at 10M+. See the sibling oddadmix/50M-Egyptian-English-v1 for the fluent reference.

Evaluation

Deterministic held-out set of 3,000 pairs (seed=42), decoded greedily (do_sample=False, no repetition penalty), scored with sacreBLEU:

Direction	sacreBLEU	chrF
English → Egyptian	19.72	47.47
Egyptian → English	29.12	48.69

Saved weights are the best checkpoint by validation loss (eval_loss = 1.194). 20 samples per direction with references are in eval_bidirectional.json.

Example translations

Real greedy-decoded outputs from the held-out set:

English → Egyptian

Source	Model output	Reference
So what should I get you? Should I get you this or that? Just imagine the scene: sitting now, wearing a sleeveless under	طب أجيبلك إيه؟ أجيبلك ده ولا ده؟ تخيل المشهد: قعد دلوقتي لابس تيشيرل لابس تيشيرل لابس أحلى حتة لحمة هتحس بيها في حياتك.	طب أجيب لك إيه؟ أجيب لك دي ولا دي؟ تخيل كده المنظر، قاعد دلوقتي لابس فانلة بحمالات وبتاكل أحلى حتة لحمة ممكن تكلها في حي
Prayer niches (mihrabs)	محل الحق	محاريب الصلاة

Egyptian → English

Source	Model output	Reference
طب أجيب لك إيه؟ أجيب لك دي ولا دي؟ تخيل كده المنظر، قاعد دلوقتي لابس فانلة بحمالات وبتاكل أحلى حتة لحمة ممكن تكلها في حي	So what should I get for you? Should I get this one or that one? Imagine the view, sitting now wearing a pan with baking	So what should I get you? Should I get you this or that? Just imagine the scene: sitting now, wearing a sleeveless under
محاريب الصلاة	Greetings to the prayer	Prayer niches (mihrabs)

Usage

ChatML format. Pick the system prompt for the direction you want:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "oddadmix/Emhotob-25M-Egyptian-English-v2"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

SYSTEM = "أنت مترجم محترف. ترجم النص الإنجليزي إلى اللهجة المصرية العامية."

def translate(text, system=SYSTEM):
    prompt = (f"<|im_start|>system\n{system}<|im_end|>\n"
              f"<|im_start|>user\n{text.strip()}<|im_end|>\n<|im_start|>assistant\n")
    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
    if tok.bos_token_id is not None:
        bos = torch.tensor([[tok.bos_token_id]], device=model.device)
        ids["input_ids"] = torch.cat([bos, ids["input_ids"]], dim=1)
        ids["attention_mask"] = torch.cat([torch.ones_like(bos), ids["attention_mask"]], dim=1)
    out = model.generate(**ids, max_new_tokens=256, do_sample=False,
                         eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)
    return tok.decode(out[0, ids["input_ids"].size(1):], skip_special_tokens=True).strip()

Training

• Base model: oddadmix/Emhotob-25M-v2 (Llama arch, hidden 384, 8 layers, 6 heads, vocab 32000, tied embeddings; 25,271,424 params after resizing for 2 ChatML tokens)
• Dataset: oddadmix/egyptian-translation-dataset-2.9-openai-batch
• Method: HuggingFace Trainer, ChatML, prompt-masked cross-entropy (loss only on the assistant turn). Each row is exploded into two training examples (one per direction).
• Hyperparameters: 3 epochs · effective batch 64 · LR 3e-4 (cosine, 5% warmup) · bf16 · max length 1024 · load_best_model_at_end on eval_loss.
• Eval split: 3,000 deterministic held-out pairs (seed=42), scored both directions.

Out-of-domain evaluation — Egyptian Arabic Translation Benchmark

The results above are in-domain: a held-out split of the same corpus this model was trained on. The numbers below are out-of-domain — the same model scored on the Egyptian Arabic Translation Benchmark (oddadmix/egyptian-arabic-translation-benchmark, 319 English→Egyptian pairs written by a different annotator with different orthographic conventions).

Expect these to be substantially lower than the in-domain scores. That gap is the generalization penalty, not a regression — both numbers are real, they measure different things.

Metric	Score
BLEU (evaluate, 0–1 — leaderboard metric)	0.0824
BLEU (sacrebleu, 0–100)	8.24
chrF	40.29
METEOR	0.3148

Decoding is deterministic greedy (do_sample=False, no repetition penalty), using the exact ChatML prompt format the model was trained with — the same protocol as every other number in this study.

Where this rung sits

Model	Params	BLEU (hf)	BLEU (sacre)	chrF	METEOR
5M v1	5.2M	0.0000	0.19	13.00	0.0558
5M v2	5.2M	0.0108	1.08	22.19	0.1352
10M v1	11.2M	0.0313	3.13	28.52	0.1985
10M v2	10.9M	0.0341	3.41	31.60	0.2235
25M v1	25.3M	0.0991	9.91	41.07	0.3223
25M v2 (this model)	25.3M	0.0824	8.24	40.29	0.3148
50M v1 (bidi)	51.8M	0.1113	11.13	44.41	0.3482
50M v1 (uni)	51.8M	0.1261	12.61	46.13	0.3559

Reading these numbers

BLEU understates quality on this set. Scoring is against a single reference, so a correct translation that picks a different valid word is penalized — e.g. فريش vs the reference's طازة for "fresh", or التليفون اللي ضاع vs تليفونها الضايع for "her lost phone". Both are good Egyptian; only one matches the reference. chrF and METEOR track perceived quality more closely here.

At 319 rows, differences of roughly 1–2 BLEU between adjacent rungs are within noise.

These are small models — 5M to 50M parameters, orders of magnitude below the large systems typically evaluated on this benchmark. The result of interest is the scaling curve and per-parameter efficiency, not absolute rank against models 100–1000× the size.

Limitations

A ~25.3M model: reliable on short/common sentences, but drift, repetition, and errors appear on long or rare inputs. Gender is disambiguated only from context. For fluent translation use the 50M sibling.

License

Apache-2.0, inherited from the base model.