README.md

---
license: apache-2.0
language:
- ar
- en
base_model: oddadmix/Emhotob-5M-v2
pipeline_tag: text-generation
library_name: transformers
tags:
- translation
- darija
- moroccan-arabic
- english
- arabic
- small-language-model
- slm
- tiny-lm
- chatml
- scaling-study
metrics:
- bleu
- chrf
---

# Emhotob-5M-Darija-English-v1 — Bidirectional Moroccan Darija ↔ English (~5.1M params)

A **5.1M-parameter** model that translates **both ways** between **Moroccan Darija (الدارجة المغربية)** and **English**.
A single set of weights serves both directions; a direction-specific system prompt selects
which way to translate.

Finetuned from [`oddadmix/Emhotob-5M-v2`](https://huggingface.co/oddadmix/Emhotob-5M-v2), a tiny Llama-architecture
base (hidden 128, 5 layers, 4 heads, vocab 32000, tied embeddings).

> **Scaling study.** This is one rung of a from-scratch Arabic scaling study that runs an
> identical SFT + eval recipe across bases from 0.5M to 50M parameters to locate where
> translation *emerges*. On the headline MSA↔Egyptian pair, output is degenerate at ≤1M,
> becomes real-but-rough at 5M, and usable at 10M+. See the sibling
> [`oddadmix/50M-Darija-English-v1`](https://huggingface.co/oddadmix/50M-Darija-English-v1) for the fluent reference.

## Evaluation

Deterministic held-out set of **3,000 pairs** (`seed=42`),
decoded **greedily** (`do_sample=False`, no repetition penalty), scored with sacreBLEU:

| Direction | sacreBLEU | chrF |
|---|---:|---:|
| **Darija → English** | **13.45** | 23.93 |
| **English → Darija** | **11.80** | 20.87 |

Saved weights are the best checkpoint by validation loss (`eval_loss = 2.690`). 20 samples
per direction with references are in [`eval_bidirectional.json`](./eval_bidirectional.json).

### Example translations

Real greedy-decoded outputs from the held-out set:

**Darija → English**

| Source | Model output | Reference |
|---|---|---|
| لا، عندنا تذاكر يا حبيبة | No, I'm sorry to be a bit | No, we have tickets darling |
| غطّا وجهو و بكا | do you want to say | He covered his face and wept |

**English → Darija**

| Source | Model output | Reference |
|---|---|---|
| No, we have tickets darling | لا، غادي نديرو | لا، عندنا تذاكر يا حبيبة |
| He covered his face and wept | هوا شي حاجة و لكن | غطّا وجهو و بكا |

## Usage

ChatML format. **Pick the system prompt for the direction you want:**

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "oddadmix/Emhotob-5M-Darija-English-v2"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

SYSTEM = "You are a professional translator. Translate the Moroccan Darija text into English."

def translate(text, system=SYSTEM):
    prompt = (f"<|im_start|>system\n{system}<|im_end|>\n"
              f"<|im_start|>user\n{text.strip()}<|im_end|>\n<|im_start|>assistant\n")
    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
    if tok.bos_token_id is not None:
        bos = torch.tensor([[tok.bos_token_id]], device=model.device)
        ids["input_ids"] = torch.cat([bos, ids["input_ids"]], dim=1)
        ids["attention_mask"] = torch.cat([torch.ones_like(bos), ids["attention_mask"]], dim=1)
    out = model.generate(**ids, max_new_tokens=256, do_sample=False,
                         eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)
    return tok.decode(out[0, ids["input_ids"].size(1):], skip_special_tokens=True).strip()
```

## Training

- **Base model:** `oddadmix/Emhotob-5M-v2` (Llama arch, hidden 128, 5 layers, 4 heads, vocab 32000, tied embeddings;
  **5,080,704 params** after resizing for 2 ChatML tokens)
- **Dataset:** `oddadmix/darija_english_msa_parallel_dataset`
- **Method:** HuggingFace `Trainer`, ChatML, **prompt-masked cross-entropy** (loss only on the
  assistant turn). Each row is exploded into **two** training examples (one per direction).
- **Hyperparameters:** 3 epochs · effective batch 64 · LR 3e-4 (cosine, 5% warmup) ·
  bf16 · max length 1024 · `load_best_model_at_end` on `eval_loss`.
- **Eval split:** 3,000 deterministic held-out pairs (`seed=42`), scored both directions.

## Limitations

A ~5.1M model: reliable on short/common sentences, but drift, repetition, and errors appear
on long or rare inputs. Gender is disambiguated only from context. For fluent translation use the
50M sibling.

## License

Apache-2.0, inherited from the base model.