Darija-English 5M SFT + card + eval

README.md

@@ -0,0 +1,117 @@
+---
+license: apache-2.0
+language:
+- ar
+- en
+base_model: oddadmix/Emhotob-5M-v2
+pipeline_tag: text-generation
+library_name: transformers
+tags:
+- translation
+- darija
+- moroccan-arabic
+- english
+- arabic
+- small-language-model
+- slm
+- tiny-lm
+- chatml
+- scaling-study
+metrics:
+- bleu
+- chrf
+---
+
+# Emhotob-5M-Darija-English-v1 — Bidirectional Moroccan Darija ↔ English (~5.1M params)
+
+A **5.1M-parameter** model that translates **both ways** between **Moroccan Darija (الدارجة المغربية)** and **English**.
+A single set of weights serves both directions; a direction-specific system prompt selects
+which way to translate.
+
+Finetuned from [`oddadmix/Emhotob-5M-v2`](https://huggingface.co/oddadmix/Emhotob-5M-v2), a tiny Llama-architecture
+base (hidden 128, 5 layers, 4 heads, vocab 32000, tied embeddings).
+
+> **Scaling study.** This is one rung of a from-scratch Arabic scaling study that runs an
+> identical SFT + eval recipe across bases from 0.5M to 50M parameters to locate where
+> translation *emerges*. On the headline MSA↔Egyptian pair, output is degenerate at ≤1M,
+> becomes real-but-rough at 5M, and usable at 10M+. See the sibling
+> [`oddadmix/50M-Darija-English-v1`](https://huggingface.co/oddadmix/50M-Darija-English-v1) for the fluent reference.
+
+## Evaluation
+
+Deterministic held-out set of **3,000 pairs** (`seed=42`),
+decoded **greedily** (`do_sample=False`, no repetition penalty), scored with sacreBLEU:
+
+| Direction | sacreBLEU | chrF |
+|---|---:|---:|
+| **Darija → English** | **13.45** | 23.93 |
+| **English → Darija** | **11.80** | 20.87 |
+
+Saved weights are the best checkpoint by validation loss (`eval_loss = 2.690`). 20 samples
+per direction with references are in [`eval_bidirectional.json`](./eval_bidirectional.json).
+
+### Example translations
+
+Real greedy-decoded outputs from the held-out set:
+
+**Darija → English**
+
+| Source | Model output | Reference |
+|---|---|---|
+| لا، عندنا تذاكر يا حبيبة | No, I'm sorry to be a bit | No, we have tickets darling |
+| غطّا وجهو و بكا | do you want to say | He covered his face and wept |
+
+**English → Darija**
+
+| Source | Model output | Reference |
+|---|---|---|
+| No, we have tickets darling | لا، غادي نديرو | لا، عندنا تذاكر يا حبيبة |
+| He covered his face and wept | هوا شي حاجة و لكن | غطّا وجهو و بكا |
+
+## Usage
+
+ChatML format. **Pick the system prompt for the direction you want:**
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "oddadmix/Emhotob-5M-Darija-English-v2"
+tok = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()
+
+SYSTEM = "You are a professional translator. Translate the Moroccan Darija text into English."
+
+def translate(text, system=SYSTEM):
+    prompt = (f"<|im_start|>system\n{system}<|im_end|>\n"
+              f"<|im_start|>user\n{text.strip()}<|im_end|>\n<|im_start|>assistant\n")
+    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
+    if tok.bos_token_id is not None:
+        bos = torch.tensor([[tok.bos_token_id]], device=model.device)
+        ids["input_ids"] = torch.cat([bos, ids["input_ids"]], dim=1)
+        ids["attention_mask"] = torch.cat([torch.ones_like(bos), ids["attention_mask"]], dim=1)
+    out = model.generate(**ids, max_new_tokens=256, do_sample=False,
+                         eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)
+    return tok.decode(out[0, ids["input_ids"].size(1):], skip_special_tokens=True).strip()
+```
+
+## Training
+
+- **Base model:** `oddadmix/Emhotob-5M-v2` (Llama arch, hidden 128, 5 layers, 4 heads, vocab 32000, tied embeddings;
+  **5,080,704 params** after resizing for 2 ChatML tokens)
+- **Dataset:** `oddadmix/darija_english_msa_parallel_dataset`
+- **Method:** HuggingFace `Trainer`, ChatML, **prompt-masked cross-entropy** (loss only on the
+  assistant turn). Each row is exploded into **two** training examples (one per direction).
+- **Hyperparameters:** 3 epochs · effective batch 64 · LR 3e-4 (cosine, 5% warmup) ·
+  bf16 · max length 1024 · `load_best_model_at_end` on `eval_loss`.
+- **Eval split:** 3,000 deterministic held-out pairs (`seed=42`), scored both directions.
+
+## Limitations
+
+A ~5.1M model: reliable on short/common sentences, but drift, repetition, and errors appear
+on long or rare inputs. Gender is disambiguated only from context. For fluent translation use the
+50M sibling.
+
+## License
+
+Apache-2.0, inherited from the base model.

config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dtype": "bfloat16",
+  "eos_token_id": 32001,
+  "head_dim": 32,
+  "hidden_act": "silu",
+  "hidden_size": 128,
+  "initializer_range": 0.02,
+  "intermediate_size": 384,
+  "max_position_embeddings": 2048,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 5,
+  "num_key_value_heads": 2,
+  "pad_token_id": 1,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 10000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.12.1",
+  "use_cache": false,
+  "vocab_size": 32002
+}

eval_bidirectional.json

@@ -0,0 +1,223 @@
+{
+  "eval_size": 3000,
+  "seed": 42,
+  "generation": {
+    "do_sample": false,
+    "repetition_penalty": 1.0,
+    "no_repeat_ngram_size": 0
+  },
+  "directions": {
+    "dar2en": {
+      "bleu": 13.446299302555513,
+      "chrf": 23.93451764854585,
+      "samples": [
+        {
+          "source": "لا، عندنا تذاكر يا حبيبة",
+          "reference": "No, we have tickets darling",
+          "hypothesis": "No, I'm sorry to be a bit"
+        },
+        {
+          "source": "غطّا وجهو و بكا",
+          "reference": "He covered his face and wept",
+          "hypothesis": "do you want to say"
+        },
+        {
+          "source": "كانتسنّا فيك ت دي هادشي و تستاهلو",
+          "reference": "I'm waiting for you to deserve this",
+          "hypothesis": "I'm waiting for you to be a bit"
+        },
+        {
+          "source": "\n\nلجدوال ديال الترتيب\n\nشوف حتى \n\n بوطولا 1\n بوطولا 2\n لهيكلة لهرمية د لبوطولات ديال كورة ف لمغريب\n\nعيون لكلام \n\nبوطولا 1\nمقالات فيها مصدر و 3000 بايت",
+          "reference": "For the dawn of the arrangement\n\nLook even\n\n Potola 1\n Potola 2\n For the structure of the pyramid D for Diyal Koura's Boutra\n\nEyes of words\n\nPotola 1\nArticles with a source and 3000 bytes",
+          "hypothesis": "For the dawn of the arrangement\n\nLook even\n\n Potola 1\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Potola 2\n Pot"
+        },
+        {
+          "source": "راهوم معشششين هنا و بان ليا قريب يخرجو لفراخ ديالهوم من لبيض",
+          "reference": "They're nesting over there and it looks as if the babies will hatch out soon.",
+          "hypothesis": "he is not someone here and someone"
+        },
+        {
+          "source": "أفادت التقارير أنه يوجد في المنطقة حوالي 9400 منزل بدون ماء وحوالي 100 منزل بلا كهرباء.",
+          "reference": "It is reported that some 9400 homes in the region are without water and approximately 100 without electricity. The previous text is in English. Here is a translation to Moroccan Arabic:",
+          "hypothesis": "The same year in the earthquake side of the country, the earthquake side of the country. The previous text is in English. Here is a translation to Moroccan Arabic:"
+        },
+        {
+          "source": "نتحداك تعتارد على هاد الموضوع",
+          "reference": "I dare you to object to this topic",
+          "hypothesis": "I dare you to be a lot"
+        },
+        {
+          "source": "walakin hadi awal tsafira ftyyara lia o ana khayfa chwiya",
+          "reference": "but it's my first flight ever and I'm a little scared",
+          "hypothesis": "Do you know what you're not sure you're not sure you're not sure"
+        },
+        {
+          "source": "شرف ليا نعاونك",
+          "reference": "Glad to help",
+          "hypothesis": "you are too late to yourself"
+        },
+        {
+          "source": "معرفتش من إمتا كايفكرو فهاد شي",
+          "reference": "i dont know why they think",
+          "hypothesis": "I don't know what to do"
+        },
+        {
+          "source": "سير دير شي حاجة تنافعك",
+          "reference": "do something that will benefit you",
+          "hypothesis": "go to see you"
+        },
+        {
+          "source": "وسبحسام",
+          "reference": "thirty five past",
+          "hypothesis": "And they're her"
+        },
+        {
+          "source": "هوا كايدحاك بزاف.",
+          "reference": "That is so funny.",
+          "hypothesis": "He is a good idea."
+        },
+        {
+          "source": "ناسّ كاتعجبهوم شيكاغو",
+          "reference": "people love chicago",
+          "hypothesis": "It's a lot of them"
+        },
+        {
+          "source": "ش† كاينين أختي وماهات",
+          "reference": "there are some mothers",
+          "hypothesis": "i took her"
+        },
+        {
+          "source": "ghadi ykhSona sway3 bach nrj3o mnin jina, f DDlam, o f wsT ga3 had lghiss ...",
+          "reference": "It will take us hours to get back the way we came, in the dark, and through all that mud ...",
+          "hypothesis": "It's a good phone, friend, friend, and friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, friend, fri"
+        },
+        {
+          "source": "تقدر تعتامد عليا أ الشاف!",
+          "reference": "You can count on me, boss!",
+          "hypothesis": "You're not sure I'm sorry!"
+        },
+        {
+          "source": "خلينا غير نبقاو ف روكك",
+          "reference": "Let's just stay rock",
+          "hypothesis": "Let's started in God"
+        },
+        {
+          "source": "ولا غادي يخاساك تخدام دروج",
+          "reference": "Otherwise you’ll have to use the stairs",
+          "hypothesis": "I'll take the two day"
+        },
+        {
+          "source": "شحال ديال لواقت ايخاسنا باش نجاهزو تاجريبا؟",
+          "reference": "How long do we have to prepare the experiment?",
+          "hypothesis": "What do you think often do you have to do together?"
+        }
+      ]
+    },
+    "en2dar": {
+      "bleu": 11.79660565975206,
+      "chrf": 20.86721324211251,
+      "samples": [
+        {
+          "source": "No, we have tickets darling",
+          "reference": "لا، عندنا تذاكر يا حبيبة",
+          "hypothesis": "لا، غادي نديرو"
+        },
+        {
+          "source": "He covered his face and wept",
+          "reference": "غطّا وجهو و بكا",
+          "hypothesis": "هوا شي حاجة و لكن"
+        },
+        {
+          "source": "I'm waiting for you to deserve this",
+          "reference": "كانتسنّا فيك ت دي هادشي و تستاهلو",
+          "hypothesis": "كانتسنّا فيك ت قاسح ليا"
+        },
+        {
+          "source": "For the dawn of the arrangement\n\nLook even\n\n Potola 1\n Potola 2\n For the structure of the pyramid D for Diyal Koura's Boutra\n\nEyes of words\n\nPotola 1\nArticles with a source and 3000 bytes",
+          "reference": "لجدوال ديال الترتيب\n\nشوف حتى \n\n بوطولا 1\n بوطولا 2\n لهيكلة لهرمية د لبوطولات ديال كورة ف لمغريب\n\nعيون لكلام \n\nبوطولا 1\nمقالات فيها مصدر و 3000 بايت",
+          "hypothesis": "لجدوال ديال الترتيب\n\nشوف حتى \n\n بوطولا 1\n بوطولا 2\n لهيكلة لهرمية د لبوطولات ديال كورة ف لمغريب\n\nعيون لكلام \n\nبوطولا 1\nمقالات فيها مصدر و 3000 بايت"
+        },
+        {
+          "source": "They're nesting over there and it looks as if the babies will hatch out soon.",
+          "reference": "راهوم معشششين هنا و بان ليا قريب يخرجو لفراخ ديالهوم من لبيض",
+          "hypothesis": "كاينين بزاف ديال ليا غادي تاخد شي حاجة باش نديرو شي حاجة."
+        },
+        {
+          "source": "It is reported that some 9400 homes in the region are without water and approximately 100 without electricity. The previous text is in English. Here is a translation to Moroccan Arabic:",
+          "reference": "أفادت التقارير أنه يوجد في المنطقة حوالي 9400 منزل بدون ماء وحوالي 100 منزل بلا كهرباء.",
+          "hypothesis": "كاين 9400 واحد، واحد واحد من 100 من 100 من 100 من 100 من 100 من 100 من 100 من 100 من 100 من 100 من 100 من 100 من 100."
+        },
+        {
+          "source": "I dare you to object to this topic",
+          "reference": "نتحداك تعتارد على هاد الموضوع",
+          "hypothesis": "نتحداك ت تڭولّا هادشي لي كاتستاهل"
+        },
+        {
+          "source": "but it's my first flight ever and I'm a little scared",
+          "reference": "walakin hadi awal tsafira ftyyara lia o ana khayfa chwiya",
+          "hypothesis": "walakin kayn 3lih 3la lmakhir"
+        },
+        {
+          "source": "Glad to help",
+          "reference": "شرف ليا نعاونك",
+          "hypothesis": "l7al l7al"
+        },
+        {
+          "source": "i dont know why they think",
+          "reference": "معرفتش من إمتا كايفكرو فهاد شي",
+          "hypothesis": "مانضنش غادي نديرو"
+        },
+        {
+          "source": "do something that will benefit you",
+          "reference": "سير دير شي حاجة تنافعك",
+          "hypothesis": "ماتبقاش نديرو هادشي"
+        },
+        {
+          "source": "thirty five past",
+          "reference": "وسبحسام",
+          "hypothesis": "هادشي كايعجبك"
+        },
+        {
+          "source": "That is so funny.",
+          "reference": "هوا كايدحاك بزاف.",
+          "hypothesis": "هادش�� غادي نديرو."
+        },
+        {
+          "source": "people love chicago",
+          "reference": "ناسّ كاتعجبهوم شيكاغو",
+          "hypothesis": "راه خدام بزاف"
+        },
+        {
+          "source": "there are some mothers",
+          "reference": "ش† كاينين أختي وماهات",
+          "hypothesis": "كاينين بزاف"
+        },
+        {
+          "source": "It will take us hours to get back the way we came, in the dark, and through all that mud ...",
+          "reference": "ghadi ykhSona sway3 bach nrj3o mnin jina, f DDlam, o f wsT ga3 had lghiss ...",
+          "hypothesis": "غادي نديرو شي حاجة فهادشي، فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهادشي، و كاين شي حاجة فهاد"
+        },
+        {
+          "source": "You can count on me, boss!",
+          "reference": "تقدر تعتامد عليا أ الشاف!",
+          "hypothesis": "مابقيتش باغي نديرو ليا، غادي نديرو"
+        },
+        {
+          "source": "Let's just stay rock",
+          "reference": "خلينا غير نبقاو ف روكك",
+          "hypothesis": "خليني نديرو"
+        },
+        {
+          "source": "Otherwise you’ll have to use the stairs",
+          "reference": "ولا غادي يخاساك تخدام دروج",
+          "hypothesis": "واخا تاخد شي حاجة"
+        },
+        {
+          "source": "How long do we have to prepare the experiment?",
+          "reference": "شحال ديال لواقت ايخاسنا باش نجاهزو تاجريبا؟",
+          "hypothesis": "شحال هادشي لي كانو كاين شي حاجة؟"
+        }
+      ]
+    }
+  }
+}

generation_config.json

@@ -0,0 +1,10 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 32001,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 1,
+  "transformers_version": "5.12.1",
+  "use_cache": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:842feddaee7e3d349a6f78dda751811ef4113c6788c278a8d41f45f8d8018c6b
+size 10166472

tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "eos_token": "<|im_end|>",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<unk>"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:57c0be99e00ec7643b955aa7a9fc22b97be9fca1f154754c865615d8078fa8ce
+size 4792