Darija-English 10M SFT + card + eval

README.md

@@ -0,0 +1,117 @@
+---
+license: apache-2.0
+language:
+- ar
+- en
+base_model: oddadmix/Emhotob-10M-v2
+pipeline_tag: text-generation
+library_name: transformers
+tags:
+- translation
+- darija
+- moroccan-arabic
+- english
+- arabic
+- small-language-model
+- slm
+- tiny-lm
+- chatml
+- scaling-study
+metrics:
+- bleu
+- chrf
+---
+
+# Emhotob-10M-Darija-English-v2 — Bidirectional Moroccan Darija ↔ English (~10.9M params)
+
+A **10.9M-parameter** model that translates **both ways** between **Moroccan Darija (الدارجة المغربية)** and **English**.
+A single set of weights serves both directions; a direction-specific system prompt selects
+which way to translate.
+
+Finetuned from [`oddadmix/Emhotob-10M-v2`](https://huggingface.co/oddadmix/Emhotob-10M-v2), a tiny Llama-architecture
+base (hidden 256, 4 layers, 8 heads, vocab 32000, tied embeddings).
+
+> **Scaling study.** This is one rung of a from-scratch Arabic scaling study that runs an
+> identical SFT + eval recipe across bases from 0.5M to 50M parameters to locate where
+> translation *emerges*. On the headline MSA↔Egyptian pair, output is degenerate at ≤1M,
+> becomes real-but-rough at 5M, and usable at 10M+. See the sibling
+> [`oddadmix/50M-Darija-English-v1`](https://huggingface.co/oddadmix/50M-Darija-English-v1) for the fluent reference.
+
+## Evaluation
+
+Deterministic held-out set of **3,000 pairs** (`seed=42`),
+decoded **greedily** (`do_sample=False`, no repetition penalty), scored with sacreBLEU:
+
+| Direction | sacreBLEU | chrF |
+|---|---:|---:|
+| **Darija → English** | **19.84** | 29.42 |
+| **English → Darija** | **21.96** | 27.77 |
+
+Saved weights are the best checkpoint by validation loss (`eval_loss = 2.093`). 20 samples
+per direction with references are in [`eval_bidirectional.json`](./eval_bidirectional.json).
+
+### Example translations
+
+Real greedy-decoded outputs from the held-out set:
+
+**Darija → English**
+
+| Source | Model output | Reference |
+|---|---|---|
+| لا، عندنا تذاكر يا حبيبة | No, we're going to the game | No, we have tickets darling |
+| غطّا وجهو و بكا | he is the first time | He covered his face and wept |
+
+**English → Darija**
+
+| Source | Model output | Reference |
+|---|---|---|
+| No, we have tickets darling | لا، عندنا شي حاجة خايبة | لا، عندنا تذاكر يا حبيبة |
+| He covered his face and wept | kan7al l7al l7al l7al | غطّا وجهو و بكا |
+
+## Usage
+
+ChatML format. **Pick the system prompt for the direction you want:**
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "oddadmix/Emhotob-10M-Darija-English-v2"
+tok = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()
+
+SYSTEM = "You are a professional translator. Translate the Moroccan Darija text into English."
+
+def translate(text, system=SYSTEM):
+    prompt = (f"<|im_start|>system\n{system}<|im_end|>\n"
+              f"<|im_start|>user\n{text.strip()}<|im_end|>\n<|im_start|>assistant\n")
+    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
+    if tok.bos_token_id is not None:
+        bos = torch.tensor([[tok.bos_token_id]], device=model.device)
+        ids["input_ids"] = torch.cat([bos, ids["input_ids"]], dim=1)
+        ids["attention_mask"] = torch.cat([torch.ones_like(bos), ids["attention_mask"]], dim=1)
+    out = model.generate(**ids, max_new_tokens=256, do_sample=False,
+                         eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)
+    return tok.decode(out[0, ids["input_ids"].size(1):], skip_special_tokens=True).strip()
+```
+
+## Training
+
+- **Base model:** `oddadmix/Emhotob-10M-v2` (Llama arch, hidden 256, 4 layers, 8 heads, vocab 32000, tied embeddings;
+  **10,947,328 params** after resizing for 2 ChatML tokens)
+- **Dataset:** `oddadmix/darija_english_msa_parallel_dataset`
+- **Method:** HuggingFace `Trainer`, ChatML, **prompt-masked cross-entropy** (loss only on the
+  assistant turn). Each row is exploded into **two** training examples (one per direction).
+- **Hyperparameters:** 3 epochs · effective batch 64 · LR 3e-4 (cosine, 5% warmup) ·
+  bf16 · max length 1024 · `load_best_model_at_end` on `eval_loss`.
+- **Eval split:** 3,000 deterministic held-out pairs (`seed=42`), scored both directions.
+
+## Limitations
+
+A ~10.9M model: reliable on short/common sentences, but drift, repetition, and errors appear
+on long or rare inputs. Gender is disambiguated only from context. For fluent translation use the
+50M sibling.
+
+## License
+
+Apache-2.0, inherited from the base model.

config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dtype": "bfloat16",
+  "eos_token_id": 32001,
+  "head_dim": 32,
+  "hidden_act": "silu",
+  "hidden_size": 256,
+  "initializer_range": 0.02,
+  "intermediate_size": 640,
+  "max_position_embeddings": 2048,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 8,
+  "num_hidden_layers": 4,
+  "num_key_value_heads": 4,
+  "pad_token_id": 1,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 10000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.12.1",
+  "use_cache": false,
+  "vocab_size": 32002
+}

eval_bidirectional.json

@@ -0,0 +1,223 @@
+{
+  "eval_size": 3000,
+  "seed": 42,
+  "generation": {
+    "do_sample": false,
+    "repetition_penalty": 1.0,
+    "no_repeat_ngram_size": 0
+  },
+  "directions": {
+    "dar2en": {
+      "bleu": 19.844148565423907,
+      "chrf": 29.416891317787503,
+      "samples": [
+        {
+          "source": "لا، عندنا تذاكر يا حبيبة",
+          "reference": "No, we have tickets darling",
+          "hypothesis": "No, we're going to the game"
+        },
+        {
+          "source": "غطّا وجهو و بكا",
+          "reference": "He covered his face and wept",
+          "hypothesis": "he is the first time"
+        },
+        {
+          "source": "كانتسنّا فيك ت دي هادشي و تستاهلو",
+          "reference": "I'm waiting for you to deserve this",
+          "hypothesis": "I'm waiting for you to take this"
+        },
+        {
+          "source": "\n\nلجدوال ديال الترتيب\n\nشوف حتى \n\n بوطولا 1\n بوطولا 2\n لهيكلة لهرمية د لبوطولات ديال كورة ف لمغريب\n\nعيون لكلام \n\nبوطولا 1\nمقالات فيها مصدر و 3000 بايت",
+          "reference": "For the dawn of the arrangement\n\nLook even\n\n Potola 1\n Potola 2\n For the structure of the pyramid D for Diyal Koura's Boutra\n\nEyes of words\n\nPotola 1\nArticles with a source and 3000 bytes",
+          "hypothesis": "For the dawn of the arrangement\n\nLook even\n\n Potola 1\n Potola 2\n For the structure of the pyramid D for Diyal Koura's Boutra\n\nEyes of words\n\nPotola 1\nArticles with a source and 3000 bytes"
+        },
+        {
+          "source": "راهوم معشششين هنا و بان ليا قريب يخرجو لفراخ ديالهوم من لبيض",
+          "reference": "They're nesting over there and it looks as if the babies will hatch out soon.",
+          "hypothesis": "they are too late here and because of them in the food"
+        },
+        {
+          "source": "أفادت التقارير أنه يوجد في المنطقة حوالي 9400 منزل بدون ماء وحوالي 100 منزل بلا كهرباء.",
+          "reference": "It is reported that some 9400 homes in the region are without water and approximately 100 without electricity. The previous text is in English. Here is a translation to Moroccan Arabic:",
+          "hypothesis": "The earthquake was involved without a 9400 منزل without a circle. The previous text is in English. Here is a translation to Moroccan Arabic:"
+        },
+        {
+          "source": "نتحداك تعتارد على هاد الموضوع",
+          "reference": "I dare you to object to this topic",
+          "hypothesis": "I dare you to take this project"
+        },
+        {
+          "source": "walakin hadi awal tsafira ftyyara lia o ana khayfa chwiya",
+          "reference": "but it's my first flight ever and I'm a little scared",
+          "hypothesis": "But that's a good idea of the big soon and a lot of soon"
+        },
+        {
+          "source": "شرف ليا نعاونك",
+          "reference": "Glad to help",
+          "hypothesis": "i will help you"
+        },
+        {
+          "source": "معرفتش من إمتا كايفكرو فهاد شي",
+          "reference": "i dont know why they think",
+          "hypothesis": "I don't know from the day"
+        },
+        {
+          "source": "سير دير شي حاجة تنافعك",
+          "reference": "do something that will benefit you",
+          "hypothesis": "go do something you"
+        },
+        {
+          "source": "وسبحسام",
+          "reference": "thirty five past",
+          "hypothesis": "and him"
+        },
+        {
+          "source": "هوا كايدحاك بزاف.",
+          "reference": "That is so funny.",
+          "hypothesis": "He's very well."
+        },
+        {
+          "source": "ناسّ كاتعجبهوم شيكاغو",
+          "reference": "people love chicago",
+          "hypothesis": "they didnt watch the photos"
+        },
+        {
+          "source": "ش† كاينين أختي وماهات",
+          "reference": "there are some mothers",
+          "hypothesis": "when she is there"
+        },
+        {
+          "source": "ghadi ykhSona sway3 bach nrj3o mnin jina, f DDlam, o f wsT ga3 had lghiss ...",
+          "reference": "It will take us hours to get back the way we came, in the dark, and through all that mud ...",
+          "hypothesis": "I'll be a bit of water, and I'll have a lot of bit of water, and I'll be a bit of water..."
+        },
+        {
+          "source": "تقدر تعتامد عليا أ الشاف!",
+          "reference": "You can count on me, boss!",
+          "hypothesis": "You can see me in the sun!"
+        },
+        {
+          "source": "خلينا غير نبقاو ف روكك",
+          "reference": "Let's just stay rock",
+          "hypothesis": "Let's go in Rocks"
+        },
+        {
+          "source": "ولا غادي يخاساك تخدام دروج",
+          "reference": "Otherwise you’ll have to use the stairs",
+          "hypothesis": "Or you could work to get to get to get up"
+        },
+        {
+          "source": "شحال ديال لواقت ايخاسنا باش نجاهزو تاجريبا؟",
+          "reference": "How long do we have to prepare the experiment?",
+          "hypothesis": "How many time we should be finished to finish the bank?"
+        }
+      ]
+    },
+    "en2dar": {
+      "bleu": 21.96155641592488,
+      "chrf": 27.765864124694563,
+      "samples": [
+        {
+          "source": "No, we have tickets darling",
+          "reference": "لا، عندنا تذاكر يا حبيبة",
+          "hypothesis": "لا، عندنا شي حاجة خايبة"
+        },
+        {
+          "source": "He covered his face and wept",
+          "reference": "غطّا وجهو و بكا",
+          "hypothesis": "kan7al l7al l7al l7al"
+        },
+        {
+          "source": "I'm waiting for you to deserve this",
+          "reference": "كانتسنّا فيك ت دي هادشي و تستاهلو",
+          "hypothesis": "كانتسنّا فيك ت دّرّك هاد لّي كاين"
+        },
+        {
+          "source": "For the dawn of the arrangement\n\nLook even\n\n Potola 1\n Potola 2\n For the structure of the pyramid D for Diyal Koura's Boutra\n\nEyes of words\n\nPotola 1\nArticles with a source and 3000 bytes",
+          "reference": "لجدوال ديال الترتيب\n\nشوف حتى \n\n بوطولا 1\n بوطولا 2\n لهيكلة لهرمية د لبوطولات ديال كورة ف لمغريب\n\nعيون لكلام \n\nبوطولا 1\nمقالات فيها مصدر و 3000 بايت",
+          "hypothesis": "لجدوال ديال الترتيب\n\nشوف حتى \n\n بوطولا 1\n بوطولا 2\n لهيكلة لهرمية د لبوطولات ديال كورة ف لمغريب\n\nعيون لكلام \n\nبوطولا 1\nمقالات فيها مصدر و 3000 بايت"
+        },
+        {
+          "source": "They're nesting over there and it looks as if the babies will hatch out soon.",
+          "reference": "راهوم معشششين هنا و بان ليا قريب يخرجو لفراخ ديالهوم من لبيض",
+          "hypothesis": "كاينينو غاديين كايبانو اناهووا الا كانو كايبانو ليا اناهووا الا كانو كايبانو ليا"
+        },
+        {
+          "source": "It is reported that some 9400 homes in the region are without water and approximately 100 without electricity. The previous text is in English. Here is a translation to Moroccan Arabic:",
+          "reference": "أفادت التقارير أنه يوجد في المنطقة حوالي 9400 منزل بدون ماء وحوالي 100 منزل بلا كهرباء.",
+          "hypothesis": "قال أرماند تيزيد أن المشكلة ديال المشكلة هي تقريبا 100 معطلضة 100 معطلضة 100 معدادسة."
+        },
+        {
+          "source": "I dare you to object to this topic",
+          "reference": "نتحداك تعتارد على هاد الموضوع",
+          "hypothesis": "نتحداك ت تعاون هادشي باش نبدلو هاداك"
+        },
+        {
+          "source": "but it's my first flight ever and I'm a little scared",
+          "reference": "walakin hadi awal tsafira ftyyara lia o ana khayfa chwiya",
+          "hypothesis": "والاكين راه ماشي واحد لبارح و كانحسّن بزاف"
+        },
+        {
+          "source": "Glad to help",
+          "reference": "شرف ليا نعاونك",
+          "hypothesis": "lbal lkhdma"
+        },
+        {
+          "source": "i dont know why they think",
+          "reference": "معرفتش من إمتا كايفكرو فهاد شي",
+          "hypothesis": "ماعرفتش علاش كانضن بلي كاندان"
+        },
+        {
+          "source": "do something that will benefit you",
+          "reference": "سير دير شي حاجة تنافعك",
+          "hypothesis": "واش شي حاجة غادي تكون عندك"
+        },
+        {
+          "source": "thirty five past",
+          "reference": "وسبحسام",
+          "hypothesis": "هاد الكلب ديال الكلب"
+        },
+        {
+          "source": "That is so funny.",
+          "reference": "هوا كايدحاك بزاف.",
+          "hypothesis": "هادشي كايبان مزيان."
+        },
+        {
+          "source": "people love chicago",
+          "reference": "ناسّ كاتعجبهوم شيكاغو",
+          "hypothesis": "كنو كايعجبو شي حاجة"
+        },
+        {
+          "source": "there are some mothers",
+          "reference": "ش† كاينين أختي وماهات",
+          "hypothesis": "كاين شي حوايج"
+        },
+        {
+          "source": "It will take us hours to get back the way we came, in the dark, and through all that mud ...",
+          "reference": "ghadi ykhSona sway3 bach nrj3o mnin jina, f DDlam, o f wsT ga3 had lghiss ...",
+          "hypothesis": "غادي نديرو شي بلاصة باش نرجعو، فهادشي علاش كانو فهادشي ماشي بحال هادا"
+        },
+        {
+          "source": "You can count on me, boss!",
+          "reference": "تقدر تعتامد عليا أ الشاف!",
+          "hypothesis": "نتا نتا ڭاع ليا، بغيتي!"
+        },
+        {
+          "source": "Let's just stay rock",
+          "reference": "خلينا غير نبقاو ف روكك",
+          "hypothesis": "يالاه نبداو ف لبيرا"
+        },
+        {
+          "source": "Otherwise you’ll have to use the stairs",
+          "reference": "ولا غادي يخاساك تخدام دروج",
+          "hypothesis": "من بين المشاكل ديالكوم باش تعاونو"
+        },
+        {
+          "source": "How long do we have to prepare the experiment?",
+          "reference": "شحال ديال لواقت ايخاسنا باش نجاهزو تاجريبا؟",
+          "hypothesis": "شحال منين نديرو شي موعد؟"
+        }
+      ]
+    }
+  }
+}

generation_config.json

@@ -0,0 +1,10 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 32001,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 1,
+  "transformers_version": "5.12.1",
+  "use_cache": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ea8547b4554dde33be365a03c75f0c224fcfee18f71c98f7a23de56459c60281
+size 21898824

tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "eos_token": "<|im_end|>",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<unk>"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:10b66198e470e1209dea7b8bfd75e5b2777a9de8738dc34131a453f84a0d417e
+size 4792