Emhotob-10M-English-MSA-v1 — Bidirectional English ↔ MSA (~11M params)

An 10.9M-parameter model that translates both ways between English and Modern Standard Arabic (الفصحى). A single set of weights serves both directions; a direction-specific system prompt selects which way to translate.

Finetuned from oddadmix/Emhotob-10M, a tiny Llama-architecture base (hidden size 256, 4 layers, 8 heads, tied embeddings).

Scaling study — strongest 10M pair. This runs the exact recipe of oddadmix/50M-English-MSA-v1 on a base ~5× smaller. Because MSA is highly standardized, at 10M this pair is genuinely usable — short and medium sentences are often translated correctly, sometimes matching the reference verbatim. Long or rare inputs can still drift.

Evaluation

Evaluated on a deterministic held-out set of 3,000 pairs (seed=42), decoded greedily (do_sample=False, no repetition penalty), scored with sacreBLEU:

Direction	sacreBLEU	chrF
English → MSA	24.78	47.74
MSA → English	32.12	49.77

The saved weights are the best checkpoint by validation loss (eval_loss = 1.346, epoch 3 of 3) — the lowest of the 10M suite. For reference, the 50M sibling scores BLEU ~46 (en→msa) / ~50 (msa→en); at 5M this pair scored ~3 both ways.

Example translations

Real greedy-decoded outputs from the held-out set:

English → MSA

English input	Model output (MSA)	Reference
Thank you very much, you are so kind.	شكرًا جزيلًا لك، أنت لطيف جدًا.	شكرًا جزيلًا لك، أنت لطيف للغاية.

MSA → English

MSA input	Model output (English)	Reference
أنا فقط أحاول أن أطمئن نفسي. ما هو الأسوأ…	I'm just trying to reassure myself. What's the worst…	I'm just trying to reassure myself. What's the worst…
شكرًا جزيلًا لك، أنت لطيف للغاية.	Thank you so much, you're so nice.	Thank you very much, you are so kind.

Common sentences are often correct (the first MSA→English row matches the reference verbatim); longer inputs may drift. 20 samples per direction with references are in eval_bidirectional.json.

Usage

ChatML format. Pick the system prompt for the direction you want:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "oddadmix/Emhotob-10M-English-MSA-v1"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

SYS_TO_MSA = "أنت مترجم محترف. ترجم النص الإنجليزي إلى اللغة العربية الفصحى."
SYS_TO_EN  = "You are a professional translator. Translate the Modern Standard Arabic text into English."

def translate(text: str, system: str) -> str:
    prompt = (
        f"<|im_start|>system\n{system}<|im_end|>\n"
        f"<|im_start|>user\n{text.strip()}<|im_end|>\n"
        f"<|im_start|>assistant\n"
    )
    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
    if tok.bos_token_id is not None:  # training prepends BOS
        bos = torch.tensor([[tok.bos_token_id]], device=model.device)
        ids["input_ids"] = torch.cat([bos, ids["input_ids"]], dim=1)
        ids["attention_mask"] = torch.cat([torch.ones_like(bos), ids["attention_mask"]], dim=1)
    out = model.generate(**ids, max_new_tokens=256, do_sample=False,
                         eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)
    return tok.decode(out[0, ids["input_ids"].size(1):], skip_special_tokens=True).strip()

print(translate("Thank you very much, you are so kind.", SYS_TO_MSA))

Training

• Base model: oddadmix/Emhotob-10M (Llama arch, hidden 256, 4 layers, 8 heads, vocab 32000, tied embeddings; 10,947,328 params after resizing for 2 ChatML tokens)
• Dataset: oddadmix/egyptian-msa-2.9-openai-bytedance-translations (132K rows; this model uses the english and msa columns)
• Method: HuggingFace Trainer, ChatML, prompt-masked cross-entropy (loss only on the assistant turn). Each row is exploded into two training examples (one per direction). Two ChatML special tokens (<|im_start|>, <|im_end|>) were added and embeddings resized.
• Hyperparameters: 3 epochs · effective batch 64 · LR 3e-4 (cosine, 5% warmup) · bf16 · max length 1024 · load_best_model_at_end on eval_loss.
• Split: 129,009 train / 3,000 deterministic held-out (seed=42), scored both directions.

Limitations

• An 11M model: strong on short/common sentences, but drift and errors appear on long or rare inputs.
• For fluent translation use oddadmix/50M-English-MSA-v1.

License

Apache-2.0, inherited from the base model.