Emhotob-5M-English-MSA-v1 — Bidirectional English ↔ MSA (~5M params)

A 5.08M-parameter model that translates both ways between English and Modern Standard Arabic (الفصحى). A single set of weights serves both directions; a direction-specific system prompt selects which way to translate.

Finetuned from oddadmix/Emhotob-5M, a tiny Llama-architecture base (hidden size 128, 5 layers, 4 heads, tied embeddings).

Scaling study. This runs the exact recipe of oddadmix/50M-English-MSA-v1 on a base ~10× smaller. At 5M short sentences translate correctly while longer inputs drift and repeat. It is a scaling demonstration, not a production translator; use the 50M sibling for fluent output.

Evaluation

Evaluated on a deterministic held-out set of 3,000 pairs (seed=42), decoded greedily (do_sample=False, no repetition penalty), scored with sacreBLEU:

Direction	sacreBLEU	chrF
English → MSA	2.79	20.23
MSA → English	3.10	17.63

The saved weights are the best checkpoint by validation loss (eval_loss = 3.105, epoch 3 of 3). For reference, the 50M sibling scores BLEU ~46 (en→msa) / ~50 (msa→en) with the same data and eval.

Example translations

Real greedy-decoded outputs from the held-out set:

English → MSA

English input	Model output (MSA)	Reference
Thank you very much, you are so kind.	شكرًا لك، أنت رائع جدًا.	شكرًا جزيلًا لك، أنت لطيف للغاية.
I'm just trying to reassure myself. What's the worst…	أنا فقط أتفق معك. ما الذي يحدث؟ …	أنا فقط أحاول أن أطمئن نفسي…

MSA → English

MSA input	Model output (English)	Reference
شكرًا جزيلًا لك، أنت لطيف للغاية.	Thank you and you're saying, you're saying.	Thank you very much, you are so kind.

Short, common sentences come out well; longer inputs lose coherence. 20 samples per direction with references are in eval_bidirectional.json.

Usage

ChatML format. Pick the system prompt for the direction you want:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "oddadmix/Emhotob-5M-English-MSA-v1"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

SYS_TO_MSA = "أنت مترجم محترف. ترجم النص الإنجليزي إلى اللغة العربية الفصحى."
SYS_TO_EN  = "You are a professional translator. Translate the Modern Standard Arabic text into English."

def translate(text: str, system: str) -> str:
    prompt = (
        f"<|im_start|>system\n{system}<|im_end|>\n"
        f"<|im_start|>user\n{text.strip()}<|im_end|>\n"
        f"<|im_start|>assistant\n"
    )
    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
    if tok.bos_token_id is not None:  # training prepends BOS
        bos = torch.tensor([[tok.bos_token_id]], device=model.device)
        ids["input_ids"] = torch.cat([bos, ids["input_ids"]], dim=1)
        ids["attention_mask"] = torch.cat([torch.ones_like(bos), ids["attention_mask"]], dim=1)
    out = model.generate(**ids, max_new_tokens=256, do_sample=False,
                         eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)
    return tok.decode(out[0, ids["input_ids"].size(1):], skip_special_tokens=True).strip()

print(translate("Thank you very much, you are so kind.", SYS_TO_MSA))

Training

• Base model: oddadmix/Emhotob-5M (Llama arch, hidden 128, 5 layers, 4 heads, vocab 32000, tied embeddings; 5,080,704 params after resizing for 2 ChatML tokens)
• Dataset: oddadmix/egyptian-msa-2.9-openai-bytedance-translations (132K rows; this model uses the english and msa columns)
• Method: HuggingFace Trainer, ChatML, prompt-masked cross-entropy (loss only on the assistant turn). Each row is exploded into two training examples (one per direction). Two ChatML special tokens (<|im_start|>, <|im_end|>) were added and embeddings resized.
• Hyperparameters: 3 epochs · effective batch 64 · LR 3e-4 (cosine, 5% warmup) · bf16 · max length 1024 · load_best_model_at_end on eval_loss.
• Split: 129,009 train / 3,000 deterministic held-out (seed=42), scored both directions.

Limitations

• A 5M model near the emergence threshold: reliable only on short, common sentences; longer or rarer inputs drift, repeat, or lose meaning.
• For fluent translation use oddadmix/50M-English-MSA-v1.

License

Apache-2.0, inherited from the base model.