Hailay/llama2-amharic-english-mt

Vocabulary-Extended Llama-2-7B for Amharic → English Translation

LoRA fine-tunes meta-llama/Llama-2-7b-hf for Amharic-to-English translation, after extending its tokenizer with Ge'ez-script tokens (vocab 32,000 → 61,819) so Amharic text isn't shredded into near-meaningless byte fragments by Llama-2's original Latin-script-oriented vocabulary. New embedding rows for embed_tokens/lm_head are trained in full (not frozen, unlike the rest of the LoRA-adapted weights) since they start from a rough mean+random initialization.

Full training/data details: [github repo] (code + README) — trained on Helsinki-NLP/opus-100 (am-en, 89,027 pairs), prompted as:

Translate the following Amharic text to English.
Amharic: {source}
English: {target}

Seeds

5 independent training runs (different random seeds, identical data/hyperparameters), each in its own seed_<N>/ subfolder (LoRA adapter only — base model must be loaded separately and its embeddings resized to 61,819 tokens before applying the adapter; see finetune_model.py/evaluate.py in the code repo).

Results (Helsinki-NLP/opus-100 am-en test split, 2,000 sentences, greedy decoding)

Seed	Best eval_loss (training)	BLEU	chrF	COMET (Unbabel/wmt22-comet-da)
10	1.531	7.66	23.34	0.6318
11	1.530	7.81	23.73	0.6353
12	1.514	7.82	23.36	0.6330
13	1.523	7.85	23.55	0.6335
14	1.528	8.12	23.91	0.6339

All 5 seeds land within a tight band (BLEU 7.66-8.12, COMET 0.632-0.635), indicating this fine-tune is stable rather than seed-sensitive. Full per-sentence outputs and qualitative examples are in each seed_<N>/eval_report.json.

Usage

Requires access to the gated base model first: meta-llama/Llama-2-7b-hf requires accepting Meta's license before you can download it. Visit meta-llama/Llama-2-7b-hf while logged in to Hugging Face and request/accept access (usually granted instantly), then authenticate wherever you run the code below:

from huggingface_hub import notebook_login  # in a notebook/Colab
notebook_login()
# or, in a plain script/terminal:
#   huggingface-cli login
# or pass a token directly to each from_pretrained() call below: token="hf_..."

Without this step, loading the base model will fail with an HTTPStatusError (401/403) partway through from_pretrained.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

SEED_DIR = "Hailay/llama2-amharic-english-mt"  # + subfolder="seed_12"
BASE = "meta-llama/Llama-2-7b-hf"

tokenizer = AutoTokenizer.from_pretrained(SEED_DIR, subfolder="seed_12")
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16)
model.resize_token_embeddings(len(tokenizer), mean_resizing=False)
model = PeftModel.from_pretrained(model, SEED_DIR, subfolder="seed_12")

prompt = "Translate the following Amharic text to English.\nAmharic: ሰላም ከመይ ኣለኻ?\nEnglish:"
inputs = tokenizer(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(out[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True))

Limitations

• Trained/evaluated one direction only (Amharic → English).
• BLEU/chrF are corpus-level averages over machine-translated web text (opus-100); no human evaluation performed.
• Ge'ez-script vocabulary extension sourced from a separate project (EXLMR/vocab.json); not independently verified against a linguistic reference beyond the Ethiopic-script filter described in the code repo.

Installation

pip install transformers peft torch

LoRA configuration

Read from seed_10/adapter_config.json.

Parameter	Value
PEFT type	LoRA
Task type	CAUSAL_LM
Rank (r)	16
lora_alpha	32
lora_dropout	0.05
bias	none
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Modules saved	embed_tokens, lm_head
Base model	meta-llama/Llama-2-7b-hf

Learning rate, batch size, number of epochs, and optimizer settings are Not documented.

Citation

Not documented. No associated publication is recorded for this artifact.