qwen2.5-3b-arabic-context-qa-lora

Fine-tuned Qwen/Qwen2.5-3B-Instruct for monolingual Arabic extractive context-based question answering, trained as a submission to the WMT26 Multilingual Instruction Shared Task (MIST), Sub-task 1 (context-based QA).

Given a passage and a question in Arabic, the model returns the shortest exact answer span copied verbatim from the passage — it does not paraphrase, explain, or generate free text.

Model Details

• Base model: Qwen/Qwen2.5-3B-Instruct
• Fine-tuning method: PEFT / LoRA adapter (base model weights unchanged)
• Language: Arabic (monolingual — both context and question in Arabic)
• Task: Extractive context-based question answering
• License: Inherits the base model's license (Qwen/Qwen2.5-3B-Instruct) — review the base model's license terms before use.

LoRA Configuration

Parameter	Value
r (rank)	32
alpha	64
dropout	0.05
target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Training Data

• Source: pinzhenchen/wmt26-mist-sample, filtered to source == "answerable_tydiqa" (the answerable subset of TyDiQA).
• Size: 3,112 Arabic context/question/answer triples.
• Format: ChatML, with the loss computed only on the gold answer span (character-offset-based completion-only masking).

Note: the training data does not include unanswerable questions or cross-lingual (context/question in different languages) examples. As a result, this model is not expected to perform well on either of those cases — see Limitations below.

Training Procedure

• Precision: BF16, no quantization
• Epochs: 3.0
• Effective batch size: 4 × 8 (per-device × grad. accumulation)
• Learning rate: 0.0002 (cosine schedule, 0.03 warmup ratio)
• Max sequence length: 1024
• Hardware: Single NVIDIA RTX 4090 (24GB VRAM)

Evaluation

Evaluated on a held-out 2% split (63 examples) of the training data.

Metric	Value
Train loss (final)	0.0822
Eval loss	0.1658
Eval mean token accuracy	0.9643

Manual review of the official WMT26 MIST test set (monolingual Arabic instances) confirms the model reliably produces short, accurate, verbatim extractive spans.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct", dtype=torch.bfloat16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("hasankh/qwen2.5-3b-arabic-context-qa-lora")
model = PeftModel.from_pretrained(base_model, "hasankh/qwen2.5-3b-arabic-context-qa-lora")

context = "..."
question = "..."
messages = [
    {"role": "system", "content": "You are an extractive multilingual question answering system..."},
    {"role": "user", "content": f"Context:\n\n{context}\n\nQuestion:\n\n{question}\n\nAnswer:\n"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)
answer = tokenizer.decode(output_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(answer)

Limitations

• Unanswerable questions: the model was not trained on unanswerable questions and will fabricate a plausible-looking span rather than indicating no answer exists.
• Cross-lingual inputs: the model was not trained on examples where the passage and question are in different languages, and performs unreliably in that setting (verbatim copying of the source-language text, script-mixing, or ignoring the target-language instruction).
• Scope: only evaluated on monolingual Arabic context-based QA; behavior on other languages or other task types (open-ended generation, summarization) is unverified.

Citation

If you use this model, please cite the WMT26 MIST shared task:

@misc{wmt26-mist,
  title  = {Multilingual Instruction Shared Task (MIST)},
  author = {Chen, Pinzhen and Schmidtov{\'a}, Patr{\'i}cia and others},
  year   = {2026},
  url    = {https://www2.statmt.org/wmt26/multilingual-instruction.html}
}