AbdulrahmanFA/saudi-dialect-qwen2.5-3b-lora-v1

Saudi Dialect LoRA (Qwen2.5-3B-Instruct) — v1

A LoRA adapter for Qwen/Qwen2.5-3B-Instruct, fine-tuned to nudge generation toward Saudi (Najdi-leaning) colloquial register.

Status: experimental proof-of-concept

This is a v1 trained on a single-domain corpus. Dialect transfer is partial: it shifts most on finance/market topics and still tends to fall back to MSA on general prompts. Not production-ready. Shared for experimentation and as a training-pipeline reference.

Training data

~19k short text segments scraped from a Saudi stock-market forum (hawamer.com), cleaned and filtered for dialect density (dedup, boilerplate removal, dialect-marker scoring, thread/author capping). Register skews adult, male, finance-oriented, Najdi-leaning.

Training setup

• Method: QLoRA (4-bit NF4 base + LoRA), bf16 compute
• LoRA: r=8, alpha=16, dropout=0.05, all attention + MLP projections
• 1 epoch, lr 2e-4 cosine, effective batch 16, max_len 512
• Held-out perplexity (300 samples): ~21.5

Limitations

• Single-domain (finance) data caps general-topic dialect quality
• Occasional refusals / MSA fallback inherited from the instruct base
• Rare language instability on longer generations
• Not evaluated by native-speaker dialect rating yet

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct", dtype=torch.bfloat16, device_map="cuda")
model = PeftModel.from_pretrained(base, "AbdulrahmanFA/saudi-dialect-qwen2.5-3b-lora-v1")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")