qwen25vl-3b-modi-synth-lora

A QLoRA adapter for Modi script → Devanagari (Marathi) transliteration, fine-tuned on MoDeTrans (real handwritten Modi documents) plus SynthMoDe (synthetic Modi images rendered from the same Devanagari text using Modi fonts).

This is the recommended adapter — it achieves the best test CER (0.328) and is the default in the project's Gradio demo and CLI inference script.

Project: modi-to-Marathi on GitHub

---

Model details

Base model	Qwen/Qwen2.5-VL-3B-Instruct
Fine-tuning method	QLoRA — LoRA rank 32, alpha 64, 4-bit NF4 quantization
LoRA target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Trainable parameters	74M (1.94% of 3.8B total)
Training data	MoDeTrans (1,635 real) + SynthMoDe (4,086 synthetic) = 5,721 total
Training epochs	2
Training time	~9.75 hours on RTX 5060 (8.5 GB VRAM)
Test CER	0.328 (on 204 held-out MoDeTrans examples; real images only)
Zero-shot baseline CER	0.930

A 65% reduction in character error rate vs zero-shot. CER 0.328 means roughly 33% of characters require expert correction — designed for human-in-the-loop use.

CER distribution (204 test examples)

Quality tier	CER range	Count
Perfect	0.0	3
Excellent	< 0.10	5
Good	0.10–0.20	38
Fair	0.20–0.40	109
Poor	0.40–0.70	44
Very poor	0.70–1.0	3
Hallucination loops	> 1.0	2

The majority of examples fall in the "fair" tier — usable as a first draft for expert review.

Effect of synthetic data vs real-only adapter

Model	Test CER	Poor + worse
Real data only (lgtk/qwen25vl-3b-modi-lora)	0.332	59 examples
Real + synthetic (this adapter)	0.328	47 examples

Synthetic data gave a small but consistent improvement, mainly reducing poor-tier examples. Vowel-length confusion was unchanged (font rendering does not capture that ambiguity).

---

Task

Input: photograph or scan of a handwritten Modi-script Marathi document Output: Devanagari transliteration of that text

This is transliteration, not translation — the language (Marathi) does not change, only the script. Modi was the administrative script of Maharashtra from roughly the 13th to the mid-20th century; most surviving documents are from the Shivakalin (17th c.), Peshwekalin (18th–early 19th c.), and Anglakalin (1818–1952) eras.

---

How to use

Install dependencies:

pip install transformers peft bitsandbytes accelerate pillow torch

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor, BitsAndBytesConfig
from peft import PeftModel
from PIL import Image
import torch

MODEL_ID    = "Qwen/Qwen2.5-VL-3B-Instruct"
ADAPTER_ID  = "lgtk/qwen25vl-3b-modi-synth-lora"   # ← this repo

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)
base  = Qwen2_5_VLForConditionalGeneration.from_pretrained(
            MODEL_ID, quantization_config=bnb, device_map="auto")
model = PeftModel.from_pretrained(base, ADAPTER_ID)
model.eval()
processor = AutoProcessor.from_pretrained(MODEL_ID, max_pixels=512 * 28 * 28)

PROMPT = (
    "This image contains handwritten text in Modi script, a historical cursive "
    "script used to write the Marathi language. "
    "Transliterate the text in this image into Devanagari script. "
    "Output only the Devanagari text, with no explanation."
)

image = Image.open("your_modi_image.jpg").convert("RGB")
messages = [{
    "role": "user",
    "content": [
        {{"type": "image", "image": image}},
        {{"type": "text",  "text": PROMPT}},
    ],
}]
text_in = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs  = processor(text=[text_in], images=[image], return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=256, do_sample=False)

result = processor.batch_decode(
    out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True
)[0].strip()
print(result)

---

Known limitations

• Deletions dominate errors (46%) — the model tends to skip characters, especially the anusvāra diacritic (ं, ~240 drops per 204-example test run)
• Vowel length confusion — ी ↔ ि (long/short /i/) and ू ↔ ु (long/short /u/) are the most common substitution pairs
• No word boundaries — Modi script is continuous (no spaces), so errors sometimes span phrase boundaries
• Best on formal letters — trained on Peshwekalin / Shivakalin administrative documents; performance on informal or personal correspondence may be lower
• Image preprocessing not yet implemented — the model receives raw images. Denoising, deskewing, and binarisation would likely improve accuracy.

---

Hardware

Developed and tested on a desktop with an NVIDIA RTX 5060 GPU (8.5 GB VRAM) running WSL2 on Windows. At 4-bit NF4 quantization, the adapter uses approximately 3–4 GB of VRAM during inference. A 5060 or better is recommended; the pipeline should also fit on any GPU with ≥6 GB VRAM.

---

Citation / acknowledgements

• MoDeTrans dataset: IIT Roorkee — historyHulk/MoDeTrans on HuggingFace
• SynthMoDe dataset: IIT Roorkee — historyHulk/SynthMoDe on HuggingFace
• Base model: Qwen/Qwen2.5-VL-3B-Instruct (Apache 2.0)
• Developed by Sachin Godse (@lgtkgtv) using Claude CLI (Anthropic)