File size: 3,541 Bytes
813b25e 17f9530 813b25e 17f9530 813b25e 17f9530 813b25e 17f9530 813b25e 17f9530 813b25e 17f9530 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 | ---
language:
- ta
- en
license: apache-2.0
base_model: Qwen/Qwen2.5-7B-Instruct
library_name: transformers
pipeline_tag: text-generation
tags:
- tamil
- qwen2
- qlora
- instruction-tuning
- morphology
- dravidian
datasets:
- Tamil-ai/samacheer-kalvi-tamil
model-index:
- name: Tamil-Qwen2.5-7B-Instruct
results: []
---
# Tamil-Qwen2.5-7B-Instruct
A Tamil-specialized instruction-tuned LLM built on Qwen2.5-7B-Instruct using QLoRA fine-tuning on 150K deduplicated Tamil instruction pairs.
**Paper:** *"A Thousand Language Problem: Morphological Understanding in Linguistic AI"*
## Model Details
| Property | Value |
|----------|-------|
| Base model | [Qwen/Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) |
| Parameters | 7.6B |
| Method | QLoRA (r=64, alpha=128, dropout=0.05) |
| Training data | 150K deduplicated Tamil instruction-response pairs |
| Tokenizer efficiency | 4.62x ratio (best among tested models for Tamil) |
| Compute | RunPod RTX 5090, ~$5 total cost |
| Sequence length | 1024 |
| Batch size | 32 (effective) |
| Epochs | 1 |
## Training Data
150,000 deduplicated instruction-response pairs from 5 Tamil datasets:
- Tamil Alpaca
- Tamil Orca
- Tamil Dolly
- Tamil-ai/samacheer-kalvi-tamil (morphological drills + grammar QA)
- Additional Tamil instruction sets
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Tamil-ai/tamil-qwen25-7b-instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype="auto",
device_map="auto",
)
messages = [
{"role": "system", "content": "You are a helpful Tamil language assistant."},
{"role": "user", "content": "வீடு என்ற சொல்லின் வேற்றுமை வடிவங்களைக் கூறுக."},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
### 4-bit Quantized (for limited VRAM)
```python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
model = AutoModelForCausalLM.from_pretrained(
"Tamil-ai/tamil-qwen25-7b-instruct",
quantization_config=BitsAndBytesConfig(load_in_4bit=True),
device_map="auto",
)
```
## Why Qwen2.5?
Tokenizer analysis across 6 base models showed Qwen2.5 has the best Tamil tokenization efficiency:
| Model | Tamil Token Ratio | Verdict |
|-------|------------------|---------|
| **Qwen2.5** | **4.62x** | Best for Tamil |
| Llama 3.1 | 5.8x | |
| Gemma 2 | 6.1x | |
| Mistral | 7.2x | |
| Falcon | 10.5x | Worst |
Lower ratio = fewer tokens per Tamil word = more efficient training and inference.
## Intended Use
- Tamil question answering and instruction following
- Tamil morphological analysis
- Tamil grammar and linguistics tasks
- Research on low-resource language LLMs
## Limitations
- Trained primarily on instructional Tamil; may underperform on colloquial/slang
- Morphological accuracy varies by category (see benchmark results)
- English capabilities may degrade compared to base Qwen2.5
## Citation
```bibtex
@misc{tamilai2026,
title={A Thousand Language Problem: Morphological Understanding in Linguistic AI},
author={Tamil-AI},
year={2026},
publisher={HuggingFace},
url={https://huggingface.co/Tamil-ai/tamil-qwen25-7b-instruct}
}
```
|