README.md

---
language:
- ta
- en
license: apache-2.0
base_model: Qwen/Qwen2.5-7B-Instruct
library_name: transformers
pipeline_tag: text-generation
tags:
- tamil
- qwen2
- qlora
- instruction-tuning
- morphology
- dravidian
datasets:
- Tamil-ai/samacheer-kalvi-tamil
model-index:
- name: Tamil-Qwen2.5-7B-Instruct
  results: []
---

# Tamil-Qwen2.5-7B-Instruct

A Tamil-specialized instruction-tuned LLM built on Qwen2.5-7B-Instruct using QLoRA fine-tuning on 150K deduplicated Tamil instruction pairs.

**Paper:** *"A Thousand Language Problem: Morphological Understanding in Linguistic AI"*

## Model Details

| Property | Value |
|----------|-------|
| Base model | [Qwen/Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) |
| Parameters | 7.6B |
| Method | QLoRA (r=64, alpha=128, dropout=0.05) |
| Training data | 150K deduplicated Tamil instruction-response pairs |
| Tokenizer efficiency | 4.62x ratio (best among tested models for Tamil) |
| Compute | RunPod RTX 5090, ~$5 total cost |
| Sequence length | 1024 |
| Batch size | 32 (effective) |
| Epochs | 1 |

## Training Data

150,000 deduplicated instruction-response pairs from 5 Tamil datasets:
- Tamil Alpaca
- Tamil Orca
- Tamil Dolly
- Tamil-ai/samacheer-kalvi-tamil (morphological drills + grammar QA)
- Additional Tamil instruction sets

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Tamil-ai/tamil-qwen25-7b-instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a helpful Tamil language assistant."},
    {"role": "user", "content": "வீடு என்ற சொல்லின் வேற்றுமை வடிவங்களைக் கூறுக."},
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### 4-bit Quantized (for limited VRAM)

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model = AutoModelForCausalLM.from_pretrained(
    "Tamil-ai/tamil-qwen25-7b-instruct",
    quantization_config=BitsAndBytesConfig(load_in_4bit=True),
    device_map="auto",
)
```

## Why Qwen2.5?

Tokenizer analysis across 6 base models showed Qwen2.5 has the best Tamil tokenization efficiency:

| Model | Tamil Token Ratio | Verdict |
|-------|------------------|---------|
| **Qwen2.5** | **4.62x** | Best for Tamil |
| Llama 3.1 | 5.8x | |
| Gemma 2 | 6.1x | |
| Mistral | 7.2x | |
| Falcon | 10.5x | Worst |

Lower ratio = fewer tokens per Tamil word = more efficient training and inference.

## Intended Use

- Tamil question answering and instruction following
- Tamil morphological analysis
- Tamil grammar and linguistics tasks
- Research on low-resource language LLMs

## Limitations

- Trained primarily on instructional Tamil; may underperform on colloquial/slang
- Morphological accuracy varies by category (see benchmark results)
- English capabilities may degrade compared to base Qwen2.5

## Citation

```bibtex
@misc{tamilai2026,
  title={A Thousand Language Problem: Morphological Understanding in Linguistic AI},
  author={Tamil-AI},
  year={2026},
  publisher={HuggingFace},
  url={https://huggingface.co/Tamil-ai/tamil-qwen25-7b-instruct}
}
```