---
language:
- tr
- en
license: apache-2.0
tags:
- llama
- causal-lm
- gqa
- slm
- base-model
- causal-lm
- pre-trained
- Llama
- Türkçe
- Turkish
- tr-llm
- Ahıska
- AhiskaTurks
- MeskhetianTurks
- AhıskaTürkleri
datasets:
- BILGEM-AI/BILGE-Wiki-Tr-Plus
- BILGEM-AI/BILGE-Synthetic-Stories
- bigcode/the-stack-v2-dedup
- HuggingFaceFW/fineweb

pipeline_tag: text-generation
---

# AhıskaAI-135M-Base-v0.3

**AhıskaAI-135M-Base-v0.3** is a ~135 million parameter Small Language Model (SLM) trained completely from scratch. Building upon the learnings of the v0.2 release, v0.3 introduces architectural optimizations (such as Grouped-Query Attention), an expanded and multi-domain pre-training corpus, and enhanced training efficiency using Liger Kernel.

## Key Improvements from v0.2 to v0.3

- **Architectural Evolution (GQA Support):** Transitioned to **Grouped-Query Attention (GQA)** with 12 query heads and 4 key-value heads (`num_key_value_heads: 4`). This significantly reduces KV-cache memory overhead and boosts inference throughput compared to standard MHA in v0.2.
- **Diversified Pre-training Mixture:** Expanded beyond single-source corpora to a multi-domain dataset totaling ~2.04 GB across Turkish Wikipedia (%53.2), Turkish Stories (%21.6), Python Code (%17.6), and English FineWeb (%7.6).
- **Training Acceleration & Efficiency:** Utilized **Liger Kernel** integration (`apply_liger_kernel_to_llama`) and `adamw_torch_fused` optimization in `bfloat16` precision, maximizing hardware utilization.
- **Hardware & Scale:** Scaled pre-training across dual Kaggle NVIDIA T4 GPUs for 11 hours, overcoming consumer laptop VRAM limitations while preserving tight parameter footprints.

## Model Details

- **Architecture:** `LlamaForCausalLM` with GQA
- **Parameters:** ~135M
- **Hidden Size:** 768
- **Intermediate Size:** 3072
- **Attention Heads:** 12 (Query) / 4 (Key/Value)
- **Layers:** 14
- **Context Length:** 512 tokens
- **Vocabulary Size:** 32,000
- **Precision:** `float16`

## Pre-training Dataset Distribution

The model was pre-trained on a balanced multi-domain dataset (~2.04 GB total):

| Data Source | Share (%) | Size | Primary Domain |
| :--- | :--- | :--- | :--- |
| **BILGEM AI Turkish Wikipedia** | 53.2% | 1.09 GB | General Knowledge & Turkish Grammar | https://huggingface.co/datasets/BILGEM-AI/BILGE-Wiki-Tr-Plus
| **BİLGEM Turkish Stories** | 21.6% | 443 MB | Narrative & Fluency                       | https://huggingface.co/datasets/BILGEM-AI/BILGE-Synthetic-Stories
| **Python Code (The Stack)** | 17.6% | 360 MB | Reasoning & Logic                        | https://huggingface.co/datasets/bigcode/the-stack-v2-dedup
| **FineWeb English** | 7.6% | 150 MB | Cross-lingual & General Alignment                 | https://huggingface.co/datasets/HuggingFaceFW/fineweb

## Training Hardware & Setup

- **Hardware:** 2x NVIDIA T4 GPUs (Kaggle)
- **Training Time:** ~11 Hours
- **Optimization:** SDPA (Scaled Dot-Product Attention), AdamW Fused, Liger Kernel
- **Precision:** `float16`

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AhiskaAI/AhiskaAI-135M-Base-v0.3"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16,
    device_map="auto"
)

prompt = "Türkiye Cumhuriyeti'nin başkenti"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=100,
    temperature=0.7,
    top_p=0.9,
    do_sample=True
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### Related Models
- **Instruct Version:** AhiskaAI/AhiskaAI-135M-IT-v0.3 (SFT ChatML model fine-tuned for instruction following)

### About AhıskaAI
**AhıskaAI** is an independent initiative dedicated to developing efficient, high-performance Small Language Models (SLMs) tailored for the Turkish language ecosystem.