stukenov/sozkz-core-llama-300m-kk-gec-v4

SozKZ Core Llama 300M — Kazakh GEC v4

Казақ тіліне арналған грамматикалық қателерді түзету моделі (Grammar Error Correction).

v4 — thinking формат: модель алдымен қатені анықтайды (💭), содан кейін түзетеді (→).

Model Details

Parameter	Value
Architecture	LlamaForCausalLM (decoder-only)
Parameters	325M
Base model	stukenov/sozkz-core-llama-300m-kk-base-v1
Training data	sozkz-corpus-synthetic-kk-gec-v1
Training	5 epochs, LR=2e-5, BS=128, cosine schedule
Clean ratio	80% (error pairs + identity pairs)
Data filter	word edit distance ≤ 2 only
Hardware	4× RTX 4090 (vast.ai), ~2.8h
License	MIT (gated access)

Format

New thinking format — model first identifies the error, then corrects:

Error example:

<тег> ошибочный текст
💭 ларде→ларда (дауысты дыбыс үндесімі)
→ исправленный текст

Clean example:

<тег> дұрыс текст
💭 қате жоқ
→ дұрыс текст

Available Tags

Tag	Error Type
<грамматика>	General grammar (catch-all)
<сингармонизм>	Vowel harmony
<септік>	Case suffixes
<тәуелдік>	Possessive
<жіктік>	Personal endings
<шылау>	Postpositions
<көптік>	Plural
<болымсыз>	Negation
<шақ>	Tense
<жалғау>	General suffixes
<қате>	Typos/noise
<таза>	Clean (no error)

Usage

from transformers import AutoModelForCausalLM, GPT2TokenizerFast
from huggingface_hub import hf_hub_download
import torch

model_id = "stukenov/sozkz-core-llama-300m-kk-gec-v4"
tok_file = hf_hub_download(repo_id=model_id, filename="tokenizer.json")
tokenizer = GPT2TokenizerFast(tokenizer_file=tok_file)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16)
model.eval()

def correct(tag, text):
    prompt = f"<{tag}> {text}\n"
    inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=480)
    with torch.no_grad():
        out = model.generate(
            **inputs,
            max_new_tokens=len(inputs["input_ids"][0]) + 60,
            temperature=0.3, top_p=0.9, do_sample=True,
            repetition_penalty=1.1,
            pad_token_id=tokenizer.eos_token_id,
        )
    result = tokenizer.decode(out[0], skip_special_tokens=True)
    # Extract thinking and correction
    if "→ " in result:
        result = result.split("→ ", 1)[1]
    for stop in ["\n<", "\n\n"]:
        if stop in result:
            result = result[:result.index(stop)]
    return result.strip() or text

print(correct("көптік", "Студенттар университетте оқиды."))
# → Студенттер университетте оқиды.

Key Improvements over v3

• Thinking format: model explains what it's fixing before correcting
• 80% clean data: reduces false positives (model defaults to "no error")
• Data filtering: only pairs with word edit distance ≤ 2 (removes noisy rewrites)

Training History

Version	Epochs	Clean%	Accuracy	FP Rate
v1	1	0%	9.0%	84%
v2a	1	15%	6.6%	82%
v3	5	50%	15.8%	76%
v3+filters	—	—	21.4%	12%
v4	5	80%	TBD	TBD

Limitations

• Research model — accuracy still improving
• Best used with inference-time retry filters (see serve_gec.py)
• Occasional non-Kazakh output from base Llama

Part of SozKZ Project

This model is part of the SozKZ project for Kazakh language AI.

Benchmark Results

Evaluated on 100-example custom GEC test (pure model inference, no pre/post pipeline).

✅ Top-3 в SozKZ GEC benchmark

Category	Score
Орфография (емле)	9/30 (30%)
Грамматика	6/20 (30%)
Пунктуация	4/15 (27%)
Смешанный	3/20 (15%)
Identity preservation	15/15 (100%)
Total	37/100 (37%)

Leaderboard (100-example custom benchmark)

Модель	Total	Емле/30	Грамм/20	Пункт/15	Смеш/20	Ident/15
sozkz-core-llama-600m-kk-gec-v1	47%	15	12	3	2	15/15
sozkz-fix-qwen-500m-kk-gec-v3	38%	0	16	9	0	13/15
sozkz-core-llama-300m-kk-gec-v4	37%	9	6	4	3	15/15
sozkz-fix-qwen-500m-kk-gec-v1	35%	0	12	8	0	15/15
sozkz-fix-qwen-500m-kk-gec-v2	30%	0	11	7	0	12/15
sozkz-core-llama-1b-kk-gec-v1	16%	2	6	1	0	7/15
sozkz-fix-qwen-500m-kk-gec-v4	5%	0	1	4	0	0/15
sozkz-fix-mt5b-kk-gec-run13-v1	5%	0	2	0	0	3/15
sozkz-nllb-1b-kk-gec-v1	1%	0	1	0	0	0/15
sozkz-nllb-1b-kk-pretrain-v1	1%	0	1	0	0	0/15
sozkz-core-llama-300m-kk-gec-v3	1%	0	1	0	0	0/15
sozkz-core-llama-300m-kk-gec-v1/v2a/v2b	0–1%	0	0	0	0	0–1
sozkz-fix-mt5-50m-kk-gec-v1	0%	0	0	0	0	0/15