emilcw/gpt-sw3-1.3b-da-saga-grpo

GPT-SW3 1.3B — Danish Grammar-Aligned (SAGA GRPO)

Fine-tuned with SAGA (Syntax-Aware Grammar Alignment) using GRPO on Danish Wikipedia data with SpaCy da_core_news_lg as parser oracle.

This is a fully merged model — no PEFT/LoRA setup needed.

Part of a method comparison (GRPO vs DPO vs SDPO) at the 1.3B scale. SFT skipped (base PS 86.0% ≥ τ=0.80).

Results (Stanza DA — independent held-out evaluator)

Metric	Base	+ GRPO
Stanza PS ↑	86.0%	89.0%
Stanza score ↑	0.421	0.508
PPL-Wiki ↓	16.0	16.0

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("emilcw/gpt-sw3-1.3b-da-saga-grpo", torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained("emilcw/gpt-sw3-1.3b-da-saga-grpo")

prompt = "Dansk er"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=60, temperature=0.8, do_sample=True)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Training details

• Base model: GPT-SW3 1.3B (Nordic pretraining, base DA PS 86%)
• Method: GRPO
• Oracle: SpaCy da_core_news_lg (Danish dependency parser)
• LoRA: rank 16, α=32, all linear layers, bfloat16 (merged into full weights)

Citation

@article{fakhar2025saga,
  title={SAGA: Syntax-Aware Grammar Alignment for Low-Resource Nordic Languages},
  author={Fakhar, Hoda and others},
  year={2025},
  note={Under review}
}

License

Inherits the AI Sweden LLM License from the base model.