GitMarco27/zagreus-0.4b-italic-kl

Zagreus 0.4B ITALIC KL

This model is a full BF16 fine-tune of mii-llm/zagreus-0.4B-ita for the Italian Post-Training Challenge 2026. It is trained to express the base model's answer-text probabilities through the first generated multiple-choice letter.

• Training code: https://github.com/GitMarco27/zagreus-italic-kl
• Training data and KL targets: https://huggingface.co/datasets/GitMarco27/zagreus-0.4b-italic-kl-data
• Complete evaluation: evaluation/official_results.json

Results

Evaluation	Setting	Questions	Accuracy
ITALIC official run_eval.py	zero-shot, fast=true, temperature 0	10,000	0.3381
Public domain development set	zero-shot, greedy	480	0.5813
KL held-out set	first-token letter accuracy	924 records after filtering	0.6190

The official evaluator is pinned to commit 92df420ff686babeea54e217b9f90f8471374916. Zero-shot evaluation is configured by setting few_shot_file: null in the official ITALIC configuration. This differs from the repository's five-shot default and was accepted for this challenge submission.

Method

For each answer option, the base model produces a domain-conditional cloze-PMI score:

s_i = log P(option_i | question) - log P(option_i | domain prompt)
q   = softmax(s / tau)

Two balanced option permutations are created for each question. At the first generation position, the model is trained with the following loss:

L = KL(q || softmax(z_ABCD))
  + lambda * CE(z_ABCD, gold)
  + mu * (-log P(first token is A, B, C, or D))

The final hyperparameters are tau=1, lambda=1, mu=0.1, learning rate 1e-4, batch size 16, five epochs, weight decay 0.01, and seed 42. The teacher and student both start from the mandatory base model. No external teacher checkpoint or generation API is used.

Training data

The 21,847-question source pool contains:

• 9,847 culture and civics questions from efederici/pinocchio;
• 6,000 definition questions derived from mik3ml/italian-dictionary;
• 6,000 synonym questions derived from fraug-library/synonyms_dictionnaries.

The KL build selects 5,000 training questions and 500 question-disjoint held-out questions, then creates two permutations per question. The 256-token prompt filter leaves 9,294 training records and 924 held-out records.

The source pool is deduplicated and checked against all ITALIC questions with exact normalized matching, MinHash LSH, and RapidFuzz confirmation. The final audit found no question overlap. ITALIC labels are not used for training.

The dataset repository records the original terms for each source. The Hugging Face card for the synonym mirror does not declare a license, so users should review its upstream terms before redistributing derived data.

Usage

This checkpoint is specialized for four-option multiple-choice questions. The following example uses the same prompt style as the official fast evaluation:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "GitMarco27/zagreus-0.4b-italic-kl"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

messages = [
    {"role": "system", "content": "Sei un assistente utile."},
    {
        "role": "user",
        "content": (
            "Rispondi alla seguente domanda a scelta multipla. "
            "Scrivi solo la lettera corretta.\n\n"
            "Qual è la capitale d'Italia?\n\n"
            "A) Milano\nB) Roma\nC) Torino\nD) Napoli\n\nRisposta:"
        ),
    },
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)
output = model.generate(**inputs, do_sample=False, max_new_tokens=1)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Final run

The final run used one NVIDIA A100 80GB PCIe GPU. The allocation lasted 0.7531 hours, including setup, target generation, training, evaluation, transfer, and verification. Based on the recorded provider price ratio, this equals 0.3622 H100-equivalent hours.

The immutable base revision is f463a5beabf054809ee3cc702c711215a5fde690. The model weights have SHA-256 5e0284467f727f22000b897a3c40f3e5c24b519de463f71a4212149fe1913c6e.

Limitations

This is a benchmark-specific model, not a general instruction model. Training directly supervises the first generated token, so later tokens may repeat. For normal multiple-choice inference, use greedy decoding with max_new_tokens=1.

The model has about 438 million parameters. Its training data covers lexical, cultural, and civic questions better than syntax, orthography, and morphology.

Development tools

Development of this project relied on AI assistants: Claude Fable 5, GPT-5.6-sol, Claude Opus, and Qwen3.6 27B. They supported code writing, experiment analysis, and documentation. The assistants were not used to generate training data: all KL targets were produced with the mandatory base model itself, as described above, and no external API was used for synthetic generation.

References

• Seveso et al. (2025), ITALIC: An Italian Culture-Aware Natural Language Benchmark.
• Holtzman et al. (2021), Surface Form Competition: Why the Highest Probability Answer Isn't Always Right.
• Hinton, Vinyals, and Dean (2015), Distilling the Knowledge in a Neural Network.