🐾 qwen3-0.6b-cat-lingo-grpo

A Qwen3-0.6B model fine-tuned in two stages:

1. SFT — supervised fine-tuning on (prompt, cat-lingo response) pairs to bootstrap cat behaviour.
2. GRPO — reinforcement learning that further strengthens cat-lingo fluency and response quality, using a lightweight LLM judge (Qwen2.5-0.5B-Instruct, 4-bit) as the quality reward signal.

Training details

Item	Value
Base model	Qwen/Qwen3-0.6B
Stage 1	SFT (TRL) + LoRA (r=32), 10 epochs
Stage 2	GRPO (TRL) + LoRA continued, 3 epochs
GRPO generations	4 per prompt
Reward	Single holistic LLM judge (cat authenticity + factual quality)
Judge	Qwen2.5-0.5B-Instruct (4-bit NF4)
β (KL penalty)	0.04
GPU budget	8 GB

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B", torch_dtype=torch.bfloat16, trust_remote_code=True
)
model = PeftModel.from_pretrained(base, "peluz/qwen3-0.6b-cat-lingo-grpo")
tokenizer = AutoTokenizer.from_pretrained("peluz/qwen3-0.6b-cat-lingo-grpo", trust_remote_code=True)

messages = [{"role": "user", "content": "Explain black holes."}]
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Limitations

Trained on a compact dataset; cat persona may occasionally drop on unusual prompts.