---
base_model: Qwen/Qwen3-0.6B
datasets:
- custom
language:
- en
license: apache-2.0
tags:
- qwen3
- sft
- grpo
- rlhf
- fine-tuned
- cat
- fun
- causal-lm
---

# 🐾 qwen3-0.6b-cat-lingo-grpo

A **Qwen3-0.6B** model fine-tuned in two stages:

1. **SFT** — supervised fine-tuning on (prompt, cat-lingo response) pairs to bootstrap cat behaviour.
2. **GRPO** — reinforcement learning that further strengthens cat-lingo fluency *and* response quality, using a lightweight LLM judge (Qwen2.5-0.5B-Instruct, 4-bit) as the quality reward signal.

## Training details
| Item | Value |
|---|---|
| Base model | [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) |
| Stage 1 | SFT (TRL) + LoRA (r=32), 10 epochs |
| Stage 2 | GRPO (TRL) + LoRA continued, 3 epochs |
| GRPO generations | 4 per prompt |
| Reward | Single holistic LLM judge (cat authenticity + factual quality) |
| Judge | Qwen2.5-0.5B-Instruct (4-bit NF4) |
| β (KL penalty) | 0.04 |
| GPU budget | 8 GB |

## Quick start

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B", torch_dtype=torch.bfloat16, trust_remote_code=True
)
model = PeftModel.from_pretrained(base, "peluz/qwen3-0.6b-cat-lingo-grpo")
tokenizer = AutoTokenizer.from_pretrained("peluz/qwen3-0.6b-cat-lingo-grpo", trust_remote_code=True)

messages = [{"role": "user", "content": "Explain black holes."}]
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
```

## Limitations
Trained on a compact dataset; cat persona may occasionally drop on unusual prompts.