# Run 1 — Collapsed

**Date:** 2026-04-25
**Model:** unsloth/Llama-3.2-1B-Instruct-bnb-4bit
**Episodes completed:** 204 / 500
**Duration:** ~31 minutes
**Outcome:** Training collapsed around episode 180, reward trajectory went negative.

## Reward Trajectory

| Phase | Episodes | Reward | Behaviour |
|---|---|---|---|
| Cold start | 1-40 | -0.06 to -0.01 | Random output, misses format |
| Format learning | 40-100 | -0.01 to +0.015 | SFT transfer kicks in, finds format pattern |
| Plateau | 100-130 | +0.015 | Stuck at format bonus ceiling, can't reach tasks |
| Drift | 130-180 | 0 to -0.025 | Longer outputs, rambling begins |
| Collapse | 180-204 | -0.025 to -0.05 | 75% of completions hit 256-token cap, clipped |

## Root Causes (5)

1. **max_completion_length=256 too tight** — model hit cap, parser saw truncated tags, returned -0.1 penalty.
2. **Flat format reward** — `0.1 if has_action and has_rev else 0.0` gave no gradient once model crossed the threshold.
3. **Plateau at +0.015** — format bonus × weight = local maximum, task rewards unreachable from here.
4. **Default temperature** — exploration triggered rambling that never terminated cleanly.
5. **Effective batch 4 × num_gen 4 = 16 forward passes per step** on T4 → 12s/it, budget-heavy.

## Fixes Applied for Run 2

- `max_completion_length=320` (more room before cap)
- Tiered format reward (0.05 + 0.05 + 0.05, max 0.15) with length penalty
- `temperature=0.6` explicitly set in GRPOConfig
- `per_device_train_batch_size=num_generations` to avoid Unsloth auto-upgrade
- `group_size=2` for faster iteration
- `total_episodes=300` (fits budget, still shows curves)
- `format_reward_cutoff=300` (active whole run, not just first 300)

## Files Saved

- `raw_logs.txt` — full SSE log from HF Space
- `metrics.csv` — parsed per-episode metrics
- `metrics.json` — same in JSON format
- `analysis.md` — this file