training_evidence/run_0_collapsed_baseline/analysis.md

Run 1 — Collapsed

Date: 2026-04-25 Model: unsloth/Llama-3.2-1B-Instruct-bnb-4bit Episodes completed: 204 / 500 Duration: ~31 minutes Outcome: Training collapsed around episode 180, reward trajectory went negative.

Reward Trajectory

Phase	Episodes	Reward	Behaviour
Cold start	1-40	-0.06 to -0.01	Random output, misses format
Format learning	40-100	-0.01 to +0.015	SFT transfer kicks in, finds format pattern
Plateau	100-130	+0.015	Stuck at format bonus ceiling, can't reach tasks
Drift	130-180	0 to -0.025	Longer outputs, rambling begins
Collapse	180-204	-0.025 to -0.05	75% of completions hit 256-token cap, clipped

Root Causes (5)

1. max_completion_length=256 too tight — model hit cap, parser saw truncated tags, returned -0.1 penalty.
2. Flat format reward — 0.1 if has_action and has_rev else 0.0 gave no gradient once model crossed the threshold.
3. Plateau at +0.015 — format bonus × weight = local maximum, task rewards unreachable from here.
4. Default temperature — exploration triggered rambling that never terminated cleanly.
5. Effective batch 4 × num_gen 4 = 16 forward passes per step on T4 → 12s/it, budget-heavy.

Fixes Applied for Run 2

• max_completion_length=320 (more room before cap)
• Tiered format reward (0.05 + 0.05 + 0.05, max 0.15) with length penalty
• temperature=0.6 explicitly set in GRPOConfig
• per_device_train_batch_size=num_generations to avoid Unsloth auto-upgrade
• group_size=2 for faster iteration
• total_episodes=300 (fits budget, still shows curves)
• format_reward_cutoff=300 (active whole run, not just first 300)

Files Saved

• raw_logs.txt — full SSE log from HF Space
• metrics.csv — parsed per-episode metrics
• metrics.json — same in JSON format
• analysis.md — this file