training_evidence/README.md

Training Evidence

Raw evaluation artifacts from every training configuration documented in docs/ABLATIONS.md. Each folder contains the results.json (per-policy summary) and comparison.csv (per-scenario row) from the held-out evaluation run for that configuration.

The numbers in ABLATIONS.md are read directly from these files.

Domain context for Meridian fields

This project's primary judged scope is the DevTools domain (filesystem, git, and database tasks). You will still see Meridian-related labels in the artifacts because the pipeline supports a secondary cross-domain transfer evaluation for architecture completeness.

• track=tech: primary DevTools evaluation track (submission focus)
• track=meridian_transfer: secondary social-domain transfer track
• n_scenarios_meridian: count of secondary transfer scenarios included in the evaluation bundle

Meridian metrics are reported for completeness, but the core claimed results in the README/RESULTS are based on the DevTools track.

---

Folder → Ablation mapping

Folder	ABLATIONS.md label	What varied
run_0_collapsed_baseline/	Pre-training baseline	Safe-action collapse before forced-variant curriculum
run_A_headline/	A (headline)	Full pipeline: forced-variant curriculum + beta_rank=0.25 + standard eval
run_B_forced_eval_track/	B	Run A adapter re-evaluated with forced-outcome eval track added
run_C_env_precondition_fix/	C	Run B with env fix: destructive DB ops on missing tables short-circuit
run_D_no_unlikeliness_shaping/	D	Disabled rank-based unlikeliness shaping (beta_rank=0.25 → 0.0)
run_E_forced_eval_no_shaping/	E	Run D adapter re-evaluated with forced-outcome eval track

---

What each file contains

results.json — per-policy summary for this run:

{
  "scripted":      { "mean_reward_tech": -0.025, "prediction_accuracy": 1.0, "catastrophe_count": 0 },
  "sft_only":      { "mean_reward_tech": +0.418, "prediction_accuracy": 1.0, "catastrophe_count": 0 },
  "grpo_trained":  { "mean_reward_tech": +0.664, "prediction_accuracy": 1.0, "catastrophe_count": 0 }
}

comparison.csv — one row per scenario × policy with columns: policy, track, task_id, seed, reward, predicted_r_level, actual_r_level

Track values:

• tech = primary DevTools scenarios
• meridian_transfer = secondary Meridian transfer scenarios

---

Run 0: the collapse (why the forced-variant curriculum exists)

run_0_collapsed_baseline/ shows what happened before the forced-outcome curriculum was introduced: the policy converged to predicting R1–R2 for every action and never took a destructive action. Mean reward plateaued near +0.4 with task completion near zero. The analysis in analysis.md documents the gradient behaviour that produced this collapse.

The forced-variant curriculum (phasing in 0 % → 50 % → 70 % destructive scenarios) was designed specifically to break this local optimum.