# Training Evidence

Raw evaluation artifacts from every training configuration documented in
[`docs/ABLATIONS.md`](../docs/ABLATIONS.md). Each folder contains the
`results.json` (per-policy summary) and `comparison.csv` (per-scenario
row) from the held-out evaluation run for that configuration.

The numbers in ABLATIONS.md are read directly from these files.

## Domain context for Meridian fields

This project's primary judged scope is the **DevTools domain** (filesystem,
git, and database tasks). You will still see Meridian-related labels in the
artifacts because the pipeline supports a secondary cross-domain transfer
evaluation for architecture completeness.

- `track=tech`: primary DevTools evaluation track (submission focus)
- `track=meridian_transfer`: secondary social-domain transfer track
- `n_scenarios_meridian`: count of secondary transfer scenarios included in the
  evaluation bundle

Meridian metrics are reported for completeness, but the core claimed results in
the README/RESULTS are based on the DevTools track.

---

## Folder → Ablation mapping

| Folder | ABLATIONS.md label | What varied |
|---|---|---|
| `run_0_collapsed_baseline/` | Pre-training baseline | Safe-action collapse before forced-variant curriculum |
| `run_A_headline/` | **A (headline)** | Full pipeline: forced-variant curriculum + beta_rank=0.25 + standard eval |
| `run_B_forced_eval_track/` | B | Run A adapter re-evaluated with forced-outcome eval track added |
| `run_C_env_precondition_fix/` | C | Run B with env fix: destructive DB ops on missing tables short-circuit |
| `run_D_no_unlikeliness_shaping/` | D | Disabled rank-based unlikeliness shaping (beta_rank=0.25 → 0.0) |
| `run_E_forced_eval_no_shaping/` | E | Run D adapter re-evaluated with forced-outcome eval track |

---

## What each file contains

**`results.json`** — per-policy summary for this run:
```json
{
  "scripted":      { "mean_reward_tech": -0.025, "prediction_accuracy": 1.0, "catastrophe_count": 0 },
  "sft_only":      { "mean_reward_tech": +0.418, "prediction_accuracy": 1.0, "catastrophe_count": 0 },
  "grpo_trained":  { "mean_reward_tech": +0.664, "prediction_accuracy": 1.0, "catastrophe_count": 0 }
}
```

**`comparison.csv`** — one row per scenario × policy with columns:
`policy, track, task_id, seed, reward, predicted_r_level, actual_r_level`

Track values:
- `tech` = primary DevTools scenarios
- `meridian_transfer` = secondary Meridian transfer scenarios

---

## Run 0: the collapse (why the forced-variant curriculum exists)

`run_0_collapsed_baseline/` shows what happened before the forced-outcome
curriculum was introduced: the policy converged to predicting R1–R2 for
every action and never took a destructive action. Mean reward plateaued
near +0.4 with task completion near zero. The analysis in `analysis.md`
documents the gradient behaviour that produced this collapse.

The forced-variant curriculum (phasing in 0 % → 50 % → 70 % destructive
scenarios) was designed specifically to break this local optimum.