Spaces:
Sleeping
Sleeping
File size: 3,033 Bytes
f6afd9e f684b80 f6afd9e f684b80 f6afd9e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | # Training Evidence
Raw evaluation artifacts from every training configuration documented in
[`docs/ABLATIONS.md`](../docs/ABLATIONS.md). Each folder contains the
`results.json` (per-policy summary) and `comparison.csv` (per-scenario
row) from the held-out evaluation run for that configuration.
The numbers in ABLATIONS.md are read directly from these files.
## Domain context for Meridian fields
This project's primary judged scope is the **DevTools domain** (filesystem,
git, and database tasks). You will still see Meridian-related labels in the
artifacts because the pipeline supports a secondary cross-domain transfer
evaluation for architecture completeness.
- `track=tech`: primary DevTools evaluation track (submission focus)
- `track=meridian_transfer`: secondary social-domain transfer track
- `n_scenarios_meridian`: count of secondary transfer scenarios included in the
evaluation bundle
Meridian metrics are reported for completeness, but the core claimed results in
the README/RESULTS are based on the DevTools track.
---
## Folder → Ablation mapping
| Folder | ABLATIONS.md label | What varied |
|---|---|---|
| `run_0_collapsed_baseline/` | Pre-training baseline | Safe-action collapse before forced-variant curriculum |
| `run_A_headline/` | **A (headline)** | Full pipeline: forced-variant curriculum + beta_rank=0.25 + standard eval |
| `run_B_forced_eval_track/` | B | Run A adapter re-evaluated with forced-outcome eval track added |
| `run_C_env_precondition_fix/` | C | Run B with env fix: destructive DB ops on missing tables short-circuit |
| `run_D_no_unlikeliness_shaping/` | D | Disabled rank-based unlikeliness shaping (beta_rank=0.25 → 0.0) |
| `run_E_forced_eval_no_shaping/` | E | Run D adapter re-evaluated with forced-outcome eval track |
---
## What each file contains
**`results.json`** — per-policy summary for this run:
```json
{
"scripted": { "mean_reward_tech": -0.025, "prediction_accuracy": 1.0, "catastrophe_count": 0 },
"sft_only": { "mean_reward_tech": +0.418, "prediction_accuracy": 1.0, "catastrophe_count": 0 },
"grpo_trained": { "mean_reward_tech": +0.664, "prediction_accuracy": 1.0, "catastrophe_count": 0 }
}
```
**`comparison.csv`** — one row per scenario × policy with columns:
`policy, track, task_id, seed, reward, predicted_r_level, actual_r_level`
Track values:
- `tech` = primary DevTools scenarios
- `meridian_transfer` = secondary Meridian transfer scenarios
---
## Run 0: the collapse (why the forced-variant curriculum exists)
`run_0_collapsed_baseline/` shows what happened before the forced-outcome
curriculum was introduced: the policy converged to predicting R1–R2 for
every action and never took a destructive action. Mean reward plateaued
near +0.4 with task completion near zero. The analysis in `analysis.md`
documents the gradient behaviour that produced this collapse.
The forced-variant curriculum (phasing in 0 % → 50 % → 70 % destructive
scenarios) was designed specifically to break this local optimum.
|