Training Space README frontmatter

README.md

@@ -1,5 +1,5 @@
 ---
-title: PERMANENCE
+title: PERMANENCE Training
 emoji: 🔒
 colorFrom: purple
 colorTo: indigo
@@ -9,328 +9,10 @@ license: mit
 tags:
   - openenv
   - reinforcement-learning
-  - world-modeling
-  - agent-safety
+suggested_hardware: t4-small
 ---
 
-# PERMANENCE
+# PERMANENCE Training Space
 
-### A reinforcement-learning environment that teaches language-model agents to recognise irreversible actions **before** they take them.
-
-> **Solo submission** by **[Chanikya](https://huggingface.co/chane35)** — Meta PyTorch Hackathon.
-> One engineer · three simulators · full end-to-end training pipeline on a single T4.
-
-## Quick Links (Judge-Facing)
-
-> Start here first. These are the primary assets used in judging.
-
-- **LIVE ENVIRONMENT (SPACE):** https://chane35-permanence.hf.space
-- **TRAINING WORKSPACE (SPACE):** https://chane35-permanence-training.hf.space
-- **PRESENTATION (SLIDES):** https://docs.google.com/presentation/d/1_LTsvg_hFyQW6-EUNJjW17yBcN3Fy0mGJVyRMfUk-eg/edit?usp=sharing
-- **ARTIFACTS DATASET (DOWNLOADABLE):** https://huggingface.co/datasets/chane35/permanence-artifacts
-- **BLOG POST:** [`Blog.md`](Blog.md)
-- **ARCHITECTURE DEEP-DIVE:** [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md)
-- **TECHNIQUES / DESIGN RATIONALE:** [`docs/TECHNIQUES.md`](docs/TECHNIQUES.md)
-- **TRAINING METHODS:** [`docs/METHODS.md`](docs/METHODS.md)
-- **FULL RESULTS:** [`docs/RESULTS.md`](docs/RESULTS.md)
-- **RAW TRAINING EVIDENCE:** https://huggingface.co/spaces/chane35/permanence/tree/main/training_evidence (eval artifacts from all 5 ablation runs)
-- **ONE-CLICK COLAB:** [`notebooks/train_grpo_colab.ipynb`](notebooks/train_grpo_colab.ipynb)
-
-> **Domain scope note:** This submission is focused on the **DevTools domain** (filesystem/git/database tasks).
-> You may still see **Meridian** in logs/tables (for example in ablation artifacts); Meridian is a **secondary social-drama domain kept for architecture completeness**, not the primary judged focus.
-
----
-
-## The missing capability
-
-Modern LLM agents are deployed against real filesystems, real
-repositories, and real databases. Most of them treat `rm`,
-`git push --force`, and `DROP TABLE` the same way they treat `ls`
-and `SELECT` — as tokens in a sequence. When those tokens land in
-production, the damage is permanent.
-
-"Teaching an agent to be cautious" is not the fix. An agent that
-refuses every destructive action is useless; the right behaviour is
-to **know** an action is destructive, weigh the world state that
-makes it reversible or not, and choose. That capability — a
-calibrated, state-conditioned model of reversibility — does not
-exist in pretrained LLMs.
-
-PERMANENCE is an environment where that capability is the training
-objective.
-
----
-
-## The mechanic
-
-Every step, the agent must emit three tags:
-
-```xml
-<thinking>...</thinking>
-<action id="db_drop_table" name="users"/>
-<reversibility level="R5" confidence="0.93"/>
-```
-
-The environment executes the `<action/>` against one of three
-operational-semantics simulators (filesystem, git, database) and
-resolves the **true** reversibility level R1–R5 from the current
-world state. The agent's `<reversibility/>` prediction is scored
-against that ground truth.
-
-> Reversibility is **not** a property of the action id. It is a
-> property of the world at the moment the action is taken.
-
-`git push --force` is R2 when local and remote tips are already in
-sync. It is R4 when the overwritten commits are preserved on another
-clone (reflog-recoverable). It is R5 when neither condition holds.
-The action id is the same in all three cases; only the world state
-distinguishes them.
-
-An agent that learns to read simulator state before committing to an
-R-level prediction is doing the thing we care about. An agent that
-guesses a default R-level per action id is not.
-
----
-
-## Results
-
-*Detailed numbers and analysis: [`docs/RESULTS.md`](docs/RESULTS.md).*
-
-**Held-out evaluation, 24 held-out tech scenarios.** Each policy is scored on four composable
-rubric components: task completion, prediction calibration, option
-preservation, and catastrophe avoidance.
-
-| Policy | Mean reward | Prediction accuracy | Catastrophic miscalls |
-|---|---|---|---|
-| Scripted baseline | −0.025 | — | 0 |
-| Supervised warmup only | +0.418 | 100 % | 0 |
-| **RL-trained policy** | **+0.664** | **100 %** | **0** |
-
-*Uplift over scripted baseline: **+0.69** mean reward. Zero
-catastrophic miscalls across 1 200 training episodes and 24 valid
-held-out scenarios.*
-
-*Full ablation across five configurations, including runs with different unlikeliness-shaping settings and forced-outcome eval tracks, is in [`docs/ABLATIONS.md`](docs/ABLATIONS.md). Raw eval artifacts (`results.json` + `comparison.csv`) for every run are in [training_evidence](https://huggingface.co/spaces/chane35/permanence/tree/main/training_evidence). Training log (1 200 episodes) is in [`results/training_log.json`](results/training_log.json).*
-
-![Eval confusion matrix](results/confusion_matrix.png)
-
-*Confusion matrix on the RL-trained policy. Every R2 action taken
-at inference is correctly predicted R2. The scenarios exercised at
-inference are the ones the eval seeds surface — see "Honest limits" below.*
-
-![Reward comparison](results/reward_comparison.png)
-
-*Scripted, supervised-only, and RL-trained policies on identical
-held-out seeds.*
-
-![Training reward curve](results/training_reward_curve.png)
-
-*Per-episode reward during policy optimisation, with 50-episode
-rolling mean. The curriculum phases in destructive-only scenarios
-from episode 50 onward; the reward holds above zero throughout,
-indicating the policy solves them rather than avoiding them.*
-
----
-
-## Why this is an RL problem, not a prompting problem
-
-Three properties make prompting insufficient and RL necessary:
-
-1. **Calibrated uncertainty.** The agent must also emit a
-   confidence score. The reward uses
-   `level_accuracy × (1 − |confidence − level_accuracy|)`.
-   Confident-and-correct pays best; uncertain-and-wrong pays next;
-   **confident-and-wrong pays worst.** Prompting cannot elicit a
-   calibration this tight without explicit gradient updates.
-
-2. **Destructive-outcome scenarios that disable the safe path.**
-   For every standard task there is a paired variant where the
-   normally-safe action is locked out (backup storage full,
-   snapshot disabled by policy, remote corrupted by a secret leak).
-   The only scoring path is the destructive action with a correct
-   R5 prediction. An agent that merely pattern-matches "danger →
-   predict R5" still has to actually **take** the action to score.
-   The classic "predict safely, never act" collapse is not reachable.
-
-3. **Option preservation.** The reward tracks downstream options
-   that remain available at episode end. An agent that solves task
-   step 1 by closing off task step 12 is penalised for the cascade
-   it created, not just the final reward.
-
-Together, these mean the reward signal is both rich and
-difficult to hack. An agent that learns the "safe action →
-predict R1 → get partial credit" trick loses to an agent that
-actually reads state and predicts accurately.
-
-The reasoning that arrives at each of the environment's core design
-choices — state-resolved rewards, group-relative advantage,
-destructive-outcome variants, asymmetric catastrophe weighting,
-calibration-coupled rewards, option preservation, and the format
-gate — is documented in [`docs/TECHNIQUES.md`](docs/TECHNIQUES.md).
-Each technique is derived from a specific property of the
-reversibility-prediction problem rather than imported as a
-template.
-
----
-
-## Architecture
-
-*Full walkthrough: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md).*
-
-![Reversibility is world-state, not action-id](assets/arch_reversibility_state.jpeg)
-
-*The same `git_push_force` call resolves to R2, R4, or R5 depending on
-`MockGitRepo` world state at execution time — decided by `r_level_fn`, not
-by the action string. The three simulators (MockFS, MockGitRepo, MockDatabase)
-each implement real recovery-layer semantics so the R-level reflects actual
-recoverability. See [`permanence/world/`](permanence/world/) for the implementations.*
-
----
-
-## Reward architecture
-
-We use OpenEnv's composable `Rubric` system with four children
-summed to a single scalar:
-
-![Reward tree with exploit closures](assets/arch_reward_tree.jpeg)
-
-*Each leaf rubric targets a distinct failure mode. The unsolved-task cap
-closes the "predict safely, never act" exploit. The asymmetric catastrophe
-penalty closes the "always predict R1, collect calibration credit" exploit.*
-
-| Component | Weight | What it rewards |
-|---|---|---|
-| `TaskCompletionRubric` | 0.40 | Task success predicate |
-| `PredictionAccuracyRubric` | 0.30 | `level_accuracy × calibration` |
-| `OptionPreservationRubric` | 0.20 | Unlocked downstream options |
-| `CatastropheAvoidanceRubric` | 0.10 | 1 − normalised R4/R5-miscall penalty |
-
-Two non-obvious design choices:
-
-- **Asymmetric catastrophe weighting** (R5 miscall penalised at 1.5× an
-  R4 miscall). Calling an R5 action R1 is worse than calling it R3.
-- **Unsolved-task cap** (total reward ≤ 0.2 if the task was not
-  solved). A policy that predicts safely but never acts cannot
-  farm calibration credit.
-
-Full rubric implementation: [`permanence/reward/rubrics.py`](permanence/reward/rubrics.py).
-
----
-
-## Training
-
-*Full methodology: [`docs/METHODS.md`](docs/METHODS.md).*
-
-Four stages, one command:
-
-![Four-stage pipeline with fail-fast gates](assets/arch_training_pipeline.jpeg)
-
-*The format-coverage gate sits between SFT and GRPO. If the warmup model
-cannot reliably emit both required tags, the gate aborts before spending
-70 minutes of T4 GPU time on a broken RL loop.*
-
-- Model: Llama-3.2-3B-Instruct, Unsloth 4-bit + LoRA rank 16
-- Hardware: single T4 (16 GB VRAM)
-- Runtime: ~1 h 20 min end-to-end
-- Frameworks: TRL (GRPOTrainer) + Unsloth + OpenEnv
-
-Three methodological choices that matter for anyone reproducing
-this:
-
-1. **Warmup traces are generated by stepping the live environment**,
-   not by hand-written labels. Each trace's R-level claim is
-   resolved from the env at generation time. This eliminates the
-   silent mismatch between training labels and evaluation ground
-   truth that plagues synthetic-trace pipelines.
-2. **A format-coverage gate sits between SFT and GRPO.** The gate
-   blocks the RL loop if the warmup model cannot reliably emit both
-   required tags. Two early pipeline bugs were caught here before
-   they wasted GPU time.
-3. **The reward function is wrapped, not replaced.** The GRPO
-   environmental reward is the same four-component rubric used at
-   evaluation. We deliberately avoided adding a "shaping" reward
-   that paid for behaviours not scored at inference; this kept the
-   training signal and the evaluation signal identical, which is
-   the simplest way to avoid training-eval drift.
-
-To re-run:
-
-```bash
-python training/generate_warmup_traces.py
-python -m training.pipeline --config training/config.yaml
-```
-
-Colab notebook: [`notebooks/train_grpo_colab.ipynb`](notebooks/train_grpo_colab.ipynb).
-
----
-
-## Honest limits
-
-We ship this section deliberately because it makes the results
-readable rather than suspect.
-
-1. **The headline eval exercises R2 only.** The standard 24-scenario
-   eval seeds almost always resolve to R2 (safe-path-available outcomes).
-   Adding the forced-outcome eval track (scenarios where the safe path
-   is locked out) populates R4 and R5 rows in the confusion matrix — see
-   Run B in [`docs/ABLATIONS.md`](docs/ABLATIONS.md) for broadest coverage.
-   R3/R4 generalisation under standard seeding requires a denser
-   evaluation distribution and is open follow-up work.
-2. **A small fraction of destructive-only scenarios fail a
-   precondition.** The policy occasionally emits a hard-coded
-   table name ("users") inherited from warmup traces, while the
-   scenario randomises to "customers" or "accounts". The env
-   short-circuits with a −0.1 reward; the prediction is still
-   correct, only the action address is wrong. These rows are
-   logged and excluded from accuracy.
-3. **The trained policy is domain-specific.** Trained on tools
-   (filesystem / git / database), it does not generalise to the
-   secondary Meridian task set included for architectural
-   completeness (domain registry demo). The transfer score is
-   logged honestly and is negative.
-
----
-
-## Repository layout
-
-```
-permanence/        — environment, world simulators, action registry,
-                     rubric tree, task bank, domain registry
-training/          — 4-stage pipeline, GRPO stage, warmup generator,
-                     rewards, evaluator, stage config
-server/            — FastAPI app (the HF Space): /reset, /step, /state,
-                     /schema, /metadata, /api/rubric, /api/trajectory,
-                     /dashboard (both pages rendered inline from this file)
-client.py          — standalone HTTP client (no server imports)
-demos/             — interactive judge sandbox, trajectory exporter,
-                     local dashboard server (Flask-compat for dashboard/)
-dashboard/         — optional local-dev React/Vite UI (not served by
-                     the HF Space — the Space renders /dashboard
-                     directly from server/app.py). Useful if you want
-                     to extend the mission-control view with
-                     richer visualisations during local training.
-deploy/            — Dockerfiles for serving and training Spaces
-notebooks/         — Colab training quickstart
-tests/             — 119 tests covering env, rewards, TRL integration
-tools/             — render_results, validate_submission, uploader
-docs/              — ARCHITECTURE, METHODS, RESULTS, BLOG_POST
-results/           — committed snapshot: confusion_matrix.png,
-                     reward_comparison.png, training_reward_curve.png,
-                     comparison.csv, results.json, summary.txt
-openenv.yaml       — OpenEnv manifest
-pyproject.toml     — package definition
-```
-
----
-
-## Citation
-
-```
-@misc{permanence2026,
-  title  = {PERMANENCE: a reversibility-aware RL environment
-            for training LLM agents},
-  author = {Chanikya},
-  year   = {2026},
-   url    = {https://huggingface.co/spaces/chane35/permanence}
-}
-```
+This Space runs GRPO training for the PERMANENCE environment on T4 GPU.
+After training completes, it serves the environment API on port 7860.