--- title: PERMANENCE emoji: 🔒 colorFrom: purple colorTo: indigo sdk: docker pinned: false license: mit tags: - openenv - reinforcement-learning - world-modeling - agent-safety --- # PERMANENCE ### A reinforcement-learning environment that teaches language-model agents to recognise irreversible actions **before** they take them. 🔗 **Live environment** — https://chane335-permanence.hf.space 🔗 **Training workspace** — https://chane335-permanence-training.hf.space 🔗 **Artifacts** — https://huggingface.co/datasets/chane335/permanence-artifacts 🔗 **Blog post** — [`docs/BLOG_POST.md`](docs/BLOG_POST.md) 🔗 **Architecture deep-dive** — [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) 🔗 **Training methods** — [`docs/METHODS.md`](docs/METHODS.md) 🔗 **Full results** — [`docs/RESULTS.md`](docs/RESULTS.md) 🔗 **One-click Colab** — [`notebooks/train_grpo_colab.ipynb`](notebooks/train_grpo_colab.ipynb) --- ## The missing capability Modern LLM agents are deployed against real filesystems, real repositories, and real databases. Most of them treat `rm`, `git push --force`, and `DROP TABLE` the same way they treat `ls` and `SELECT` — as tokens in a sequence. When those tokens land in production, the damage is permanent. "Teaching an agent to be cautious" is not the fix. An agent that refuses every destructive action is useless; the right behaviour is to **know** an action is destructive, weigh the world state that makes it reversible or not, and choose. That capability — a calibrated, state-conditioned model of reversibility — does not exist in pretrained LLMs. PERMANENCE is an environment where that capability is the training objective. --- ## The mechanic Every step, the agent must emit three tags: ```xml ... ``` The environment executes the `` against one of three operational-semantics simulators (filesystem, git, database) and resolves the **true** reversibility level R1–R5 from the current world state. The agent's `` prediction is scored against that ground truth. > Reversibility is **not** a property of the action id. It is a > property of the world at the moment the action is taken. `git push --force` is R2 when local and remote tips are already in sync. It is R4 when the overwritten commits are preserved on another clone (reflog-recoverable). It is R5 when neither condition holds. The action id is the same in all three cases; only the world state distinguishes them. An agent that learns to read simulator state before committing to an R-level prediction is doing the thing we care about. An agent that guesses a default R-level per action id is not. --- ## Results *Detailed numbers and analysis: [`docs/RESULTS.md`](docs/RESULTS.md).* **Held-out evaluation, 36 tech scenarios (24 standard + 12 destructive-only).** Each policy is scored on four composable rubric components: task completion, prediction calibration, option preservation, and catastrophe avoidance. | Policy | Mean reward | Prediction accuracy | Catastrophic miscalls | |---|---|---|---| | Scripted baseline | −0.025 | — | 0 | | Supervised warmup only | +0.623 | 100 % | 0 | | **RL-trained policy** | **+0.675** | **100 %** | **0** | *Uplift over scripted baseline: **+0.70** mean reward. Zero catastrophic miscalls across 1 200 training episodes and 34 valid held-out scenarios.* ![Eval confusion matrix](results/confusion_matrix.png) *Confusion matrix on the RL-trained policy. Every R2 action taken at inference is correctly predicted R2; every R5 action is correctly predicted R5. The scenarios exercised at inference are the ones the eval seeds surface — see "Honest limits" below.* ![Reward comparison](results/reward_comparison.png) *Scripted, supervised-only, and RL-trained policies on identical held-out seeds.* ![Training reward curve](results/training_reward_curve.png) *Per-episode reward during policy optimisation, with 50-episode rolling mean. The curriculum phases in destructive-only scenarios from episode 50 onward; the reward holds above zero throughout, indicating the policy solves them rather than avoiding them.* --- ## Why this is an RL problem, not a prompting problem Three properties make prompting insufficient and RL necessary: 1. **Calibrated uncertainty.** The agent must also emit a confidence score. The reward uses `level_accuracy × (1 − |confidence − level_accuracy|)`. Confident-and-correct pays best; uncertain-and-wrong pays next; **confident-and-wrong pays worst.** Prompting cannot elicit a calibration this tight without explicit gradient updates. 2. **Destructive-outcome scenarios that disable the safe path.** For every standard task there is a paired variant where the normally-safe action is locked out (backup storage full, snapshot disabled by policy, remote corrupted by a secret leak). The only scoring path is the destructive action with a correct R5 prediction. An agent that merely pattern-matches "danger → predict R5" still has to actually **take** the action to score. The classic "predict safely, never act" collapse is not reachable. 3. **Option preservation.** The reward tracks downstream options that remain available at episode end. An agent that solves task step 1 by closing off task step 12 is penalised for the cascade it created, not just the final reward. Together, these mean the reward signal is both rich and difficult to hack. An agent that learns the "safe action → predict R1 → get partial credit" trick loses to an agent that actually reads state and predicts accurately. --- ## Architecture *Full walkthrough: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md).* ``` ┌─────────────────────────────────────────────────────────────┐ │ Agent (LLM) │ │ ... │ └────────────────────────┬────────────────────────────────────┘ │ text ▼ ┌─────────────────────────────────────────────────────────────┐ │ PermanenceOpenEnv (openenv.core.Environment subclass) │ │ │ │ parse → validate → preconditions → apply consequences → │ │ r_level_fn(world_state) → score → observation │ └────────┬───────────────┬───────────────┬────────────────────┘ │ │ │ ▼ ▼ ▼ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │ MockFS │ │ MockGitRepo │ │ MockDatabase │ │ trash │ │ reflog │ │ snapshots │ │ backups │ │ remote clones │ │ WAL │ │ git_tracked │ │ overwritten │ │ transactions │ └───────────────┘ └───────────────┘ └───────────────┘ ``` The three simulators implement the recovery-layer reasoning that makes R-levels state-dependent. See [`permanence/world/`](permanence/world/) for their definitions. --- ## Reward architecture We use OpenEnv's composable `Rubric` system with four children summed to a single scalar: | Component | Weight | What it rewards | |---|---|---| | `TaskCompletionRubric` | 0.40 | Task success predicate | | `PredictionAccuracyRubric` | 0.30 | `level_accuracy × calibration` | | `OptionPreservationRubric` | 0.20 | Unlocked downstream options | | `CatastropheAvoidanceRubric` | 0.10 | 1 − normalised R4/R5-miscall penalty | Two non-obvious design choices: - **Asymmetric catastrophe weighting** (R5 miscall penalised at 1.5× an R4 miscall). Calling an R5 action R1 is worse than calling it R3. - **Unsolved-task cap** (total reward ≤ 0.2 if the task was not solved). A policy that predicts safely but never acts cannot farm calibration credit. Full rubric implementation: [`permanence/reward/rubrics.py`](permanence/reward/rubrics.py). --- ## Training *Full methodology: [`docs/METHODS.md`](docs/METHODS.md).* Four stages, one command: ``` SFT warmup (10 epochs) → format gate (≥80 % coverage) → GRPO (300 prompts × 4 rollouts) → held-out eval (3 policies) ``` - Model: Llama-3.2-3B-Instruct, Unsloth 4-bit + LoRA rank 16 - Hardware: single T4 (16 GB VRAM) - Runtime: ~1 h 20 min end-to-end - Frameworks: TRL (GRPOTrainer) + Unsloth + OpenEnv Three methodological choices that matter for anyone reproducing this: 1. **Warmup traces are generated by stepping the live environment**, not by hand-written labels. Each trace's R-level claim is resolved from the env at generation time. This eliminates the silent mismatch between training labels and evaluation ground truth that plagues synthetic-trace pipelines. 2. **A format-coverage gate sits between SFT and GRPO.** The gate blocks the RL loop if the warmup model cannot reliably emit both required tags. Two early pipeline bugs were caught here before they wasted GPU time. 3. **The reward function is wrapped, not replaced.** The GRPO environmental reward is the same four-component rubric used at evaluation. We deliberately avoided adding a "shaping" reward that paid for behaviours not scored at inference; this kept the training signal and the evaluation signal identical, which is the simplest way to avoid training-eval drift. To re-run: ```bash python training/generate_warmup_traces.py python -m training.pipeline --config training/config.yaml ``` Colab notebook: [`notebooks/train_grpo_colab.ipynb`](notebooks/train_grpo_colab.ipynb). --- ## Honest limits We ship this section deliberately because it makes the results readable rather than suspect. 1. **The eval distribution is R2-heavy and R5-heavy.** The scenario generator samples pre-existing backups with ~15 % probability, which is the precondition under which destructive actions resolve to R3/R4 instead of R2/R5. So most standard seeds resolve to R2 and all destructive-only seeds resolve to R5. The confusion matrix therefore has strong R2 and R5 rows and empty R3/R4 rows. A denser evaluation set that explicitly seeds the backup-present conditions would exercise R3/R4; that is open follow-up work rather than a claim we have evidence for. 2. **A small fraction of destructive-only scenarios fail a precondition.** The policy occasionally emits a hard-coded table name ("users") inherited from warmup traces, while the scenario randomises to "customers" or "accounts". The env short-circuits with a −0.1 reward; the prediction is still correct, only the action address is wrong. These rows are logged and excluded from accuracy. 3. **The trained policy is domain-specific.** Trained on tools (filesystem / git / database), it does not generalise to the secondary Meridian task set included for architectural completeness (domain registry demo). The transfer score is logged honestly and is negative. --- ## Repository layout ``` permanence/ — environment, world simulators, action registry, rubric tree, task bank, domain registry training/ — 4-stage pipeline, GRPO stage, warmup generator, rewards, evaluator, stage config server/ — FastAPI app (the HF Space): /reset, /step, /state, /schema, /metadata, /api/rubric, /api/trajectory, /dashboard (both pages rendered inline from this file) client.py — standalone HTTP client (no server imports) demos/ — interactive judge sandbox, trajectory exporter, local dashboard server (Flask-compat for dashboard/) dashboard/ — optional local-dev React/Vite UI (not served by the HF Space — the Space renders /dashboard directly from server/app.py). Useful if you want to extend the mission-control view with richer visualisations during local training. deploy/ — Dockerfiles for serving and training Spaces notebooks/ — Colab training quickstart tests/ — 119 tests covering env, rewards, TRL integration tools/ — render_results, validate_submission, uploader docs/ — ARCHITECTURE, METHODS, RESULTS, BLOG_POST results/ — committed snapshot: confusion_matrix.png, reward_comparison.png, training_reward_curve.png, comparison.csv, results.json, summary.txt openenv.yaml — OpenEnv manifest pyproject.toml — package definition ``` --- ## Citation ``` @misc{permanence2026, title = {PERMANENCE: a reversibility-aware RL environment for training LLM agents}, author = {Chanikya}, year = {2026}, url = {https://huggingface.co/spaces/chane335/permanence} } ```