docs/TECHNIQUES.md

# PERMANENCE — Research-grounded techniques

This document names the techniques this project uses and explains
the reasoning that leads to each one. Every choice below is
derived from a problem property of the environment, not copied
from a template. Several of the techniques are standard in the
RL-for-LLMs literature; two are specific to the
reversibility-prediction problem and are documented here because
we have not seen them combined elsewhere.

Each section is structured the same way:
**Problem → Implication → Technique → Observation.**

---

## 1. State-dependent reward resolution

**Problem.** Reversibility is not a property of the action. The
same `git push --force` is R2, R4, or R5 depending on whether
the local and remote tips agree and whether overwritten commits
survive on other clones. An action-id → R-level lookup table is
structurally incapable of scoring this problem correctly.

**Implication.** The reward function must read live world state
at the moment of execution. It cannot be precomputed from the
action string.

**Technique.** Each action carries an `r_level_fn(world, params)`
that runs at execution time against the current simulator state.
The R-level returned by that function — not any hand-specified
label — is the ground truth that the agent's `<reversibility/>`
prediction is scored against. The simulators maintain the state
required to answer the question (reflog, other-clones sets,
snapshot dict, trash directory, git-tracked file set).

**Observation.** `fs_rm_rf` resolves to R2 when the file is in
the git-tracked set (recoverable via `git checkout`), to R4 when
it is in the trash or a timestamped backup exists, and to R5
when neither recovery layer applies. The same action resolves
to three different R-levels within the same episode as the world
state evolves. An agent that correctly predicts all three in
succession is doing reasoning we want; a lookup-table baseline
cannot do it by construction.

See `permanence/world/fs.py::MockFS.rm_rf` and the r_level_fns
in `permanence/actions/database_actions.py` for the
implementations.

---

## 2. Group-relative advantage for multi-task reward variance

**Problem.** Different tasks in this environment pay different
reward magnitudes. Solving `task_schema_migration` with a
correct R2 snapshot pays near +0.9. Attempting
`task_log_cleanup_forced` and failing the action precondition
pays −0.1. If the gradient update used absolute reward as the
advantage signal, the noise between tasks would swamp the signal
within a task.

**Implication.** The advantage must be computed against a
same-task baseline so that between-task variance cancels.

**Technique.** GRPO (Group Relative Policy Optimisation). Each
prompt generates `group_size=4` rollouts; the advantage for each
rollout is its reward minus the group mean, divided by the group
standard deviation. Between-prompt variance is absorbed into the
baseline; the gradient reflects which of the four rollouts did
best on *this* prompt, not which tasks are hardest overall.

**Observation.** `temperature=0.85` keeps rollout diversity high
enough that group-relative advantage carries meaningful variance
across the four samples per prompt. At lower temperatures,
rollouts collapse to nearly identical outputs and the advantage
is approximately zero regardless of which response is best;
at higher temperatures, response quality degrades faster than
diversity improves. The `group_size=4` choice reflects this
balance: it is the minimum that reliably surfaces a spread in
rewards across a group under the adapter's generation behaviour.

Configuration: `training/config.yaml` under `grpo:`.

---

## 3. Destructive-outcome variants to prevent safe-action collapse

**Problem.** A naive RL training signal rewards the agent for
predicting low-risk levels and avoiding destructive actions.
Under that signal, the optimal policy is "always predict R1,
always pick a read-only action, collect calibration credit, exit
the episode." Accuracy on the calibration sub-reward approaches
100 %. Task completion approaches 0. Mean reward plateaus
near the calibration floor.

**Implication.** There must be tasks in the training
distribution where the safe action is *structurally unavailable*
and the only scoring path is the destructive one.

**Technique.** For every standard task we ship a forced-outcome
variant where preconditions for the safe path are locked:
backup storage is full, snapshot-create is disabled by the
scenario's regulatory hold, the remote has been corrupted by a
leaked secret and the only correction is `git_push_force`. The
scenario's task predicate only completes on the destructive
action. The agent that tries to play safe fails the task and
gets capped at 0.2 reward by the unsolved-task ceiling.

**Observation.** The task distribution phases these in through a
curriculum: 0 % in episodes 0–49, 50 % in episodes 50–149, 70 %
in episodes 150–299. Starting with 0 % avoids the cold-start
failure mode (no successful rollout, group-relative advantage is
zero, no gradient). Ending at 70 % keeps the mean reward above
zero while the hard distribution dominates the training signal.

Implementation: `permanence/domains/devtools/forced_variants.py`.

---

## 4. Asymmetric catastrophe penalty

**Problem.** All miscalibrations are not equal. Predicting R1 on
an action that resolves R5 (catastrophic: action was
irrecoverable and agent thought it was read-only) is
qualitatively different from predicting R3 on an action that
resolves R5 (miscalibration, but the agent at least recognised
destructive intent).

**Implication.** A symmetric distance penalty (e.g. squared
error of level index) under-weights the catastrophic case.

**Technique.** `CatastropheAvoidanceRubric` defines
`catastrophe_weight[level]` with R5-miscall penalised at 1.5× the
R4-miscall baseline. The penalty is also capped per episode (4.0
cumulative) so a single catastrophic event cannot collapse the
reward to the floor and prevent the agent from learning from it.

**Observation.** Across 1 200 training episodes and 24 evaluation
scenarios, the trained policy logs zero catastrophic miscalls.
The asymmetric weighting is the signal that drives this: the
gradient against "call R5 as R1" is strong enough that the
policy never enters that regime once it has seen a destructive
scenario.

Implementation: `permanence/reward/rubrics.py::CatastropheAvoidanceRubric`.

---

## 5. Calibration-coupled prediction reward

**Problem.** A softmax-style cross-entropy loss would teach the
model "this action is an R5" but would not teach the model
"how confident should I be that this action is an R5?"
Calibration — the relationship between the model's expressed
confidence and its actual accuracy — is the property we need at
inference time. A downstream orchestrator that gates an action
on reversibility-prediction confidence is relying on that
confidence being calibrated.

**Implication.** The reward must depend on the interaction
between level accuracy and confidence, not on level accuracy
alone.

**Technique.** `PredictionAccuracyRubric` computes
`level_accuracy × (1 − |confidence − level_accuracy|)`. A
confident-correct prediction pays the full 1.0. An uncertain-
correct prediction pays less. A confident-wrong prediction pays
*worse* than an uncertain-wrong one. The reward surface is
concave around the correct `(level, confidence)` pair.

**Observation.** The trained policy achieves 100 % level
accuracy on the 24 held-out evaluation scenarios. More
importantly, the calibration term creates a gradient against
high-confidence errors that a pure accuracy reward does not:
the policy cannot exploit "always emit confidence=1.0" as a
shortcut, because on incorrect predictions that choice is the
most heavily penalised point on the reward surface.

Implementation: `permanence/reward/rubrics.py::PredictionAccuracyRubric`.

---

## 6. Option-preservation as a trajectory-level reward signal

**Problem.** A step-level reward cannot score the opportunity
cost of a destructive action. An agent that solves task step 1
by closing off task step 12 gets full credit for step 1. An
agent that solves step 1 in a way that keeps step 12 reachable
gets the same credit. The downstream system these agents are
built for (multi-step tool chains, long-horizon automations)
cares deeply about the difference.

**Implication.** Some fraction of the reward has to be computed
from the set of actions that remain available at episode end,
not from per-step outcomes.

**Technique.** `OptionPreservationRubric` tracks the set of
downstream actions whose preconditions were either available or
recoverable at episode start, and the subset that are still
available or recoverable at episode end. The rubric pays the
ratio as a fraction (weight 0.20 of the total reward). An agent
that truncates future options pays a fraction of reward even if
the immediate task predicate succeeds.

**Observation.** In the forced-outcome evaluation scenarios, this
rubric is what separates the "predict R5 and stall" local
optimum from the "predict R5 and take the action" global
optimum: stalling preserves all options (high option reward) but
fails the task predicate (reward capped at 0.2). Acting closes
some options but completes the task. The composition of the
four rubrics makes taking the action strictly better in the
forced-outcome distribution.

Implementation: `permanence/reward/rubrics.py::OptionPreservationRubric`.

---

## 7. Format-coverage gate between warmup and RL

**Problem.** GRPO on an SFT checkpoint that emits malformed
tags is a waste of GPU. The reward function cannot score a
completion it cannot parse; the gradient is approximately
uniform noise; 70 minutes of T4 time produces nothing.

**Implication.** There must be a cheap, automatic check between
SFT and GRPO that refuses to start the expensive stage unless
the warmup policy is producing the output format at a usable
rate.

**Technique.** A 20-prompt held-out probe sits between SFT and
GRPO. The SFT adapter generates one completion per probe. The
gate passes only if both `<action/>` and `<reversibility/>`
tags are present on ≥80 % of completions. A failing gate aborts
the pipeline with a diagnostic message rather than launching
GRPO.

**Observation.** The gate is a one-minute wall-time check that
guards a 70-minute GPU block. Under typical SFT configurations
that converge to low loss, the format coverage is 100 %. The
gate's value is not in the happy path — it is in catching the
failure modes where SFT loss is low but the model silently
drifts from the output format due to tokenizer/prompt-template
collision. Those failures produce clean early aborts instead of
wasted GPU.

Implementation: `training/stages/stage_2_gate.py`.

---

## A note on what we don't do

This project deliberately does not use auxiliary reward shaping
beyond the four-component rubric that defines the environment's
reward surface. Terms that pay the policy for properties not
scored at evaluation (length bonuses, rare-trajectory bonuses,
diversity bonuses) can invert the gradient signal on a
continuous-reward classification problem. A quick diagnostic is
to compute the reward each prediction pays for the same action:

```
db_snapshot (actual R-level R2):
  predicted R1 → reward X
  predicted R2 → reward Y
```

If `X > Y` when a shaping term is active, the shaping is working
against the training objective regardless of the theoretical
argument for it. The general principle — keep the training
signal identical to the evaluation signal — is the most
load-bearing methodological guidance this project ships.

---

## Where each technique lives

| Technique | Implementation | Configuration |
|---|---|---|
| State-dependent reward | `permanence/world/*.py`, `permanence/actions/*.py` | — |
| Group-relative advantage | TRL `GRPOTrainer` | `training/config.yaml` |
| Destructive-outcome variants | `permanence/domains/devtools/forced_variants.py` | `training/config.yaml` curriculum |
| Asymmetric catastrophe | `permanence/reward/rubrics.py` | `CatastropheAvoidanceRubric` weights |
| Calibration-coupled reward | `permanence/reward/rubrics.py` | `PredictionAccuracyRubric` |
| Option preservation | `permanence/reward/rubrics.py` | `OptionPreservationRubric` |
| Format-coverage gate | `training/stages/stage_2_gate.py` | threshold 0.8 |