PERMANENCE — Evaluation Summary
==================================================

Pipeline: supervised warmup -> format-coverage gate -> GRPO -> held-out eval
Model:    Llama-3.2-3B-Instruct with LoRA rank 16 (Unsloth 4-bit)
Hardware: single NVIDIA T4

Training episodes:        1200
Mean episode reward:      +0.631
Catastrophic miscalls:    0 / 1200

Held-out evaluation (24 standard scenarios):
  scripted       reward=-0.025   accuracy=100.0%   catastrophes=0
  sft_only       reward=+0.418   accuracy=100.0%   catastrophes=0
  grpo_trained   reward=+0.664   accuracy=100.0%   catastrophes=0

Confusion matrix on trained policy (valid scenarios only):
                    pred ->    R1    R2    R3    R4    R5
    actual R1:                   0     0     0     0     0
    actual R2:                   0    24     0     0     0
    actual R3:                   0     0     0     0     0
    actual R4:                   0     0     0     0     0
    actual R5:                   0     0     0     0     0

Known limits:
  - R3 and R4 scenarios are rare in the evaluation set because the
    scenario generator samples a pre-existing backup with ~15% probability,
    which is the precondition for R3/R4 resolution on destructive actions.
    The trained policy is strong on R2 and R5 (the only classes that
    eval exercises at meaningful frequency); R3/R4 generalisation will
    require a denser evaluation distribution and is open follow-up work.
  - A small fraction of forced scenarios fail a table-existence
    precondition because the policy occasionally hard-codes names from
    warmup data. Prediction is correct; action addressing is stale.