AegisGuard-RL-stable

The validation-selected, CI-hardened variant of the AegisGuard RL guardrail (Qwen3-4B). Policy-adaptive, pre-effect per-step judging: reads a runtime custom policy + agent trajectory → first-token verdict safe/unsafe → P(unsafe) for pre-effect gating.

This is the honest-science endpoint of the direction-B RL line. The original AegisGuard-RL was a single lucky seed; this model is selected on a held-out validation split and reported on an untouched test split, after a full ablation exposed that GRPO on small target-distribution data is high-variance (see below).

Why "stable": the honest story

A 3-seed CI showed GRPO on 502 R-Judge points is unstable: exec accuracy 0.758 ± 0.185 across seeds (2/3 survive, 1/3 collapse); plain SFT on the same data collapses 3/3. The R-Judge gain itself comes from adding R-Judge data (SFT matches it); GRPO's unique value is KL-anchored resistance to catastrophic forgetting — but only probabilistically.

Stabilization = capability replay (exec data as an anti-forgetting anchor) + 5 seeds + validation-set selection (pick the best non-collapsed seed on a held-out val, never peeking at test). This reliably yields a good model even though not every seed is stable.

Training (2 stages from base)

1. Combined SFT (base Qwen3-4B-Instruct-2507, full-param, 2 ep, lr 3e-6): policy_train(counterfactual pairs, 1150) + policy_p4(applicability/conditional, 1260) + policy_p6(conflict hierarchy, 880) + policy_adversarial(injection, 496) + policy_exec(executable rules, 1400) = 5186 samples → p3-agp-sft.
2. GRPO (warm p3-agp-sft, TRL, β=0.1, lr=7e-7, num_gen 8, 1 ep): rjstable_train_mix = R-Judge-train (430, 3-judge consensus labels) + exec replay (430, anti-forgetting anchor) = 860. Verifiable symmetric verdict reward (first-token invariant baked in, no learned reward model). 5 seeds → validation-selected (seed 3).

Data: AegisGuard-Train / AegisGuard-Eval.

Evaluation (untouched test split, never used for selection)

Metric	value
R-Judge test net-discrimination (187 pts, content-deduped)	91pt @th0.7
exec verdict acc / continuous AUC	0.903 / 0.947
calibration ECE	0.258
over-flag / miss	0.077 / 0.117
real-policy prose (main / conflict)	0.836 / 0.917

Validation-selection generalized cleanly to the untouched test → no test-set overfitting.

Usage

from openai import OpenAI  # vLLM OpenAI-compatible server, --dtype bfloat16
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
prompt = build_policy_prompt(policy=RUNTIME_POLICY, trajectory=AGENT_TRAJECTORY)
r = client.chat.completions.create(model="aegisguard-rl-stable",
        messages=[{"role":"user","content":prompt}], temperature=0, max_tokens=8, logprobs=True, top_logprobs=20)
# First token is safe/unsafe; P(unsafe) from the top-20 first-token logprobs → GuardedExecutor pre-effect gating.

Limitations

• The training process is high-variance (a re-run may collapse); reliability comes from val-selection, not from any single run being stable. For production, keep the val-selection + untouched-test protocol.
• Verdict is grounded in the provided runtime policy; not a substitute for full safety review. English agent trajectories.
• Cross-domain deployment: fit a small isotonic calibrator on target-domain scores (P(unsafe) ranks well but shifts across domains).