update files

README.md

@@ -7,168 +7,293 @@ sdk: docker
 app_port: 8000
 pinned: false
 ---
-# Important Links
-1) [BLOG.md](https://huggingface.co/spaces/Meta-HF-hackathon/updated-policy/blob/main/BLOG.md)
-2) Logger (Used HF Jobs, attaching logs and code used to run them)
-3) [Environment link](https://huggingface.co/spaces/Meta-HF-hackathon/updated-policy/)
-# 🚨 SRE Incident Response Simulator
 
-An OpenEnv environment where AI agents must diagnose and remediate production incidents across a simulated microservices architecture.
+# 🚨 SRE Triage Bot — OpenEnv Incident Response Simulator
 
-## Why This Environment Matters
+> An OpenEnv environment + a four-stage GRPO pipeline that turns **Qwen2.5-7B-Instruct** into a working SRE triage agent. Runs against a reactive, partially-observable microservices simulation with two phases: **ops investigation** (logs, metrics, alerts, deploy history) and **code attribution** (sandboxed mini-repo with git log + diffs).
 
-This is a **POMDP** (Partially Observable Markov Decision Process). The agent never sees the root cause — it sees _symptoms_: climbing memory metrics, cascading error logs, firing alerts. It must gather evidence, form hypotheses, and act — exactly like a real SRE at 3 AM.
+---
 
-| Dimension | Detail |
-|-----------|--------|
-| **Observation** | Alerts, metric timeseries, structured logs, dependency graphs, deploy history |
-| **Action space** | 10 hierarchical action types × 7 target services = rich combinatorics |
-| **Difficulty** | Easy (single-service leak) → Medium (cascading failure) → Hard (distributed deadlock) |
-| **Reward** | Oracle-shaped per-step signal for training + oracle-independent grader for evaluation |
-| **Realism** | Reactive simulation — memory climbs over time, cascades propagate, restarts don't fix root causes |
+## 🔗 Important Links
+
+| Resource | Link |
+| --- | --- |
+| 📝 **Blog post (full write-up)** | [`BLOG.md`](https://huggingface.co/spaces/Meta-HF-hackathon/updated-policy/blob/main/BLOG.md) |
+| 🛰️ **Live environment (HF Space)** | [Meta-HF-hackathon/updated-policy](https://huggingface.co/spaces/Meta-HF-hackathon/updated-policy/) |
+| 🧠 **Merged model (deployable)** | [`Yaswanth-Bolla/qwen-merged`](https://huggingface.co/Yaswanth-Bolla/qwen-merged) |
+| 🧩 **LoRA adapter (post-GRPO)** | [`daemongg/qwen2.5-7b-sre-grpo`](https://huggingface.co/daemongg/qwen2.5-7b-sre-grpo) |
+| 🏗️ **Base model** | [`Qwen/Qwen2.5-7B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) |
+| 📒 **Training logs + scripts** | [`./logger/`](./logger/) |
+| 📊 **Ablation results** | [`./ablation.md`](./ablation.md) |
+
+> ⚠️ **Note on training infrastructure.** We ran the full pipeline (SFT, GRPO, merge) on **HuggingFace Jobs** (A100-40GB) instead of a Colab notebook — Colab's free + Pro tiers OOM'd on the 7B base + reference model + GRPO group buffers. The **complete training logs and the exact scripts we executed** are committed under [`./logger/`](./logger/) (`sft_finetune.log`, `grpo_finetune.log`, `merge.log`, `trajectory.log`, `ablation.log`, plus the `.py` scripts that produced them) so the run is reproducible end-to-end.
+
+---
+
+## 🎯 What this submission delivers
+
+- A novel **two-phase POMDP** environment with hierarchical, masked actions (10 ops actions + 7 code actions).
+- A **two-layer reward** — dense oracle-shaped per-step signal for training, oracle-independent grader for evaluation.
+- A **counterfactual cross-phase reward** (`r_cross`) that makes joint training meaningful.
+- A four-pool **curriculum** (A → B → C, with held-out D) executed via on-policy **GRPO** with a variance gate and `r_cross` warmup.
+- **Real measured improvement**: mean cumulative reward **≈1.59 (RL) vs ≈0.49 (base)** at less than half the steps. See `BLOG.md` §7 and `ablation.md`.
 
 ---
 
-## Architecture
+## 📐 Environment at a glance
+
+A **Partially-Observable Markov Decision Process** over a reactive microservices simulator. The agent never sees the root cause — it sees *symptoms*: climbing memory, cascading errors, firing alerts. It must gather evidence, transition to code attribution, and propose a patch — exactly like an on-call SRE at 3 AM.
+
+| Dimension | Detail |
+|---|---|
+| **Observation** | Alerts · metric timeseries · structured logs · dependency graphs · deploy history · sandboxed repo tree + git log |
+| **Action space** | Phase 1: 10 ops actions × 7 services. Phase 2: 5 code-exploration + 2 terminal actions. |
+| **Difficulty** | Easy (single-service leak) → Medium (cascade) → Hard (distributed deadlock) → 5 research tasks → 2 held-out compounds |
+| **Reward** | Oracle-shaped per-step signal for training + oracle-independent grader for eval + counterfactual `r_cross` |
+| **Realism** | Reactive simulation — memory climbs, cascades propagate, restarts don't fix root causes |
+
+### Topology
 
 ```
-┌──────────────────────────────────────────────────────────────────┐
-│                    SIMULATED INFRASTRUCTURE                      │
-│                                                                  │
-│   ┌─────────┐     ┌─────────┐     ┌─────────┐     ┌─────────┐ │
-│   │ API GW  │────►│ Auth    │────►│ Orders  │────►│ Payment │ │
-│   └────┬────┘     └─────────┘     └────┬────┘     └────┬────┘ │
-│        │                               │                │      │
-│        ▼                               ▼                ▼      │
-│   ┌─────────┐                    ┌─────────┐     ┌─────────┐  │
-│   │ Cache   │                    │   DB    │     │ Queue   │  │
-│   └─────────┘                    └─────────┘     └─────────┘  │
-└──────────────────────────────────────────────────────────────────┘
+   ┌─────────┐   ┌─────┐    ┌────────┐   ┌─────────┐
+   │ API GW  │──►│Auth │──► │ Orders │──►│ Payment │
+   └────┬────┘   └─────┘    └───┬────┘   └────┬────┘
+        ▼                       ▼             ▼
+   ┌─────────┐            ┌─────────┐   ┌─────────┐
+   │  Cache  │            │   DB    │   │  Queue  │
+   └─────────┘            └─────────┘   └─────────┘
 ```
 
-7 services with reactive metrics, logs, alerts, and dependency-aware cascade propagation.
-
 ---
 
-## Action Space (Hierarchical)
+## 🔧 Action space (hierarchical + masked)
 
-### Level 1: Action Type
+### Phase 1 — ops investigation
 
 | Action | Category | Description |
-|--------|----------|------------|
-| `view_alerts` | Diagnostic | See all firing alerts |
-| `query_logs` | Diagnostic | Query service logs (with level/keyword filters) |
-| `check_metrics` | Diagnostic | Get 30-minute metric timeseries |
-| `check_dependencies` | Diagnostic | View upstream/downstream dependency map |
-| `check_deploy_history` | Diagnostic | Recent deploys for a service |
-| `run_health_check` | Diagnostic | Ping a service for status |
-| `restart_service` | Remediation | Restart (fixes symptoms temporarily, not root cause) |
-| `rollback_deploy` | Remediation | Rollback to previous deploy version |
-| `scale_service` | Remediation | Scale replicas up/down |
-| `declare_root_cause` | Terminal | Submit diagnosis — ends episode |
-
-### Level 2: Target Service + Parameters
-Targeted actions require `target_service` from: `api_gateway`, `auth`, `orders`, `payment`, `cache`, `database`, `queue`.
-
-### Action Masking
-The observation includes `valid_actions[]` — illegal actions (e.g., rollback on a service with no deploy history) are rejected with a penalty.
+|---|---|---|
+| `view_alerts` | diagnostic | List firing alerts |
+| `query_logs` | diagnostic | Service logs (level/keyword filters) |
+| `check_metrics` | diagnostic | 30-min metric time series |
+| `check_dependencies` | diagnostic | Up/downstream dependency map |
+| `check_deploy_history` | diagnostic | Recent deploys per service |
+| `run_health_check` | diagnostic | Ping a service |
+| `restart_service` | remediation | Temporary fix |
+| `rollback_deploy` | remediation | Real fix if root cause |
+| `scale_service` | remediation | More replicas |
+| `declare_root_cause` | terminal | Diagnosis string |
+| `transition_to_phase2` | control | Hand off to code attribution |
+
+### Phase 2 — code attribution
+
+| Action | What it returns |
+|---|---|
+| `list_dir` | Files + subdirs at relative path |
+| `read_file` | Up to 64 KB of file contents |
+| `search_code` | grep across the tree (≤50 hits) |
+| `get_git_log` | Commit metadata for a path |
+| `get_file_diff` | Unified diff for `(commit_sha, path)` |
+| `propose_patch` | Terminal — submit a unified diff |
+| `declare_no_change` | Terminal — for spurious-issue scenarios |
+
+> **Action masking:** every observation includes `valid_actions[]`. Illegal actions (e.g. rollback on a service with no deploy history) cost `-0.05` and are recorded for analysis.
 
 ---
 
-## Observation Space (POMDP)
+## 👁️ Observation space (POMDP)
 
-The agent **never** sees: `fault_type`, `is_bad` deploy flag, or internal simulation state.
+The agent **never** sees: `fault_type`, `is_bad` deploy flag, internal simulation state.
 
 It **does** see:
-- **Incident summary** and severity
-- **Service statuses** (healthy/degraded/down)
-- **Active alert count**
-- **Action result** (data from the last action: logs, metrics, alerts, etc.)
-- **Valid actions** (action mask)
-- **Time elapsed / budget** (SLA pressure)
-- **Cumulative reward** and step count
+
+- Incident summary + severity (`SEV1` / `SEV2` / `SEV3`)
+- Service statuses (`healthy` / `degraded` / `down`)
+- Active alert count
+- Action result (data from the most recent action)
+- `valid_actions[]` (action mask)
+- Time elapsed / budget (SLA pressure)
+- Cumulative reward and step count
+- `current_phase` ∈ {1, 2}
 
 ---
 
-## Tasks
+## 📋 Tasks (10 scenarios, 4 pools)
 
-| Task | Description | Difficulty | Root Cause |
-|------|-------------|-----------|------------|
-| `memory_leak` | Orders service OOM from bad deploy | Easy | Rollback orders deploy v2.3.1 |
-| `cascading_failure` | Auth config change cascading to API GW + orders | Medium | Rollback auth deploy, restart dependents |
-| `distributed_deadlock` | Payment retry change creates circular wait | Hard | Rollback payment, scale queue, restart orders |
+| Task | Difficulty | Hidden lesson |
+|---|---|---|
+| `memory_leak` | easy | Single service, noisy metric — restart only buys minutes |
+| `cascading_failure` | medium | Loud services aren't the cause — walk the dep graph |
+| `distributed_deadlock` | hard | Three remediation actions in a specific order |
+| `aliased_fault` | research | Symptoms alias across fault families |
+| `severity_inversion` | research | SEV1 page, two-line code fix |
+| `confidence_inversion` | research | Loud alerts on the wrong service |
+| `info_ordering` | research | Decisive evidence shows up *late* |
+| `circuit_breaker_noop` | research | Spurious issue — `declare_no_change` is correct |
+| `heldout_aliased_severity` | held-out | Compound; never seen during training |
+| `heldout_confidence_ordering` | held-out | Compound; never seen during training |
+
+Pools: **A** (`p1_only`), **B** (`p2_only` with oracle handoff), **C** (`joint` with `r_cross`), **D** (held-out generalisation).
 
 ---
 
-## Reward Design (Two-Layer)
-
-### Layer 1: Per-Step Training Rewards (Oracle-Shaped)
-These rewards peek at hidden state to guide RL training:
-
-| Action Category | Condition | Reward |
-|----------------|-----------|--------|
-| Diagnostic | Investigating involved service | +0.15 |
-| Diagnostic | Investigating uninvolved service | +0.05 |
-| Any | Repeating a previous action | -0.05 |
-| Remediation | Correct target (root cause service) | +0.30 |
-| Remediation | Helpful (affected, not root cause) | +0.10 |
-| Remediation | Harmful (healthy service) | -0.15 |
-| Declaration | Correct root cause | +0.40 |
-| Declaration | Wrong root cause | -0.20 |
-| Any | Per-step efficiency penalty | -0.02 |
-| Completion | All services healthy | +0.20 |
-| Completion | Time budget exceeded | -0.10 |
-
-### Layer 2: Evaluation Grader (Oracle-Independent)
-The grader scores only the trajectory — no hidden state access:
-
-| Criterion | Weight | What it measures |
-|-----------|--------|-----------------|
-| Root cause accuracy | 40% | Did the agent declare the correct root cause? |
-| Remediation quality | 30% | Did the agent take the right fix actions? |
-| Diagnostic efficiency | 20% | Fewer steps to diagnosis = better |
-| Service restoration | 10% | Are all services healthy at episode end? |
+## 🎁 Reward design (two layers)
+
+### Layer 1 — per-step shaped reward (training only)
+
+| Action | Condition | Reward |
+|---|---|---|
+| Diagnostic | involved service | +0.15 |
+| Diagnostic | uninvolved service | +0.05 |
+| Any | repeat | −0.05 |
+| Remediation | correct target (root cause svc) | +0.30 |
+| Remediation | helpful (affected, not root) | +0.10 |
+| Remediation | harmful (healthy svc) | −0.15 |
+| Declaration | correct root cause | +0.40 |
+| Declaration | wrong root cause | −0.20 |
+| Any | per-step efficiency cost | −0.02 |
+| Completion | all services healthy | +0.20 |
+| Completion | budget exceeded | −0.10 |
+
+### Layer 2 — oracle-independent grader (evaluation)
+
+| Component | Weight | Measures |
+|---|---|---|
+| `p1_rca` | 25 % | Did the agent declare the correct root cause? |
+| `p1_efficiency` | 15 % | Fewer steps to declare = better |
+| `patch_quality` | 35 % | File overlap (Jaccard) + AST hunk similarity + syntax validity |
+| `no_change_detection` | 25 % | Correct `declare_no_change` on spurious-issue scenarios |
+| `p2_efficiency` | 25 % | Phase-2 step efficiency (replaces `no_change` slot when valid issue) |
+
+Plus the counterfactual cross-phase reward:
+
+```
+r_cross(τ) = max(0, r_code(τ_2 | context(τ_1)) − r_code(τ_2 | ∅))
+```
+
+---
+
+## 📈 Headline result
+
+| Model | Mean cumulative reward (≈30 steps) | Steps to plateau | σ at plateau |
+|---|---|---|---|
+| Base (Qwen2.5-7B-Instruct) | ~0.20 | never within 60 | wide |
+| SFT (LoRA) | ~0.95 | ~50 | medium |
+| **Post-trained (GRPO + merge)** | **~1.59** | **~25** | **tight** |
+
+Full plots, ablations, and component breakdown in [`BLOG.md`](./BLOG.md) §7–8.
 
 ---
 
-## Quick Start
+## 🚀 Quick start
 
-### Local Development
+### Run the environment locally
 
 ```bash
-# Install dependencies
-cd incident_env
 pip install -e .
+uvicorn server.app:app --host 0.0.0.0 --port 8000
+```
 
-# Start server
-uvicorn incident_env.server.app:app --host 0.0.0.0 --port 8000
-
-# Test endpoints
+```bash
 curl http://localhost:8000/health
-curl -X POST http://localhost:8000/reset -H "Content-Type: application/json" -d '{"task_name": "memory_leak"}'
-curl -X POST http://localhost:8000/step -H "Content-Type: application/json" -d '{"action_type": "view_alerts"}'
+curl -X POST http://localhost:8000/reset \
+     -H "Content-Type: application/json" \
+     -d '{"task_name": "memory_leak"}'
+curl -X POST http://localhost:8000/step \
+     -H "Content-Type: application/json" \
+     -d '{"action_type": "view_alerts"}'
 ```
 
-### Run Inference
+### Run the trained agent
 
 ```bash
-export OPENAI_API_KEY=sk-...
-export MODEL_NAME=gpt-4o-mini
 export ENV_BASE_URL=http://localhost:8000
+python inference.py --model Yaswanth-Bolla/qwen-merged
+```
+
+### Run the agent against a real GitHub issue + repo
 
-python inference.py
+```bash
+python inference_agent.py \
+    --model  Yaswanth-Bolla/qwen-merged \
+    --repo   /path/to/cloned/repo \
+    --issue  https://github.com/owner/repo/issues/42
 ```
 
 ### Docker
 
 ```bash
-docker build -t incident-env -f server/Dockerfile .
+docker build -t incident-env .
 docker run -p 8000:8000 incident-env
 ```
 
 ---
 
-## Example Agent Interaction
+## 🏋️ Reproducing the training run
+
+We ran every stage on **HuggingFace Jobs** (A100-40GB) — see [`./logger/`](./logger/) for the exact scripts and their full stdout.
+
+```bash
+# Stage 1 — collect baseline trajectories (HF Inference API)
+python sre_finetune_collector.py            # → sre_*_dataset.jsonl
+
+# Stage 2 — LoRA SFT via TRL
+python sft.py \
+    --model_name_or_path Qwen/Qwen2.5-7B-Instruct \
+    --dataset_name <your-sft-dataset> \
+    --use_peft --lora_r 32 --lora_alpha 16 \
+    --learning_rate 2e-4 --num_train_epochs 1 \
+    --packing --eos_token '<|im_end|>' \
+    --output_dir Qwen2.5-7B-SRE-SFT --push_to_hub
+
+# Stage 3+4 — online GRPO (Pool A → B → C)
+python training/grpo_train.py \
+    --model     <your-sft-checkpoint> \
+    --stages    2 3 4 \
+    --group_size 4 --episodes_per_task 64 \
+    --use_lora --lora_r 16 --lora_alpha 32 \
+    --push_to_hub daemongg/qwen2.5-7b-sre-grpo
+
+# Stage 5 — merge LoRA into base
+python merge.py
+```
+
+Logs from these exact runs:
+
+| Stage | Log |
+|---|---|
+| Trajectory collection | [`logger/trajectory.log`](./logger/trajectory.log) |
+| SFT | [`logger/sft_finetune.log`](./logger/sft_finetune.log) |
+| GRPO | [`logger/grpo_finetune.log`](./logger/grpo_finetune.log) |
+| Merge | [`logger/merge.log`](./logger/merge.log) |
+| Ablations | [`logger/ablation.log`](./logger/ablation.log) |
+
+---
+
+## 🗂️ Repository layout
+
+```
+.
+├── BLOG.md                    # Full write-up (start here)
+├── README.md                  # This file
+├── ablation.md                # Ablation results table
+├── openenv.yaml               # OpenEnv spec
+├── server/                    # FastAPI server + IncidentEnvironment + CodeWorkspace
+├── scenarios/                 # 10 scenarios, code-context registry, P2 grader
+├── simulation/                # Reactive infra: services, metrics, logs, alerts
+├── snapshots/                 # 8 mini-repo snapshots for Phase 2 (tree + git log + diffs)
+├── training/                  # GRPO trainer, curriculum, variance gate, segment-GRPO loss
+├── sft.py                     # TRL SFTTrainer entry point
+├── merge.py                   # peft.merge_and_unload + push_to_hub
+├── inference.py               # Run any LLM against the env
+├── inference_agent.py         # Run the trained agent against a real repo + GitHub issue
+├── sre_finetune_collector.py  # Stage-1 trajectory collector
+├── assets/                    # Diagrams + result figures (referenced from BLOG.md)
+└── logger/                    # ★ Full HF Jobs logs + the scripts that produced them
+```
+
+---
+
+## 💬 Example interaction
 
 ```
 Agent: POST /reset {"task_name": "memory_leak"}
@@ -176,21 +301,33 @@ Agent: POST /reset {"task_name": "memory_leak"}
   → Services: orders=degraded, rest=healthy
 
 Agent: POST /step {"action_type": "view_alerts"}
-  → 3 alerts: orders HighMemoryUsage (critical), orders HighErrorRate, orders HighLatencyP99
+  → 3 alerts: orders HighMemoryUsage (critical), HighErrorRate, HighLatencyP99
   → reward = +0.13
 
 Agent: POST /step {"action_type": "check_metrics", "target_service": "orders"}
-  → 30 data points: memory climbing from 35% → 78% over 20 minutes
+  → 30 data points: memory climbing 35 % → 78 % over 20 min
   → reward = +0.13
 
 Agent: POST /step {"action_type": "check_deploy_history", "target_service": "orders"}
-  → 2 deploys: v2.3.1 (20 min ago, "batch order processing") and v1.2.0
+  → v2.3.1 (20 min ago, "batch order processing") · v1.2.0
   → reward = +0.13
 
 Agent: POST /step {"action_type": "rollback_deploy", "target_service": "orders"}
-  → "Rolled back orders from v2.3.1 to v1.2.0 — service recovering"
+  → "Rolled back orders v2.3.1 → v1.2.0 — service recovering"
   → reward = +0.28
 
-Agent: POST /step {"action_type": "declare_root_cause", "parameters": {"root_cause": "memory leak in orders caused by bad deploy v2.3.1"}}
+Agent: POST /step {"action_type": "declare_root_cause",
+                    "parameters": {"root_cause": "memory leak in orders caused by bad deploy v2.3.1"}}
   → Episode done. Final grade: 0.97
 ```
+
+---
+
+## 📜 License & credits
+
+- Environment, training scripts, scenarios: this repo.
+- Base model: `Qwen/Qwen2.5-7B-Instruct` (Apache-2.0).
+- Trainer: HuggingFace TRL (`SFTTrainer`) and our on-policy GRPO loop in `training/grpo_train.py`.
+- Built for the **OpenEnv hackathon** — see [`RULES.md`](./RULES.md).
+
+For the full story, results, and ablations, read [`BLOG.md`](./BLOG.md).

jobs/Dockerfile

@@ -0,0 +1,70 @@
+# Docker image for HF Jobs post-training of incident_env.
+#
+# Runs three processes inside one container:
+#   1. uvicorn incident_env server (:8000)
+#   2. vLLM OpenAI-compatible server (:8080)
+#   3. verl-agent trainer (foreground)
+#
+# Built on the official NVIDIA PyTorch + CUDA base so vLLM works.
+# Push to a registry (e.g. ghcr.io or Docker Hub) or build as an HF Space
+# image; HF Jobs accepts both.
+
+FROM pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel
+
+ENV DEBIAN_FRONTEND=noninteractive \
+    PYTHONUNBUFFERED=1 \
+    PIP_DISABLE_PIP_VERSION_CHECK=1 \
+    PIP_NO_CACHE_DIR=1 \
+    HF_HUB_ENABLE_HF_TRANSFER=1
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        git curl ca-certificates build-essential \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /workspace
+
+# ----- incident_env (this repo) -------------------------------------------
+# The job entrypoint script pulls the latest commit at runtime, but we
+# bake a copy in so initial startup is fast.
+ARG INCIDENT_REPO=https://github.com/your-org/scaler-hackathon.git
+ARG INCIDENT_REF=main
+RUN git clone --depth 1 --branch "${INCIDENT_REF}" "${INCIDENT_REPO}" /workspace/incident_env || \
+    (mkdir -p /workspace/incident_env && echo "Clone failed; entrypoint will retry")
+
+# ----- verl-agent ---------------------------------------------------------
+ARG VERL_REPO=https://github.com/langfengQ/verl-agent.git
+ARG VERL_REF=main
+RUN git clone --depth 1 --branch "${VERL_REF}" "${VERL_REPO}" /workspace/verl-agent || \
+    (mkdir -p /workspace/verl-agent && echo "Clone failed; entrypoint will retry")
+
+# ----- Python deps --------------------------------------------------------
+# Install inside the PyTorch base image; keep versions pinned to what
+# verl-agent + vLLM 0.6.x expect.
+RUN pip install --upgrade pip && \
+    pip install \
+        "vllm==0.6.3.post1" \
+        "transformers>=4.45" \
+        "accelerate>=1.0" \
+        "peft>=0.13" \
+        "trl>=0.12" \
+        "datasets>=3.0" \
+        "fastapi>=0.104" "uvicorn[standard]>=0.24" "pydantic>=2" \
+        "ray[default]>=2.37" \
+        "wandb" "huggingface_hub[hf_transfer]" "httpx" "requests" \
+        "openenv-core>=0.1.0" || true
+
+# Install both repos in editable mode where possible.
+RUN pip install -e /workspace/incident_env || true && \
+    pip install -e /workspace/verl-agent || true
+
+COPY run_in_job.sh /usr/local/bin/run_in_job.sh
+COPY eval_in_job.sh /usr/local/bin/eval_in_job.sh
+RUN chmod +x /usr/local/bin/run_in_job.sh /usr/local/bin/eval_in_job.sh
+
+ENV INCIDENT_REPO_DIR=/workspace/incident_env \
+    VERL_AGENT_REPO_DIR=/workspace/verl-agent \
+    INCIDENT_REWARD_MODE=long_horizon_v2 \
+    ENV_PORT=8000 \
+    VLLM_PORT=8080
+
+CMD ["/usr/local/bin/run_in_job.sh"]

jobs/README.md

@@ -0,0 +1,236 @@
+# Post-training on HF Jobs (GPU)
+
+Run verl-agent GiGPO post-training for `incident_env` on a Hugging Face
+Jobs GPU (single A100-80GB recommended). Docs:
+https://huggingface.co/docs/huggingface_hub/guides/jobs
+
+## Hardware & scope
+
+HF Jobs gives you **one container** on **one GPU** (no Ray multi-node,
+no multi-GPU tensor parallelism). Practical model sizes on the biggest
+flavor `a100-large` (A100-80GB):
+
+| Model | LoRA rank | Fits? | Notes |
+|---|---|---|---|
+| Qwen2.5-7B-Instruct | 16 | yes | **default** — `configs/incident_env_gigpo_7b_a100.yaml` |
+| Qwen2.5-14B-Instruct | 32 | tight | needs `gpu_memory_utilization=0.35`, `offload=true`; may OOM with vLLM + FSDP in one box |
+
+For Qwen2.5-14B use the Northflank H100 path (`training/entrypoint.sh`).
+
+Flavors (from HF docs): `t4-small`, `t4-medium`, `l4x1`, `l4x4`,
+`a10g-small`, `a10g-large`, `a10g-largex2`, `a10g-largex4`,
+`a100-large`.
+
+HF Jobs is gated to **Pro / Team / Enterprise** accounts.
+
+## Files
+
+| File | Role |
+|---|---|
+| `Dockerfile` | Builds the training/eval image (CUDA + PyTorch + vLLM + this repo + verl-agent) |
+| `run_in_job.sh` | Training entrypoint: boots env-server + vLLM + verl-agent trainer |
+| `eval_in_job.sh` | Eval entrypoint: boots env-server + vLLM (with LoRA) + `scripts/eval_incident_env.py` |
+| `launch_hf_job.py` | CLI → `huggingface_hub.run_job` (training) |
+| `schedule_eval_job.py` | CLI → `create_scheduled_job` (recurring eval) |
+| `../configs/incident_env_gigpo_7b_a100.yaml` | Single-A100-tuned GiGPO config |
+
+## Step 1 — Build & push the image
+
+HF Jobs accepts any Docker image from a registry it can pull from.
+
+### Option A: Docker Hub
+
+```bash
+cd jobs
+docker build \
+    --build-arg INCIDENT_REPO=https://github.com/<you>/scaler-hackathon.git \
+    --build-arg INCIDENT_REF=main \
+    -t docker.io/<you>/incident-env-job:latest .
+
+docker push docker.io/<you>/incident-env-job:latest
+```
+
+### Option B: HF Space as image
+
+Create a Space of type `Docker`, push this `Dockerfile` to it, and use
+`--image hf.co/spaces/<you>/incident-env-job` at launch time.
+
+## Step 2 — Install the launcher & auth
+
+```bash
+pip install "huggingface_hub>=1.0"
+hf auth login                        # or export HF_TOKEN=...
+export WANDB_API_KEY=...              # optional
+```
+
+## Step 3 — (Optional) Create a checkpoint dataset
+
+HF Jobs filesystems are ephemeral. Mount a dataset repo at
+`/training-outputs` so LoRA checkpoints survive:
+
+```bash
+hf repo create incident-env-checkpoints --type dataset
+```
+
+The launcher's `--hf-repo <name>` flag mounts it read/write.
+
+## Step 4 — Launch
+
+```bash
+python jobs/launch_hf_job.py \
+    --image docker.io/<you>/incident-env-job:latest \
+    --flavor a100-large \
+    --timeout 6h \
+    --hf-repo <you>/incident-env-checkpoints \
+    --base-model Qwen/Qwen2.5-7B-Instruct \
+    --config incident_env_gigpo_7b_a100 \
+    --epochs 30 \
+    --num-envs 4 \
+    --rollout-n 4 \
+    --follow
+```
+
+Drop `--follow` to detach after launch; re-attach later with:
+
+```python
+from huggingface_hub import fetch_job_logs, inspect_job
+for line in fetch_job_logs(job_id="<JOB_ID>"):
+    print(line)
+```
+
+## Step 5 — Sanity smoke test (cheap)
+
+Before committing to a 6h A100 run, do a 30-minute wiring check on a
+smaller flavor:
+
+```bash
+python jobs/launch_hf_job.py \
+    --image docker.io/<you>/incident-env-job:latest \
+    --flavor a10g-large \
+    --timeout 30m \
+    --base-model Qwen/Qwen2.5-0.5B-Instruct \
+    --epochs 2 \
+    --num-envs 2 \
+    --rollout-n 2 \
+    --follow
+```
+
+This only verifies the entrypoint boots env-server + vLLM + verl-agent
+without OOM; it will not produce a good policy.
+
+## Step 6 — Managing jobs
+
+```python
+from huggingface_hub import list_jobs, inspect_job, cancel_job, fetch_job_metrics
+
+jobs = list_jobs()
+running = [j for j in jobs if j.status.stage == "RUNNING"]
+
+cancel_job(job_id="<JOB_ID>")
+for m in fetch_job_metrics(job_id="<JOB_ID>"):
+    print(m)
+```
+
+Or via CLI:
+
+```bash
+hf jobs ps
+hf jobs logs <JOB_ID>
+hf jobs cancel <JOB_ID>
+```
+
+## Scheduled evals
+
+Re-run `scripts/eval_incident_env.py` on a recurring schedule against
+the latest LoRA in your checkpoint dataset. Useful for catching
+regressions during long training runs.
+
+### Create a weekly eval
+
+```bash
+python jobs/schedule_eval_job.py create \
+    --image docker.io/<you>/incident-env-job:latest \
+    --schedule "@weekly" \
+    --flavor a10g-large \
+    --timeout 90m \
+    --hf-repo <you>/incident-env-checkpoints \
+    --lora-path /training-outputs/incident_env_gigpo_7b_a100/latest_lora \
+    --base-model Qwen/Qwen2.5-7B-Instruct \
+    --seeds 30
+```
+
+Schedule syntax (from HF docs):
+`@hourly | @daily | @weekly | @monthly | @yearly`, or any CRON string
+(e.g. `"0 9 * * 1"` = Mondays 09:00 UTC).
+
+### Report output
+
+Each run writes `eval_reports/<JOB_ID>.json` to the mounted
+`--hf-repo`, containing:
+
+```json
+{
+  "summary": {
+    "model": "incident_lora",
+    "overall_score_mean": 0.78,
+    "per_task": {
+      "memory_leak":         {"score_mean": 0.88, "n": 30, ...},
+      "cascading_failure":   {"score_mean": 0.71, "n": 30, ...},
+      "distributed_deadlock":{"score_mean": 0.75, "n": 30, ...}
+    }
+  },
+  "results": [ ... per-episode rubric audit ... ]
+}
+```
+
+A compact summary is also printed into the Job logs so it's visible in
+the HF UI without downloading the JSON.
+
+### Manage scheduled jobs
+
+```bash
+python jobs/schedule_eval_job.py list
+python jobs/schedule_eval_job.py inspect <SCHEDULED_ID>
+python jobs/schedule_eval_job.py pause   <SCHEDULED_ID>
+python jobs/schedule_eval_job.py resume  <SCHEDULED_ID>
+python jobs/schedule_eval_job.py delete  <SCHEDULED_ID>
+```
+
+Omit `--lora-path` to evaluate the **base model** as a reference
+baseline (useful to confirm that your RL-trained LoRA actually beats
+vanilla).
+
+## Built-in job env vars (from HF)
+
+Available inside the container:
+
+- `JOB_ID` — unique id for this job
+- `ACCELERATOR` — e.g. `a100-large`
+- `CPU_CORES`, `MEMORY`
+
+`run_in_job.sh` already echoes these at startup.
+
+## Monitoring
+
+- **W&B**: set `WANDB_API_KEY` before launching; runs appear under
+  `$WANDB_PROJECT` (default `incident_env_rl`) with the job id in the
+  run name.
+- **Rubric audit**: every terminal step logs `rubric_version`,
+  `rubric_raw_components`, and `score` (see `RUBRICS.md`). Use
+  `scripts/eval_incident_env.py` against a saved LoRA adapter for
+  held-out evaluation.
+
+## Troubleshooting
+
+- **OOM during vLLM startup** → lower
+  `actor_rollout_ref.rollout.gpu_memory_utilization` (default 0.45) or
+  drop `max_response_length`.
+- **Trainer OOM during FSDP** → raise
+  `actor_rollout_ref.actor.fsdp_config.param_offload: true` and
+  `optimizer_offload: true` (already set in 7B config).
+- **vLLM and trainer both allocate the same GPU**: yes, that's intended
+  — `hybrid_engine: true` + low `gpu_memory_utilization` shares the
+  GPU. If you see `CUDA out of memory`, the balance needs tuning.
+- **Job hits the 30-minute default timeout**: pass `--timeout 6h`.
+- **Pool cleanup** between episodes is automatic via the
+  `IncidentEnvManager.close()` path; no action required.

jobs/eval_in_job.sh

@@ -0,0 +1,169 @@
+#!/usr/bin/env bash
+# HF Jobs entrypoint for SCHEDULED EVAL runs of incident_env checkpoints.
+#
+# Boots (in order) inside a single GPU container:
+#   1. incident_env FastAPI server on :$ENV_PORT
+#   2. vLLM OpenAI-compatible server on :$VLLM_PORT serving $BASE_MODEL,
+#      with the target LoRA adapter hot-loaded (if LORA_PATH provided).
+#   3. scripts/eval_incident_env.py against the two endpoints.
+#
+# The eval report (JSON with per-task score_mean/std + rubric audit) is
+# written under $OUTPUT_DIR/eval_reports/<timestamp>.json, which is a
+# mounted HF dataset/bucket → survives job termination.
+#
+# Env vars consumed (defaults in brackets):
+#   BASE_MODEL              [Qwen/Qwen2.5-7B-Instruct]
+#   LORA_PATH               [<unset>]   path/repo for LoRA adapter to eval;
+#                                       if unset, evaluates the base model
+#   LORA_NAME               [incident_lora]  served name for vLLM
+#   SEEDS                   [30]        seeds per task
+#   TASKS                   [memory_leak cascading_failure distributed_deadlock]
+#   TEMPERATURE             [0.0]
+#   MAX_TURNS               [20]
+#   REWARD_MODE             [long_horizon_v2]
+#   OUTPUT_DIR              [/training-outputs]
+#   REPORT_TAG              [<JOB_ID or timestamp>]
+
+set -euo pipefail
+
+BASE_MODEL="${BASE_MODEL:-Qwen/Qwen2.5-7B-Instruct}"
+LORA_PATH="${LORA_PATH:-}"
+LORA_NAME="${LORA_NAME:-incident_lora}"
+SEEDS="${SEEDS:-30}"
+TASKS="${TASKS:-memory_leak cascading_failure distributed_deadlock}"
+TEMPERATURE="${TEMPERATURE:-0.0}"
+MAX_TURNS="${MAX_TURNS:-20}"
+REWARD_MODE="${REWARD_MODE:-long_horizon_v2}"
+ENV_PORT="${ENV_PORT:-8000}"
+VLLM_PORT="${VLLM_PORT:-8080}"
+OUTPUT_DIR="${OUTPUT_DIR:-/training-outputs}"
+REPORT_TAG="${REPORT_TAG:-${JOB_ID:-$(date +%Y%m%d_%H%M%S)}}"
+
+INCIDENT_REPO_DIR="${INCIDENT_REPO_DIR:-/workspace/incident_env}"
+
+echo "=========================================="
+echo " HF Job: incident_env EVAL"
+echo "=========================================="
+echo " JOB_ID:        ${JOB_ID:-<local>}"
+echo " ACCELERATOR:   ${ACCELERATOR:-unknown}"
+echo " BASE_MODEL:    $BASE_MODEL"
+echo " LORA_PATH:     ${LORA_PATH:-<base model only>}"
+echo " SEEDS:         $SEEDS"
+echo " TASKS:         $TASKS"
+echo " REWARD_MODE:   $REWARD_MODE"
+echo " REPORT_TAG:    $REPORT_TAG"
+echo "=========================================="
+
+# ---- Auth ----------------------------------------------------------------
+if [ -n "${HF_TOKEN:-}" ]; then
+    huggingface-cli login --token "$HF_TOKEN" --add-to-git-credential 2>/dev/null || true
+fi
+
+# ---- Refresh incident_env repo if a ref is pinned ------------------------
+if [ -n "${INCIDENT_REPO_REF:-}" ]; then
+    (cd "$INCIDENT_REPO_DIR" && git fetch --all --tags && git checkout "$INCIDENT_REPO_REF") || true
+fi
+pip install -e "$INCIDENT_REPO_DIR" 2>/dev/null || true
+export PYTHONPATH="$INCIDENT_REPO_DIR:${PYTHONPATH:-}"
+
+# ---- Background process management ---------------------------------------
+PIDS=()
+cleanup() {
+    echo ">> cleanup"
+    for pid in "${PIDS[@]}"; do
+        kill "$pid" 2>/dev/null || true
+    done
+    wait 2>/dev/null || true
+}
+trap cleanup EXIT INT TERM
+
+# ---- 1. env server -------------------------------------------------------
+echo ">> Starting incident_env server on :$ENV_PORT"
+(
+    cd "$INCIDENT_REPO_DIR"
+    uvicorn server.app:app --host 0.0.0.0 --port "$ENV_PORT" \
+        > /tmp/env_server.log 2>&1
+) &
+PIDS+=($!)
+
+for i in $(seq 1 60); do
+    if curl -sf "http://localhost:$ENV_PORT/health" > /dev/null 2>&1; then
+        echo ">> env server ready"
+        break
+    fi
+    sleep 1
+done
+curl -sf "http://localhost:$ENV_PORT/health" > /dev/null || { echo "!! env server failed"; tail -50 /tmp/env_server.log; exit 1; }
+
+# ---- 2. vLLM (with optional LoRA) ----------------------------------------
+VLLM_ARGS=(
+    --model "$BASE_MODEL"
+    --port "$VLLM_PORT"
+    --host 0.0.0.0
+    --gpu-memory-utilization 0.85
+    --max-model-len 4096
+    --enforce-eager
+)
+if [ -n "$LORA_PATH" ]; then
+    echo ">> vLLM will serve LoRA '$LORA_NAME' from $LORA_PATH"
+    VLLM_ARGS+=(
+        --enable-lora
+        --max-loras 2
+        --max-lora-rank 32
+        --lora-modules "${LORA_NAME}=${LORA_PATH}"
+    )
+    SERVED_MODEL="$LORA_NAME"
+else
+    echo ">> vLLM will serve BASE model only (no LoRA)"
+    SERVED_MODEL="$BASE_MODEL"
+fi
+
+VLLM_ALLOW_RUNTIME_LORA_UPDATING=True python -m vllm.entrypoints.openai.api_server \
+    "${VLLM_ARGS[@]}" > /tmp/vllm.log 2>&1 &
+PIDS+=($!)
+
+for i in $(seq 1 180); do
+    if curl -sf "http://localhost:$VLLM_PORT/health" > /dev/null 2>&1; then
+        echo ">> vLLM ready"
+        break
+    fi
+    sleep 2
+done
+curl -sf "http://localhost:$VLLM_PORT/health" > /dev/null || { echo "!! vLLM failed"; tail -80 /tmp/vllm.log; exit 1; }
+
+# ---- 3. Run eval ---------------------------------------------------------
+REPORT_DIR="$OUTPUT_DIR/eval_reports"
+mkdir -p "$REPORT_DIR" 2>/dev/null || true
+REPORT_PATH="$REPORT_DIR/${REPORT_TAG}.json"
+
+echo ">> Running eval → $REPORT_PATH"
+# shellcheck disable=SC2086
+python "$INCIDENT_REPO_DIR/scripts/eval_incident_env.py" \
+    --env-url "http://localhost:$ENV_PORT" \
+    --endpoint "http://localhost:$VLLM_PORT/v1" \
+    --model "$SERVED_MODEL" \
+    --seeds "$SEEDS" \
+    --tasks $TASKS \
+    --temperature "$TEMPERATURE" \
+    --max-turns "$MAX_TURNS" \
+    --reward-mode "$REWARD_MODE" \
+    --out "$REPORT_PATH"
+
+echo ">> Eval complete."
+echo ">> Report: $REPORT_PATH"
+
+# Print a compact summary so it surfaces in HF Jobs logs
+python - <<PY
+import json, sys
+with open("$REPORT_PATH") as f:
+    out = json.load(f)
+s = out["summary"]
+print("=" * 60)
+print("EVAL SUMMARY")
+print(f"  model:         {s['model']}")
+print(f"  overall_mean:  {s['overall_score_mean']}")
+print(f"  overall_std:   {s['overall_score_std']}")
+for task, v in s["per_task"].items():
+    print(f"  {task:25s} score={v['score_mean']:.3f}  reward={v['reward_mean']:.3f}  turns={v['turns_mean']:.1f}  n={v['n']}")
+print("=" * 60)
+PY

jobs/launch_hf_job.py

@@ -0,0 +1,226 @@
+#!/usr/bin/env python3
+"""Launch incident_env post-training on HF Jobs.
+
+Thin wrapper around `huggingface_hub.run_job` that:
+- selects a GPU flavor (default `a100-large` = single A100-80GB),
+- sets a generous timeout (default 6 h),
+- forwards HF_TOKEN / WANDB_API_KEY as secrets,
+- optionally mounts an HF dataset repo or bucket at /training-outputs
+  so checkpoints persist between jobs.
+
+Prerequisites
+-------------
+- HF Pro / Team / Enterprise account (HF Jobs is gated).
+- `pip install "huggingface_hub>=1.0"`
+- `hf auth login` (or set HF_TOKEN).
+- A Docker image already pushed somewhere HF Jobs can pull from — either
+  Docker Hub (`user/incident-env-job:tag`) or an HF Space used as an image
+  (`hf.co/spaces/user/incident-env-job`). See jobs/README.md for build steps.
+
+Usage
+-----
+    python jobs/launch_hf_job.py \\
+        --image docker.io/youruser/incident-env-job:latest \\
+        --flavor a100-large \\
+        --timeout 6h \\
+        --hf-repo youruser/incident-env-checkpoints \\
+        --wandb-project incident_env_rl \\
+        --base-model Qwen/Qwen2.5-7B-Instruct \\
+        --config incident_env_gigpo_7b_a100 \\
+        --epochs 30
+"""
+
+from __future__ import annotations
+
+import argparse
+import os
+import sys
+import time
+from typing import Any, Dict, List, Optional
+
+try:
+    from huggingface_hub import (
+        Volume,
+        cancel_job,
+        fetch_job_logs,
+        inspect_job,
+        run_job,
+    )
+except ImportError:
+    print(
+        "ERROR: huggingface_hub>=1.0 required. "
+        "Install with: pip install 'huggingface_hub>=1.0'",
+        file=sys.stderr,
+    )
+    sys.exit(1)
+
+
+VALID_FLAVORS = {
+    # GPU — see https://huggingface.co/docs/huggingface_hub/guides/jobs
+    "t4-small",
+    "t4-medium",
+    "l4x1",
+    "l4x4",
+    "a10g-small",
+    "a10g-large",
+    "a10g-largex2",
+    "a10g-largex4",
+    "a100-large",  # single A100-80GB — recommended for Qwen2.5-7B + LoRA
+}
+
+
+def build_args() -> argparse.Namespace:
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--image", required=True, help="Docker image (Docker Hub or hf.co/spaces/...).")
+    ap.add_argument(
+        "--flavor",
+        default="a100-large",
+        choices=sorted(VALID_FLAVORS),
+        help="HF Jobs hardware flavor.",
+    )
+    ap.add_argument("--timeout", default="6h", help="Job timeout (e.g. 6h, 30m, 1d). Default 6h.")
+    ap.add_argument("--namespace", default=None, help="Org namespace (optional).")
+
+    # Training knobs forwarded as env vars to the container.
+    ap.add_argument("--base-model", default="Qwen/Qwen2.5-7B-Instruct")
+    ap.add_argument("--config", default="incident_env_gigpo_7b_a100", help="YAML config basename in configs/.")
+    ap.add_argument("--epochs", type=int, default=30)
+    ap.add_argument("--num-envs", type=int, default=4)
+    ap.add_argument("--rollout-n", type=int, default=4)
+    ap.add_argument("--reward-mode", default="long_horizon_v2", choices=["legacy", "long_horizon_v2"])
+    ap.add_argument("--incident-repo-ref", default=None, help="Optional git ref for this repo (otherwise baked image SHA).")
+    ap.add_argument("--verl-agent-repo-ref", default=None)
+    ap.add_argument("--wandb-project", default="incident_env_rl")
+
+    # Persistence — checkpoints survive job termination.
+    ap.add_argument(
+        "--hf-repo",
+        default=None,
+        help="HF repo (dataset / bucket) to mount at /training-outputs. Requires write access.",
+    )
+    ap.add_argument(
+        "--hf-repo-type",
+        default="dataset",
+        choices=["dataset", "model", "bucket"],
+        help="Repo type for the output mount.",
+    )
+
+    # Monitoring.
+    ap.add_argument("--follow", action="store_true", help="Tail logs after launch.")
+    ap.add_argument("--poll-interval", type=float, default=5.0)
+    return ap.parse_args()
+
+
+def build_env_vars(args: argparse.Namespace) -> Dict[str, str]:
+    env = {
+        "BASE_MODEL": args.base_model,
+        "CONFIG_NAME": args.config,
+        "EPOCHS": str(args.epochs),
+        "NUM_ENVS": str(args.num_envs),
+        "ROLLOUT_N": str(args.rollout_n),
+        "INCIDENT_REWARD_MODE": args.reward_mode,
+        "WANDB_PROJECT": args.wandb_project,
+    }
+    if args.incident_repo_ref:
+        env["INCIDENT_REPO_REF"] = args.incident_repo_ref
+    if args.verl_agent_repo_ref:
+        env["VERL_AGENT_REPO_REF"] = args.verl_agent_repo_ref
+    if args.hf_repo:
+        # The entrypoint writes checkpoints under OUTPUT_DIR.
+        env["OUTPUT_DIR"] = "/training-outputs"
+        env["HF_REPO"] = args.hf_repo
+    return env
+
+
+def build_secrets() -> Dict[str, str]:
+    """Forward HF_TOKEN + WANDB_API_KEY from the caller's env as encrypted secrets."""
+    out: Dict[str, str] = {}
+    hf = os.environ.get("HF_TOKEN") or os.environ.get("HUGGING_FACE_HUB_TOKEN")
+    if hf:
+        out["HF_TOKEN"] = hf
+    wb = os.environ.get("WANDB_API_KEY")
+    if wb:
+        out["WANDB_API_KEY"] = wb
+    return out
+
+
+def build_volumes(args: argparse.Namespace) -> Optional[List[Volume]]:
+    if not args.hf_repo:
+        return None
+    vol = Volume(
+        type=args.hf_repo_type,
+        source=args.hf_repo,
+        mount_path="/training-outputs",
+    )
+    return [vol]
+
+
+def follow_logs(job_id: str, poll_interval: float) -> int:
+    """Stream logs until job terminates. Returns 0 on COMPLETED, non-zero on ERROR."""
+    print(f">> Following logs for job {job_id} (Ctrl-C to detach; job keeps running)")
+    last_line = 0
+    try:
+        while True:
+            status = inspect_job(job_id=job_id).status
+            logs = list(fetch_job_logs(job_id=job_id))
+            for line in logs[last_line:]:
+                print(line)
+            last_line = len(logs)
+            if status.stage in ("COMPLETED", "ERROR", "CANCELED"):
+                print(f">> Job finished: stage={status.stage} message={status.message}")
+                return 0 if status.stage == "COMPLETED" else 1
+            time.sleep(poll_interval)
+    except KeyboardInterrupt:
+        print(">> Detached. Job continues running on HF infrastructure.")
+        return 0
+
+
+def main() -> int:
+    args = build_args()
+
+    env = build_env_vars(args)
+    secrets = build_secrets()
+    volumes = build_volumes(args)
+
+    print("=" * 60)
+    print(" HF Jobs — incident_env post-training")
+    print("=" * 60)
+    print(f" image:        {args.image}")
+    print(f" flavor:       {args.flavor}")
+    print(f" timeout:      {args.timeout}")
+    print(f" namespace:    {args.namespace or '<user>'}")
+    print(f" base_model:   {args.base_model}")
+    print(f" config:       {args.config}")
+    print(f" epochs:       {args.epochs}")
+    print(f" num_envs:     {args.num_envs}")
+    print(f" rollout_n:    {args.rollout_n}")
+    print(f" reward_mode:  {args.reward_mode}")
+    print(f" volumes:      {args.hf_repo or '<none>'}")
+    print(f" env:          {sorted(env.keys())}")
+    print(f" secrets:      {sorted(secrets.keys())}")
+    print("=" * 60)
+
+    kwargs: Dict[str, Any] = dict(
+        image=args.image,
+        command=["/usr/local/bin/run_in_job.sh"],
+        flavor=args.flavor,
+        timeout=args.timeout,
+        env=env,
+        secrets=secrets,
+    )
+    if args.namespace:
+        kwargs["namespace"] = args.namespace
+    if volumes:
+        kwargs["volumes"] = volumes
+
+    job = run_job(**kwargs)
+    print(f">> Job launched: {job.url}")
+    print(f">> Job ID:       {job.id}")
+
+    if args.follow:
+        return follow_logs(job.id, args.poll_interval)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

jobs/run_in_job.sh

@@ -0,0 +1,169 @@
+#!/usr/bin/env bash
+# HF Jobs entrypoint.
+#
+# Inside a single GPU container, boot (in order):
+#   1. incident_env FastAPI server on :$ENV_PORT
+#   2. vLLM OpenAI-compatible server on :$VLLM_PORT (the actor backend)
+#   3. verl-agent trainer (foreground) — when this exits, the job ends
+#
+# Env vars consumed (all optional, reasonable defaults):
+#   BASE_MODEL            default: Qwen/Qwen2.5-7B-Instruct
+#   CONFIG_NAME           default: incident_env_gigpo_7b_a100
+#   EPOCHS                total training epochs
+#   NUM_ENVS              parallel incident envs
+#   ROLLOUT_N             GRPO group size
+#   HF_REPO               HF dataset or model repo to push checkpoints to
+#   HF_TOKEN              (secret)
+#   WANDB_API_KEY         (secret)
+#   WANDB_PROJECT         default: incident_env_rl
+#   INCIDENT_REPO_REF     optional git ref to check out (otherwise baked-in)
+#   VERL_AGENT_REPO_REF   optional git ref
+
+set -euo pipefail
+
+BASE_MODEL="${BASE_MODEL:-Qwen/Qwen2.5-7B-Instruct}"
+CONFIG_NAME="${CONFIG_NAME:-incident_env_gigpo_7b_a100}"
+EPOCHS="${EPOCHS:-30}"
+NUM_ENVS="${NUM_ENVS:-4}"
+ROLLOUT_N="${ROLLOUT_N:-4}"
+ENV_PORT="${ENV_PORT:-8000}"
+VLLM_PORT="${VLLM_PORT:-8080}"
+WANDB_PROJECT="${WANDB_PROJECT:-incident_env_rl}"
+
+INCIDENT_REPO_DIR="${INCIDENT_REPO_DIR:-/workspace/incident_env}"
+VERL_AGENT_REPO_DIR="${VERL_AGENT_REPO_DIR:-/workspace/verl-agent}"
+
+echo "=========================================="
+echo " HF Job: incident_env post-training"
+echo "=========================================="
+echo " JOB_ID:        ${JOB_ID:-<local>}"
+echo " ACCELERATOR:   ${ACCELERATOR:-unknown}"
+echo " BASE_MODEL:    $BASE_MODEL"
+echo " CONFIG:        $CONFIG_NAME"
+echo " EPOCHS:        $EPOCHS"
+echo " NUM_ENVS:      $NUM_ENVS"
+echo " ROLLOUT_N:     $ROLLOUT_N"
+echo " REWARD_MODE:   ${INCIDENT_REWARD_MODE:-long_horizon_v2}"
+echo "=========================================="
+
+# ---- Auth ----------------------------------------------------------------
+if [ -n "${HF_TOKEN:-}" ]; then
+    echo ">> HF login"
+    huggingface-cli login --token "$HF_TOKEN" --add-to-git-credential 2>/dev/null || true
+fi
+if [ -n "${WANDB_API_KEY:-}" ]; then
+    export WANDB_PROJECT
+    echo ">> W&B authenticated"
+fi
+
+# ---- Refresh repos if refs provided --------------------------------------
+if [ -n "${INCIDENT_REPO_REF:-}" ]; then
+    echo ">> Checkout incident_env @ $INCIDENT_REPO_REF"
+    (cd "$INCIDENT_REPO_DIR" && git fetch --all --tags && git checkout "$INCIDENT_REPO_REF") || true
+fi
+if [ -n "${VERL_AGENT_REPO_REF:-}" ]; then
+    echo ">> Checkout verl-agent @ $VERL_AGENT_REPO_REF"
+    (cd "$VERL_AGENT_REPO_DIR" && git fetch --all --tags && git checkout "$VERL_AGENT_REPO_REF") || true
+fi
+
+pip install -e "$INCIDENT_REPO_DIR" 2>/dev/null || true
+pip install -e "$VERL_AGENT_REPO_DIR" 2>/dev/null || true
+
+export PYTHONPATH="$INCIDENT_REPO_DIR:$VERL_AGENT_REPO_DIR:${PYTHONPATH:-}"
+export INCIDENT_ENV_BASE_URL="http://localhost:$ENV_PORT"
+
+# ---- Wire adapter into verl-agent ----------------------------------------
+ADAPTER_DST="$VERL_AGENT_REPO_DIR/agent_system/environments/incident_env"
+if [ ! -e "$ADAPTER_DST" ]; then
+    echo ">> Linking env adapter -> $ADAPTER_DST"
+    mkdir -p "$(dirname "$ADAPTER_DST")"
+    ln -s "$INCIDENT_REPO_DIR/verl_agent_adapter" "$ADAPTER_DST"
+fi
+
+RM_DST="$VERL_AGENT_REPO_DIR/agent_system/reward_manager/incident_episode.py"
+if [ ! -f "$RM_DST" ]; then
+    echo ">> Installing reward-manager shim"
+    mkdir -p "$(dirname "$RM_DST")"
+    cat > "$RM_DST" <<'PYEOF'
+from agent_system.environments.incident_env.reward_registry import IncidentEpisodeRewardManager  # noqa: F401
+__all__ = ["IncidentEpisodeRewardManager"]
+PYEOF
+fi
+
+# ---- Background process management ---------------------------------------
+PIDS=()
+cleanup() {
+    echo ">> Cleaning up subprocesses"
+    for pid in "${PIDS[@]}"; do
+        kill "$pid" 2>/dev/null || true
+    done
+    wait 2>/dev/null || true
+}
+trap cleanup EXIT INT TERM
+
+# ---- 1. incident_env server ----------------------------------------------
+echo ">> Starting incident_env server on :$ENV_PORT"
+(
+    cd "$INCIDENT_REPO_DIR"
+    uvicorn server.app:app --host 0.0.0.0 --port "$ENV_PORT" \
+        > /tmp/incident_env_server.log 2>&1
+) &
+PIDS+=($!)
+
+for i in $(seq 1 60); do
+    if curl -sf "http://localhost:$ENV_PORT/health" > /dev/null 2>&1; then
+        echo ">> env server ready"
+        break
+    fi
+    sleep 1
+done
+if ! curl -sf "http://localhost:$ENV_PORT/health" > /dev/null 2>&1; then
+    echo "!! env server failed to start; logs:"; tail -50 /tmp/incident_env_server.log; exit 1
+fi
+
+# ---- 2. vLLM -------------------------------------------------------------
+# One GPU is shared between vLLM and the actor/ref FSDP. Keep vLLM small:
+# gpu_memory_utilization=0.45 leaves room for the FSDP actor + optim state.
+echo ">> Starting vLLM for $BASE_MODEL on :$VLLM_PORT"
+VLLM_ALLOW_RUNTIME_LORA_UPDATING=True python -m vllm.entrypoints.openai.api_server \
+    --model "$BASE_MODEL" \
+    --port "$VLLM_PORT" \
+    --host 0.0.0.0 \
+    --enable-lora \
+    --max-loras 4 \
+    --max-lora-rank 32 \
+    --gpu-memory-utilization 0.45 \
+    --max-model-len 4096 \
+    --enforce-eager \
+    > /tmp/vllm.log 2>&1 &
+PIDS+=($!)
+
+for i in $(seq 1 180); do
+    if curl -sf "http://localhost:$VLLM_PORT/health" > /dev/null 2>&1; then
+        echo ">> vLLM ready"
+        break
+    fi
+    sleep 2
+done
+if ! curl -sf "http://localhost:$VLLM_PORT/health" > /dev/null 2>&1; then
+    echo "!! vLLM failed to start; logs:"; tail -80 /tmp/vllm.log; exit 1
+fi
+
+# ---- 3. verl-agent trainer (foreground) ----------------------------------
+CONFIG_PATH="$INCIDENT_REPO_DIR/configs"
+OUT_DIR="${OUTPUT_DIR:-/training-outputs}"
+mkdir -p "$OUT_DIR" || true
+
+echo ">> Launching verl-agent trainer"
+cd "$VERL_AGENT_REPO_DIR"
+exec python -m verl.trainer.main_ppo \
+    --config-path "$CONFIG_PATH" \
+    --config-name "$CONFIG_NAME" \
+    actor_rollout_ref.model.path="$BASE_MODEL" \
+    env.num_envs="$NUM_ENVS" \
+    env.rollout.n="$ROLLOUT_N" \
+    actor_rollout_ref.rollout.n="$ROLLOUT_N" \
+    trainer.total_epochs="$EPOCHS" \
+    trainer.default_local_dir="$OUT_DIR/$CONFIG_NAME" \
+    trainer.project_name="$WANDB_PROJECT" \
+    "$@"

jobs/schedule_eval_job.py

@@ -0,0 +1,272 @@
+#!/usr/bin/env python3
+"""Create / manage a SCHEDULED eval job on HF Jobs.
+
+Runs `jobs/eval_in_job.sh` on a recurring schedule (weekly by default)
+against the latest LoRA adapter in a given HF repo. Writes a JSON report
+to the mounted checkpoint dataset under `eval_reports/<tag>.json`.
+
+Reference:
+https://huggingface.co/docs/huggingface_hub/guides/jobs#scheduled-jobs
+
+Supported schedules (from HF docs):
+    @annually | @yearly | @monthly | @weekly | @daily | @hourly
+    or a CRON expression, e.g. "0 9 * * 1" (Mon 09:00 UTC)
+
+Prerequisites
+-------------
+- The same Docker image built in `jobs/Dockerfile` (which now also
+  contains `/usr/local/bin/eval_in_job.sh`).
+- HF Pro / Team / Enterprise account.
+- `pip install "huggingface_hub>=1.0"`.
+
+Usage
+-----
+    # Create a weekly eval (Mondays 09:00 UTC) on a10g-large
+    python jobs/schedule_eval_job.py create \\
+        --image docker.io/<you>/incident-env-job:latest \\
+        --schedule "@weekly" \\
+        --flavor a10g-large \\
+        --timeout 90m \\
+        --hf-repo <you>/incident-env-checkpoints \\
+        --lora-path "/training-outputs/incident_env_gigpo_7b_a100/latest_lora" \\
+        --base-model Qwen/Qwen2.5-7B-Instruct \\
+        --seeds 30
+
+    # List / inspect / pause / delete
+    python jobs/schedule_eval_job.py list
+    python jobs/schedule_eval_job.py inspect <SCHEDULED_ID>
+    python jobs/schedule_eval_job.py pause   <SCHEDULED_ID>
+    python jobs/schedule_eval_job.py resume  <SCHEDULED_ID>
+    python jobs/schedule_eval_job.py delete  <SCHEDULED_ID>
+"""
+
+from __future__ import annotations
+
+import argparse
+import os
+import sys
+from typing import Any, Dict, List, Optional
+
+try:
+    from huggingface_hub import (
+        Volume,
+        create_scheduled_job,
+        delete_scheduled_job,
+        inspect_scheduled_job,
+        list_scheduled_jobs,
+        resume_scheduled_job,
+        suspend_scheduled_job,
+    )
+except ImportError:
+    print(
+        "ERROR: huggingface_hub>=1.0 required. "
+        "Install with: pip install 'huggingface_hub>=1.0'",
+        file=sys.stderr,
+    )
+    sys.exit(1)
+
+
+VALID_FLAVORS = {
+    "t4-small", "t4-medium", "l4x1", "l4x4",
+    "a10g-small", "a10g-large", "a10g-largex2", "a10g-largex4",
+    "a100-large",
+}
+
+
+# ---------------------------------------------------------------------------
+# Builders
+# ---------------------------------------------------------------------------
+
+
+def build_env(args: argparse.Namespace) -> Dict[str, str]:
+    env: Dict[str, str] = {
+        "BASE_MODEL": args.base_model,
+        "SEEDS": str(args.seeds),
+        "TASKS": " ".join(args.tasks),
+        "TEMPERATURE": str(args.temperature),
+        "MAX_TURNS": str(args.max_turns),
+        "REWARD_MODE": args.reward_mode,
+    }
+    if args.lora_path:
+        env["LORA_PATH"] = args.lora_path
+    if args.lora_name:
+        env["LORA_NAME"] = args.lora_name
+    if args.incident_repo_ref:
+        env["INCIDENT_REPO_REF"] = args.incident_repo_ref
+    if args.hf_repo:
+        env["OUTPUT_DIR"] = "/training-outputs"
+    return env
+
+
+def build_secrets() -> Dict[str, str]:
+    out: Dict[str, str] = {}
+    hf = os.environ.get("HF_TOKEN") or os.environ.get("HUGGING_FACE_HUB_TOKEN")
+    if hf:
+        out["HF_TOKEN"] = hf
+    return out
+
+
+def build_volumes(args: argparse.Namespace) -> Optional[List[Volume]]:
+    if not args.hf_repo:
+        return None
+    return [
+        Volume(
+            type=args.hf_repo_type,
+            source=args.hf_repo,
+            mount_path="/training-outputs",
+        )
+    ]
+
+
+# ---------------------------------------------------------------------------
+# Commands
+# ---------------------------------------------------------------------------
+
+
+def cmd_create(args: argparse.Namespace) -> int:
+    env = build_env(args)
+    secrets = build_secrets()
+    volumes = build_volumes(args)
+
+    print("=" * 60)
+    print(" Scheduled eval job — incident_env")
+    print("=" * 60)
+    print(f" image:     {args.image}")
+    print(f" schedule:  {args.schedule}")
+    print(f" flavor:    {args.flavor}")
+    print(f" timeout:   {args.timeout}")
+    print(f" hf_repo:   {args.hf_repo or '<none>'}")
+    print(f" lora:      {args.lora_path or '<base model>'}")
+    print(f" tasks:     {args.tasks}")
+    print(f" seeds:     {args.seeds}")
+    print("=" * 60)
+
+    kwargs: Dict[str, Any] = dict(
+        image=args.image,
+        command=["/usr/local/bin/eval_in_job.sh"],
+        schedule=args.schedule,
+        flavor=args.flavor,
+        timeout=args.timeout,
+        env=env,
+        secrets=secrets,
+    )
+    if args.namespace:
+        kwargs["namespace"] = args.namespace
+    if volumes:
+        kwargs["volumes"] = volumes
+
+    sched = create_scheduled_job(**kwargs)
+    print(f">> created scheduled_job: {sched}")
+    # `sched` has .id; surface it for follow-up commands.
+    sid = getattr(sched, "id", None)
+    if sid:
+        print(f">> id: {sid}")
+    return 0
+
+
+def cmd_list(args: argparse.Namespace) -> int:
+    for s in list_scheduled_jobs():
+        print(s)
+    return 0
+
+
+def cmd_inspect(args: argparse.Namespace) -> int:
+    print(inspect_scheduled_job(args.scheduled_id))
+    return 0
+
+
+def cmd_pause(args: argparse.Namespace) -> int:
+    suspend_scheduled_job(args.scheduled_id)
+    print(f">> paused {args.scheduled_id}")
+    return 0
+
+
+def cmd_resume(args: argparse.Namespace) -> int:
+    resume_scheduled_job(args.scheduled_id)
+    print(f">> resumed {args.scheduled_id}")
+    return 0
+
+
+def cmd_delete(args: argparse.Namespace) -> int:
+    delete_scheduled_job(args.scheduled_id)
+    print(f">> deleted {args.scheduled_id}")
+    return 0
+
+
+# ---------------------------------------------------------------------------
+# CLI wiring
+# ---------------------------------------------------------------------------
+
+
+def _add_create_args(sp: argparse.ArgumentParser) -> None:
+    sp.add_argument("--image", required=True)
+    sp.add_argument(
+        "--schedule",
+        default="@weekly",
+        help='"@hourly" | "@daily" | "@weekly" | "@monthly" | CRON expr (e.g. "0 9 * * 1")',
+    )
+    sp.add_argument("--flavor", default="a10g-large", choices=sorted(VALID_FLAVORS))
+    sp.add_argument("--timeout", default="90m")
+    sp.add_argument("--namespace", default=None)
+
+    # eval config forwarded as env
+    sp.add_argument("--base-model", default="Qwen/Qwen2.5-7B-Instruct")
+    sp.add_argument("--lora-path", default=None,
+                    help="Path inside the container (usually under /training-outputs/...) "
+                         "or HF repo of the LoRA adapter to evaluate. "
+                         "If omitted, evaluates the base model only.")
+    sp.add_argument("--lora-name", default="incident_lora",
+                    help="Served model name used to address the LoRA in vLLM.")
+    sp.add_argument("--seeds", type=int, default=30)
+    sp.add_argument(
+        "--tasks",
+        nargs="+",
+        default=["memory_leak", "cascading_failure", "distributed_deadlock"],
+    )
+    sp.add_argument("--temperature", type=float, default=0.0)
+    sp.add_argument("--max-turns", type=int, default=20)
+    sp.add_argument("--reward-mode", default="long_horizon_v2",
+                    choices=["legacy", "long_horizon_v2"])
+    sp.add_argument("--incident-repo-ref", default=None)
+
+    # Output persistence
+    sp.add_argument(
+        "--hf-repo",
+        default=None,
+        help="HF repo to mount at /training-outputs (writes eval_reports/<tag>.json). "
+             "Usually the same repo used for training checkpoints.",
+    )
+    sp.add_argument(
+        "--hf-repo-type",
+        default="dataset",
+        choices=["dataset", "model", "bucket"],
+    )
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    sub = ap.add_subparsers(dest="cmd", required=True)
+
+    sp_create = sub.add_parser("create", help="Create a scheduled eval job")
+    _add_create_args(sp_create)
+    sp_create.set_defaults(func=cmd_create)
+
+    sub.add_parser("list", help="List scheduled jobs").set_defaults(func=cmd_list)
+
+    for name, fn in [
+        ("inspect", cmd_inspect),
+        ("pause", cmd_pause),
+        ("resume", cmd_resume),
+        ("delete", cmd_delete),
+    ]:
+        p = sub.add_parser(name, help=f"{name} a scheduled job")
+        p.add_argument("scheduled_id")
+        p.set_defaults(func=fn)
+
+    args = ap.parse_args()
+    return args.func(args)
+
+
+if __name__ == "__main__":
+    sys.exit(main())

reward_manager/episode.py

@@ -0,0 +1,189 @@
+# Copyright 2025 Nanyang Technological University (NTU), Singapore
+# and the verl-agent (GiGPO) team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Episode-level reward assignment for RL training on **incident_env**.
+
+Rollout collation should attach episode summaries in ``non_tensor_batch`` that
+mirror the HTTP env (see ``server/incident_environment.py`` / AGENTS.md):
+
+- ``episode_rewards`` (float): cumulative **oracle-shaped** training reward
+  (sum of per-step rewards from ``_compute_reward()``).
+- ``episode_lengths`` (int): number of agent steps in the episode (same idea
+  as ``step_count`` / ``info["steps_taken"]`` on the final step).
+- ``data_source`` **or** ``task_name``: incident task id — ``memory_leak``,
+  ``cascading_failure``, or ``distributed_deadlock`` (see ``tasks.py``).
+
+Evaluation **grader** scores from ``BaseScenario.grade()`` live in rollout
+metadata separately; this manager uses training episode return by default.
+
+If ``rm_scores`` is already present in ``batch``, it is returned unchanged
+(optional override from an external reward model).
+
+``data`` may be verl's ``DataProto`` when installed; any batched object with
+``__len__``, ``__getitem__``, ``.batch``, and ``.non_tensor_batch`` matching
+the above keys works.
+"""
+
+from __future__ import annotations
+
+import random
+from typing import Any, Dict, Union
+
+try:
+    import torch
+except ImportError:  # pragma: no cover - training stacks normally ship torch
+    torch = None  # type: ignore[misc, assignment]
+
+
+def _as_float(x: Any) -> float:
+    if isinstance(x, (float, int)):
+        return float(x)
+    if hasattr(x, "item"):
+        return float(x.item())
+    return float(x)
+
+
+def _as_int(x: Any) -> int:
+    if isinstance(x, int):
+        return x
+    if isinstance(x, float):
+        return int(x)
+    if hasattr(x, "item"):
+        return int(x.item())
+    return int(x)
+
+
+def _resolve_task_label(non_tensor_batch: Dict[str, Any]) -> str:
+    if "data_source" in non_tensor_batch:
+        return str(non_tensor_batch["data_source"])
+    if "task_name" in non_tensor_batch:
+        return str(non_tensor_batch["task_name"])
+    return "unknown"
+
+
+def _episode_score(
+    episode_rewards: Any,
+    episode_lengths: Any,
+    normalize_by_length: bool,
+) -> float:
+    r = _as_float(episode_rewards)
+    if normalize_by_length:
+        length = max(1, _as_int(episode_lengths))
+        return r / length
+    return r
+
+
+class EpisodeRewardManager:
+    """Maps incident episode return onto the last valid response token for RL."""
+
+    def __init__(
+        self,
+        tokenizer: Any,
+        num_examine: int,
+        normalize_by_length: bool = False,
+    ) -> None:
+        self.tokenizer = tokenizer
+        self.num_examine = num_examine
+        self.normalize_by_length = normalize_by_length
+
+    def __call__(
+        self,
+        data: Any,
+        return_dict: bool = False,
+    ) -> Union["torch.Tensor", Dict[str, Any]]:
+        if torch is None:
+            raise ImportError(
+                "EpisodeRewardManager requires `torch` (e.g. pip install torch)."
+            )
+
+        # If there is rm score, we directly return rm score.
+        batch_keys = data.batch.keys()
+        if "rm_scores" in batch_keys:
+            if return_dict:
+                return {"reward_tensor": data.batch["rm_scores"]}
+            return data.batch["rm_scores"]
+
+        reward_tensor = torch.zeros_like(
+            data.batch["responses"], dtype=torch.float32
+        )
+
+        already_print_data_sources: Dict[str, int] = {}
+        reward_extra_info: Dict[str, list] = {
+            "task_name": [],
+            "episode_return": [],
+            "episode_length": [],
+        }
+
+        for i in range(len(data)):
+            data_item = data[i]
+
+            prompt_ids = data_item.batch["prompts"]
+            prompt_length = prompt_ids.shape[-1]
+
+            valid_prompt_length = int(
+                data_item.batch["attention_mask"][:prompt_length].sum().item()
+            )
+            valid_prompt_ids = prompt_ids[-valid_prompt_length:]
+
+            response_ids = data_item.batch["responses"]
+            valid_response_length = int(
+                data_item.batch["attention_mask"][prompt_length:].sum().item()
+            )
+            valid_response_ids = response_ids[:valid_response_length]
+
+            prompt_str = self.tokenizer.decode(
+                valid_prompt_ids, skip_special_tokens=False
+            )
+            response_str = self.tokenizer.decode(
+                valid_response_ids, skip_special_tokens=False
+            )
+
+            ntb = data_item.non_tensor_batch
+            data_source = _resolve_task_label(ntb)
+
+            episode_rewards = ntb["episode_rewards"]
+            episode_lengths = ntb["episode_lengths"]
+            score = _episode_score(
+                episode_rewards, episode_lengths, self.normalize_by_length
+            )
+
+            last_tok = max(0, valid_response_length - 1)
+            reward_tensor[i, last_tok] = torch.tensor(
+                score, dtype=torch.float32, device=prompt_ids.device
+            )
+
+            reward_extra_info["task_name"].append(data_source)
+            reward_extra_info["episode_return"].append(_as_float(episode_rewards))
+            reward_extra_info["episode_length"].append(_as_int(episode_lengths))
+
+            if data_source not in already_print_data_sources:
+                already_print_data_sources[data_source] = 0
+
+            if (
+                already_print_data_sources[data_source] < self.num_examine
+                and random.random() < 0.1
+            ):
+                already_print_data_sources[data_source] += 1
+                print(f"[{data_source}][prompt]", prompt_str)
+                print(f"[{data_source}][response]", response_str)
+                print(f"[{data_source}][score]", score)
+
+        if return_dict:
+            return {
+                "reward_tensor": reward_tensor,
+                "reward_extra_info": reward_extra_info,
+            }
+        return reward_tensor

reward_manager/long_horizon_reward.py

@@ -0,0 +1,225 @@
+"""
+Long-horizon reward design for incident_env training.
+
+This module is a drop-in, more extensive alternative to the current
+`IncidentEnvironment._compute_reward()` shaping. It keeps the same action space
+and scenario contract while adding:
+
+- progressive time pressure
+- novelty / anti-thrashing incentives
+- alert and service-health progress shaping
+- phase-aware milestone bonuses
+- terminal quality bonus based on restoration + time efficiency
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any, Dict, Optional, Set, Tuple
+
+from models import ActionType, DIAGNOSTIC_ACTIONS, IncidentAction, REMEDIATION_ACTIONS
+
+
+@dataclass
+class LongHorizonRewardConfig:
+    # Step pressure
+    base_step_penalty: float = 0.01
+    late_step_penalty: float = 0.04
+
+    # Action quality
+    invalid_action_penalty: float = 0.08
+    repeat_action_penalty: float = 0.06
+    novelty_bonus: float = 0.03
+    oscillation_penalty: float = 0.03
+
+    # Investigation
+    diagnostic_relevant_bonus: float = 0.10
+    diagnostic_irrelevant_bonus: float = 0.02
+    first_view_alerts_bonus: float = 0.08
+
+    # Remediation
+    remediation_root_bonus: float = 0.20
+    remediation_related_bonus: float = 0.08
+    remediation_wrong_penalty: float = 0.14
+    first_correct_remediation_bonus: float = 0.10
+
+    # Root cause declaration
+    declare_correct_bonus: float = 0.30
+    declare_partial_bonus: float = 0.12
+    declare_wrong_penalty: float = 0.20
+
+    # Progress shaping (state deltas)
+    healthy_service_delta_weight: float = 0.35
+    alert_delta_weight: float = 0.20
+    potential_weight: float = 0.25
+
+    # Terminal shaping
+    terminal_full_restore_bonus: float = 0.35
+    terminal_partial_restore_scale: float = 0.20
+    terminal_no_declaration_penalty: float = 0.20
+    terminal_over_budget_penalty: float = 0.12
+
+    # Output clipping
+    min_step_reward: float = -0.60
+    max_step_reward: float = 0.80
+
+
+@dataclass
+class EpisodeRewardState:
+    seen_action_targets: Set[Tuple[str, Optional[str]]] = field(default_factory=set)
+    saw_view_alerts: bool = False
+    first_correct_remediation_awarded: bool = False
+    previous_action: Optional[Tuple[str, Optional[str]]] = None
+    max_alerts_seen: int = 1
+
+
+@dataclass
+class RewardBreakdown:
+    total: float
+    components: Dict[str, float]
+
+
+def _keyword_match_ratio(declared_text: str, keywords: list[str]) -> float:
+    if not keywords:
+        return 0.0
+    declared = (declared_text or "").lower()
+    matched = sum(1 for kw in keywords if kw.lower() in declared)
+    return matched / max(1, len(keywords))
+
+
+def _health_ratio(obs: Dict[str, Any]) -> float:
+    statuses = obs.get("service_statuses", {}) or {}
+    if not statuses:
+        return 0.0
+    healthy = sum(1 for s in statuses.values() if s == "healthy")
+    return healthy / max(1, len(statuses))
+
+
+def _alerts_count(obs: Dict[str, Any]) -> int:
+    return int(obs.get("active_alerts_count", 0) or 0)
+
+
+class LongHorizonRewardEngine:
+    """Stateful per-episode reward engine."""
+
+    def __init__(self, config: Optional[LongHorizonRewardConfig] = None) -> None:
+        self.cfg = config or LongHorizonRewardConfig()
+        self.state = EpisodeRewardState()
+
+    def reset(self) -> None:
+        self.state = EpisodeRewardState()
+
+    def compute(
+        self,
+        *,
+        action: IncidentAction,
+        scenario: Any,
+        prev_obs: Dict[str, Any],
+        curr_obs: Dict[str, Any],
+        is_valid_action: bool,
+        is_repeat: bool,
+        root_cause_declared: bool,
+        episode_done: bool,
+        step_count: int,
+        max_steps: int,
+        time_elapsed_minutes: int,
+        time_budget_minutes: int,
+    ) -> RewardBreakdown:
+        c: Dict[str, float] = {}
+        at = action.parsed_type()
+        target = action.target_service
+        pair = (action.action_type, target)
+
+        progress = min(1.0, max(0.0, step_count / max(1, max_steps)))
+        c["step_penalty"] = -(
+            self.cfg.base_step_penalty
+            + (self.cfg.late_step_penalty - self.cfg.base_step_penalty) * progress
+        )
+
+        if not is_valid_action:
+            c["invalid_action"] = -self.cfg.invalid_action_penalty
+            total = sum(c.values())
+            total = max(self.cfg.min_step_reward, min(self.cfg.max_step_reward, total))
+            return RewardBreakdown(total=round(total, 3), components=c)
+
+        if is_repeat:
+            c["repeat_action"] = -self.cfg.repeat_action_penalty
+        elif pair not in self.state.seen_action_targets:
+            c["novelty"] = self.cfg.novelty_bonus
+            self.state.seen_action_targets.add(pair)
+
+        # Investigation / remediation / declaration
+        if at in DIAGNOSTIC_ACTIONS:
+            if at == ActionType.VIEW_ALERTS and not self.state.saw_view_alerts:
+                c["first_view_alerts"] = self.cfg.first_view_alerts_bonus
+                self.state.saw_view_alerts = True
+            if target and target in scenario.involved_services:
+                c["diagnostic_relevant"] = self.cfg.diagnostic_relevant_bonus
+            elif target and target not in scenario.involved_services:
+                c["diagnostic_irrelevant"] = self.cfg.diagnostic_irrelevant_bonus
+
+        elif at in REMEDIATION_ACTIONS:
+            if target == scenario.root_cause_service:
+                c["remediation_root"] = self.cfg.remediation_root_bonus
+                if not self.state.first_correct_remediation_awarded:
+                    c["first_correct_remediation"] = self.cfg.first_correct_remediation_bonus
+                    self.state.first_correct_remediation_awarded = True
+            elif target and target in scenario.involved_services:
+                c["remediation_related"] = self.cfg.remediation_related_bonus
+            else:
+                c["remediation_wrong"] = -self.cfg.remediation_wrong_penalty
+
+        elif at == ActionType.DECLARE_ROOT_CAUSE:
+            ratio = _keyword_match_ratio(
+                action.parameters.get("root_cause", ""),
+                list(scenario.root_cause_keywords),
+            )
+            if ratio >= 0.6:
+                c["declare_correct"] = self.cfg.declare_correct_bonus
+            elif ratio >= 0.3:
+                c["declare_partial"] = self.cfg.declare_partial_bonus
+            else:
+                c["declare_wrong"] = -self.cfg.declare_wrong_penalty
+
+        # Anti-thrashing: alternating remediation targets often hurts long-horizon stability.
+        if self.state.previous_action is not None:
+            prev_at, prev_target = self.state.previous_action
+            if (
+                prev_at in {a.value for a in REMEDIATION_ACTIONS}
+                and action.action_type in {a.value for a in REMEDIATION_ACTIONS}
+                and prev_target != target
+            ):
+                c["oscillation"] = -self.cfg.oscillation_penalty
+        self.state.previous_action = pair
+
+        # Progress shaping from observation deltas
+        prev_h, curr_h = _health_ratio(prev_obs), _health_ratio(curr_obs)
+        c["healthy_delta"] = (curr_h - prev_h) * self.cfg.healthy_service_delta_weight
+
+        prev_alerts = _alerts_count(prev_obs)
+        curr_alerts = _alerts_count(curr_obs)
+        self.state.max_alerts_seen = max(self.state.max_alerts_seen, prev_alerts, curr_alerts, 1)
+        c["alert_delta"] = (
+            (prev_alerts - curr_alerts) / float(self.state.max_alerts_seen)
+        ) * self.cfg.alert_delta_weight
+
+        # Potential-based shaping to stabilize long-horizon learning
+        prev_phi = prev_h + (1.0 - min(1.0, prev_alerts / float(self.state.max_alerts_seen)))
+        curr_phi = curr_h + (1.0 - min(1.0, curr_alerts / float(self.state.max_alerts_seen)))
+        c["potential"] = (curr_phi - prev_phi) * self.cfg.potential_weight
+
+        # Terminal shaping
+        if episode_done:
+            if curr_h >= 0.999:
+                c["terminal_restore"] = self.cfg.terminal_full_restore_bonus
+            else:
+                c["terminal_partial_restore"] = curr_h * self.cfg.terminal_partial_restore_scale
+            if not root_cause_declared:
+                c["terminal_no_declaration"] = -self.cfg.terminal_no_declaration_penalty
+            if time_elapsed_minutes > time_budget_minutes:
+                c["terminal_over_budget"] = -self.cfg.terminal_over_budget_penalty
+
+        total = sum(c.values())
+        total = max(self.cfg.min_step_reward, min(self.cfg.max_step_reward, total))
+        return RewardBreakdown(total=round(total, 3), components={k: round(v, 4) for k, v in c.items()})
+