Spaces:
Runtime error
Runtime error
| title: Claude Code For Health Environment Server | |
| emoji: ⏲️ | |
| colorFrom: yellow | |
| colorTo: yellow | |
| sdk: docker | |
| pinned: false | |
| app_port: 8000 | |
| base_path: /web | |
| tags: | |
| - openenv | |
| # Claude Code for Health | |
| Medical AI is mostly tested on multiple-choice questions. This environment tests something harder: can an agent actually *do* clinical work, step by step, the way a real clinician does? | |
| Three task types, 194,000+ real cases, fully programmatic grading. | |
| ## Motivation | |
| Medical errors are the third leading cause of death in the US, yet most AI benchmarks for medicine (MedQA, USMLE, PubMedQA) hand the model a complete question and ask for a single answer. That measures pattern recognition, not judgment. | |
| Real clinical work looks nothing like that. A clinician doesn't receive a pre-packaged summary — they read a history, order the right tests, interpret results in context, update their hypothesis, and commit to a plan. The reasoning happens over time, with incomplete information at every step. | |
| This environment forces that workflow. The agent interacts with a CLI — the same metaphor used by Claude Code, aider, and Codex for software engineering — but applied to patient care. It can only see what it explicitly requests, and it's graded on the full trajectory: what it looked up, how efficiently it got there, and whether its final answer was right. | |
| ## Architecture | |
|  | |
| ## Tasks | |
| | Task | Difficulty | Description | Dataset | Cases | Est. Frontier Score | | |
| |---|---|---|---|---|---| | |
| | **Clinical Note Review** | Easy | Read a clinical note, identify errors, correct them or approve | MEDEC | 55,394 | 0.70–0.85 | | |
| | **Medical Calculation** | Medium | Read a patient scenario, identify the formula, compute the answer | MedCalc-Bench | ~138,000 | 0.45–0.65 | | |
| | **Diagnostic Workup** | Hard | Explore a patient chart via CLI tools, build a differential, confirm diagnosis | MedCaseReasoning | 766 | 0.30–0.50 | | |
| ## Datasets | |
| - **MEDEC** — 55,394 clinical notes with annotated errors and corrections across 3 splits (training: 36,766 / validation: 9,129 / test: 9,499) | |
| - **MedCalc-Bench** — ~138,000 medical calculation problems with ground truth answers and tolerance bounds (train: 136,969 / test: 16,521; numeric-answer rows only) | |
| - **MedCaseReasoning** — 766 structured clinical cases with demographics, vitals, labs, imaging, physical exam, and ground truth diagnoses (JSONL) | |
| Each inference run picks one random case per difficulty level — the full pools are available for training and evaluation at scale. | |
| ## Prior Work & Novelty | |
| Most medical AI benchmarks test static knowledge: MedQA, USMLE, and PubMedQA all hand the model a complete question and ask for a single answer. That measures pattern recognition on full context, not the sequential decision-making that clinical work actually requires. | |
| The closest sequential benchmark is **SDBench** (Nori et al., 2025 — [arXiv:2506.22405](https://arxiv.org/abs/2506.22405)), which converts 304 NEJM case records into interactive diagnosis. Physicians scored only ~36% on hard cases, confirming that sequential reasoning under information constraints is a genuine challenge. | |
| This environment extends that line of work with three key additions: | |
| 1. **Multi-skill integration** — note error correction, quantitative reasoning, and sequential diagnosis in one unified CLI interface, rather than diagnosis alone | |
| 2. **Developer-native interaction** — pure CLI commands (exactly like Claude Code / aider / Codex), making this the first medical environment optimized for tool-use agents that already excel at software tasks | |
| 3. **194,000+ real cases** from MEDEC, MedCalc-Bench, and MedCaseReasoning — not curated vignettes; dense per-step rewards mirror real clinical cost-efficiency trade-offs | |
| ## Action / Observation Space | |
| **Action** — one CLI command string per step: | |
| ```python | |
| class MedAction(Action): | |
| command: str = Field(..., description="CLI command string, e.g. 'chart.labs CBC'") | |
| ``` | |
| The agent sends a single JSON object to the `/step` endpoint: | |
| ```json | |
| {"command": "chart.labs CBC"} | |
| {"command": "ddx.confirm Adult-onset Still disease"} | |
| {"command": "submit 4.5"} | |
| {"command": "note.correct 3 Patient presented with altered mental status."} | |
| ``` | |
| Example command strings: | |
| ``` | |
| chart.history # view past medical history | |
| chart.labs CBC # view CBC panel results | |
| ddx.add Pulmonary Embolism # add to differential | |
| ddx.confirm Pulmonary Embolism # submit final diagnosis (ends episode) | |
| calculate Wells_PE # declare which calculator to use | |
| submit 4.5 # submit numeric answer (ends episode) | |
| note.correct 3 Corrected text here # fix error in sentence 3 | |
| note.approve # approve note as-is (ends episode) | |
| ``` | |
| **Observation** — command output + episode context after each step: | |
| ```python | |
| class MedObservation(Observation): | |
| output: str # Command output text (chart data, lab results, etc.) | |
| error: str # Error message if command was invalid (empty string if none) | |
| available_commands: list[str] # Tools valid for the current task type | |
| task_type: str # "diagnosis" | "calculation" | "note_review" | |
| step_number: int # Current step count | |
| max_steps: int # Hard cap (50 steps per episode) | |
| done: bool # Inherited — episode complete? | |
| reward: float # Inherited — step reward | |
| metadata: dict # Inherited — additional info | |
| ``` | |
| **State** — episode tracking via GET /state: | |
| ```python | |
| class MedState(State): | |
| episode_id: str # Inherited — unique episode identifier | |
| step_count: int # Inherited — total steps taken | |
| task_type: str # "diagnosis" | "calculation" | "note_review" | |
| difficulty: str # "easy" | "medium" | "hard" | |
| total_score: float # Cumulative reward accumulated so far | |
| commands_issued: int # Number of commands sent | |
| is_submitted: bool # Whether the agent has submitted a final answer | |
| ``` | |
| ## Available Tools | |
| The environment simulates a real CLI tool interface - the same interaction pattern used by Claude Code, OpenCode, and Codex CLI for software engineering, but applied to clinical medicine. The agent issues text commands one at a time, receives structured output, and decides what to do next. No menus, no dropdowns - just a terminal and clinical judgment. | |
| ### Diagnosis Tools | |
| ``` | |
| chart.history View past medical history, medications, allergies | |
| chart.vitals View vital signs | |
| chart.labs [panel] View lab results (list panels or view specific) | |
| chart.imaging [type] View imaging findings | |
| chart.exam [system] View physical exam findings | |
| chart.medications View current medications | |
| chart.allergies View known allergies | |
| ddx.add <diagnosis> Add to differential | |
| ddx.remove <diagnosis> Remove from differential | |
| ddx.list Show current differential | |
| ddx.confirm <diagnosis> Submit final diagnosis (ends episode) | |
| ``` | |
| ### Calculation Tools | |
| ``` | |
| case.read Read the full patient note + question | |
| calculate <name> Declare which calculator you're using | |
| submit <number> Submit numeric answer (ends episode) | |
| ``` | |
| ### Note Review Tools | |
| ``` | |
| note.read Read the clinical note with numbered sentences | |
| note.correct <id> <text> Correct a sentence by ID | |
| note.approve Approve note / submit corrections (ends episode) | |
| ``` | |
| ### Reference Tools (all tasks) | |
| ``` | |
| reference.ranges <test> Normal range lookup (e.g. sodium, troponin) | |
| reference.criteria <condition> Diagnostic criteria (e.g. DKA, sepsis, PE) | |
| reference.drug_info <drug> Drug mechanism, indications, contraindications | |
| interpret <test> <value> Interpret a lab value against normal range | |
| ``` | |
| ## Reward Design | |
| Dense rewards over the full trajectory. Every step can yield signal, not just the terminal action. | |
| | Task | Intermediate Budget | Terminal Budget | Total | | |
| |---|---|---|---| | |
| | Note Review | 0.10 (read note) | 0.90 (detection + correction quality) | 1.0 | | |
| | Calculation | 0.15 (read case + declare calculator) | 0.85 (numeric accuracy + correct calculator + efficiency) | 1.0 | | |
| | Diagnosis | 0.30 (chart exploration credit per relevant section) | 0.70 (diagnostic accuracy + workup completeness + efficiency + reasoning) | 1.0 | | |
| **Penalties:** | |
| - Protocol violations: -0.05 (imaging without vitals, confirming with <2 differentials, specialized labs without basic panels) | |
| - Duplicate tool calls: -0.05 | |
| ## Baseline Scores | |
| Model: `meta-llama/Llama-3.1-8B-Instruct` via HuggingFace Router (20 runs, random case per difficulty): | |
| | Task | Avg Score | Min | Max | | |
| |---|---|---|---| | |
| | Easy (note review) | 0.49 | 0.19 | 0.73 | | |
| | Medium (calculation) | 0.27 | 0.01 | 0.84 | | |
| | Hard (diagnosis) | 0.22 | 0.12 | 0.41 | | |
| `inference.py` produces the exact `[START]` / `[STEP]` / `[END]` stdout format required by OpenEnv and completes all 3 tasks in under 5 minutes on standard hardware. | |
| ## Example Episode (Diagnosis - Hard) | |
| ``` | |
| > reset(options={"task": "hard"}) | |
| Patient: 45M, presenting with fever, rash, and joint pain | |
| Type 'help' for available tools. | |
| > chart.history reward: +0.02 | |
| PMH: None significant | |
| Medications: None | |
| Social: Non-smoker, occasional alcohol | |
| > chart.vitals reward: +0.02 | |
| BP: 130/85 | HR: 102 | Temp: 39.2C | RR: 18 | SpO2: 98% | |
| > chart.labs reward: 0.00 | |
| Available lab panels: CBC, BMP, inflammatory_markers, LFTs | |
| > chart.labs inflammatory_markers reward: +0.02 | |
| inflammatory_markers: | |
| ESR: 85 mm/hr | |
| CRP: 12.4 mg/dL | |
| Ferritin: 26,250 ng/mL | |
| > reference.ranges ferritin reward: 0.00 | |
| FERRITIN: Normal range 12-300 ng/mL | |
| Female 12-150, Male 12-300. Very high in HLH, Still disease | |
| > interpret ferritin 26250 reward: 0.00 | |
| FERRITIN 26250.0 ng/mL: HIGH - critically elevated (normal 12-300) | |
| Female 12-150, Male 12-300. Very high in HLH, Still disease | |
| > reference.criteria hlh reward: 0.00 | |
| HLH (HScore): Fever, organomegaly, cytopenias (2-3 lineages), | |
| hypertriglyceridemia (>=265) or hypofibrinogenemia (<=150), | |
| ferritin >=500 (often >10,000), elevated soluble CD25... | |
| > ddx.add HLH reward: 0.00 | |
| Added 'HLH'. Differential has 1 entry(ies). | |
| > ddx.add Adult-onset Still disease reward: 0.00 | |
| Added 'Adult-onset Still disease'. Differential has 2 entry(ies). | |
| > ddx.confirm Adult-onset Still disease reward: +0.34 | |
| Diagnosis submitted: 'Adult-onset Still disease'. Score: 0.34 | |
| [STATUS] DDX: [HLH, Adult-onset Still disease] | Step: 10/50 | |
| Total episode score: 0.40 | |
| ``` | |
| The agent earned intermediate rewards for each relevant chart section explored (+0.02 each), used reference tools to interpret the critically elevated ferritin (no reward, but informed its reasoning), built a 2-item differential (avoiding the -0.05 penalty), and got partial terminal credit for a close but not exact diagnosis match. | |
| ## Setup | |
| ```bash | |
| # Install | |
| uv sync | |
| # Run server | |
| uv run uvicorn server.app:app --port 8000 | |
| # Run inference (set HF_TOKEN first) | |
| export HF_TOKEN="your_token" | |
| uv run python inference.py | |
| ``` | |
| ## Docker | |
| ```bash | |
| docker build -t claude_code_for_health . | |
| docker run -p 8000:8000 claude_code_for_health | |
| ``` | |
| ## Environment Variables | |
| | Variable | Description | Default | | |
| |---|---|---| | |
| | `API_BASE_URL` | LLM endpoint | `https://router.huggingface.co/v1` | | |
| | `MODEL_NAME` | Model identifier | `meta-llama/Llama-3.1-8B-Instruct` | | |
| | `HF_TOKEN` | HuggingFace API key | (required) | | |
| | `IMAGE_NAME` | Docker image for `from_docker_image()` | (optional) | | |
| ## Project Structure | |
| ``` | |
| claude_code_for_health/ | |
| ├── Dockerfile # Container image definition | |
| ├── openenv.yaml # OpenEnv manifest | |
| ├── pyproject.toml # Dependencies | |
| ├── inference.py # Baseline inference script | |
| ├── models.py # MedAction, MedObservation, MedState | |
| ├── client.py # EnvClient wrapper | |
| ├── __init__.py # Module exports | |
| ├── data/ | |
| │ ├── MedCaseReasoning/ # Diagnosis cases (JSONL) | |
| │ ├── MedCalcBench/ # Calculation cases (CSV) | |
| │ ├── MEDEC/ # Note review cases (CSV) | |
| │ └── reference/ # Lab ranges, criteria, drug info (JSON) | |
| └── server/ | |
| ├── app.py # FastAPI application | |
| ├── claude_code_for_health_environment.py # Core environment | |
| ├── command_parser.py # CLI command parsing | |
| ├── data_loader.py # Dataset loading | |
| ├── task_configs.py # Difficulty tiers + case selection | |
| ├── graders.py # Dense reward functions | |
| ├── constants.py # Reference data loader | |
| └── ui.py # Custom Gradio dashboard | |
| ``` | |