README.md

metadata

title: Claude Code For Health Environment Server
emoji: ⏲️
colorFrom: yellow
colorTo: yellow
sdk: docker
pinned: false
app_port: 8000
base_path: /web
tags:
  - openenv

Claude Code for Health

Medical AI is mostly tested on multiple-choice questions. This environment tests something harder: can an agent actually do clinical work, step by step, the way a real clinician does?

Three task types, 194,000+ real cases, fully programmatic grading.

Motivation

Medical errors are the third leading cause of death in the US, yet most AI benchmarks for medicine (MedQA, USMLE, PubMedQA) hand the model a complete question and ask for a single answer. That measures pattern recognition, not judgment.

Real clinical work looks nothing like that. A clinician doesn't receive a pre-packaged summary — they read a history, order the right tests, interpret results in context, update their hypothesis, and commit to a plan. The reasoning happens over time, with incomplete information at every step.

This environment forces that workflow. The agent interacts with a CLI — the same metaphor used by Claude Code, aider, and Codex for software engineering — but applied to patient care. It can only see what it explicitly requests, and it's graded on the full trajectory: what it looked up, how efficiently it got there, and whether its final answer was right.

Architecture

Tasks

Task	Difficulty	Description	Dataset	Cases	Est. Frontier Score
Clinical Note Review	Easy	Read a clinical note, identify errors, correct them or approve	MEDEC	55,394	0.70–0.85
Medical Calculation	Medium	Read a patient scenario, identify the formula, compute the answer	MedCalc-Bench	~138,000	0.45–0.65
Diagnostic Workup	Hard	Explore a patient chart via CLI tools, build a differential, confirm diagnosis	MedCaseReasoning	766	0.30–0.50

Datasets

• MEDEC — 55,394 clinical notes with annotated errors and corrections across 3 splits (training: 36,766 / validation: 9,129 / test: 9,499)
• MedCalc-Bench — ~138,000 medical calculation problems with ground truth answers and tolerance bounds (train: 136,969 / test: 16,521; numeric-answer rows only)
• MedCaseReasoning — 766 structured clinical cases with demographics, vitals, labs, imaging, physical exam, and ground truth diagnoses (JSONL)

Each inference run picks one random case per difficulty level — the full pools are available for training and evaluation at scale.

Prior Work & Novelty

Most medical AI benchmarks test static knowledge: MedQA, USMLE, and PubMedQA all hand the model a complete question and ask for a single answer. That measures pattern recognition on full context, not the sequential decision-making that clinical work actually requires.

The closest sequential benchmark is SDBench (Nori et al., 2025 — arXiv:2506.22405), which converts 304 NEJM case records into interactive diagnosis. Physicians scored only ~36% on hard cases, confirming that sequential reasoning under information constraints is a genuine challenge.

This environment extends that line of work with three key additions:

1. Multi-skill integration — note error correction, quantitative reasoning, and sequential diagnosis in one unified CLI interface, rather than diagnosis alone
2. Developer-native interaction — pure CLI commands (exactly like Claude Code / aider / Codex), making this the first medical environment optimized for tool-use agents that already excel at software tasks
3. 194,000+ real cases from MEDEC, MedCalc-Bench, and MedCaseReasoning — not curated vignettes; dense per-step rewards mirror real clinical cost-efficiency trade-offs

Action / Observation Space

Action — one CLI command string per step:

class MedAction(Action):
    command: str = Field(..., description="CLI command string, e.g. 'chart.labs CBC'")

The agent sends a single JSON object to the /step endpoint:

{"command": "chart.labs CBC"}
{"command": "ddx.confirm Adult-onset Still disease"}
{"command": "submit 4.5"}
{"command": "note.correct 3 Patient presented with altered mental status."}

Example command strings:

chart.history                       # view past medical history
chart.labs CBC                      # view CBC panel results
ddx.add Pulmonary Embolism          # add to differential
ddx.confirm Pulmonary Embolism      # submit final diagnosis (ends episode)
calculate Wells_PE                  # declare which calculator to use
submit 4.5                          # submit numeric answer (ends episode)
note.correct 3 Corrected text here  # fix error in sentence 3
note.approve                        # approve note as-is (ends episode)

Observation — command output + episode context after each step:

class MedObservation(Observation):
    output: str                    # Command output text (chart data, lab results, etc.)
    error: str                     # Error message if command was invalid (empty string if none)
    available_commands: list[str]  # Tools valid for the current task type
    task_type: str                 # "diagnosis" | "calculation" | "note_review"
    step_number: int               # Current step count
    max_steps: int                 # Hard cap (50 steps per episode)
    done: bool                     # Inherited — episode complete?
    reward: float                  # Inherited — step reward
    metadata: dict                 # Inherited — additional info

State — episode tracking via GET /state:

class MedState(State):
    episode_id: str        # Inherited — unique episode identifier
    step_count: int        # Inherited — total steps taken
    task_type: str         # "diagnosis" | "calculation" | "note_review"
    difficulty: str        # "easy" | "medium" | "hard"
    total_score: float     # Cumulative reward accumulated so far
    commands_issued: int   # Number of commands sent
    is_submitted: bool     # Whether the agent has submitted a final answer

Available Tools

The environment simulates a real CLI tool interface - the same interaction pattern used by Claude Code, OpenCode, and Codex CLI for software engineering, but applied to clinical medicine. The agent issues text commands one at a time, receives structured output, and decides what to do next. No menus, no dropdowns - just a terminal and clinical judgment.

Diagnosis Tools

chart.history              View past medical history, medications, allergies
chart.vitals               View vital signs
chart.labs [panel]         View lab results (list panels or view specific)
chart.imaging [type]       View imaging findings
chart.exam [system]        View physical exam findings
chart.medications          View current medications
chart.allergies            View known allergies
ddx.add <diagnosis>        Add to differential
ddx.remove <diagnosis>     Remove from differential
ddx.list                   Show current differential
ddx.confirm <diagnosis>    Submit final diagnosis (ends episode)

Calculation Tools

case.read                  Read the full patient note + question
calculate <name>           Declare which calculator you're using
submit <number>            Submit numeric answer (ends episode)

Note Review Tools

note.read                  Read the clinical note with numbered sentences
note.correct <id> <text>   Correct a sentence by ID
note.approve               Approve note / submit corrections (ends episode)

Reference Tools (all tasks)

reference.ranges <test>           Normal range lookup (e.g. sodium, troponin)
reference.criteria <condition>    Diagnostic criteria (e.g. DKA, sepsis, PE)
reference.drug_info <drug>        Drug mechanism, indications, contraindications
interpret <test> <value>          Interpret a lab value against normal range

Reward Design

Dense rewards over the full trajectory. Every step can yield signal, not just the terminal action.

Task	Intermediate Budget	Terminal Budget	Total
Note Review	0.10 (read note)	0.90 (detection + correction quality)	1.0
Calculation	0.15 (read case + declare calculator)	0.85 (numeric accuracy + correct calculator + efficiency)	1.0
Diagnosis	0.30 (chart exploration credit per relevant section)	0.70 (diagnostic accuracy + workup completeness + efficiency + reasoning)	1.0

Penalties:

• Protocol violations: -0.05 (imaging without vitals, confirming with <2 differentials, specialized labs without basic panels)
• Duplicate tool calls: -0.05

Baseline Scores

Model: meta-llama/Llama-3.1-8B-Instruct via HuggingFace Router (20 runs, random case per difficulty):

Task	Avg Score	Min	Max
Easy (note review)	0.49	0.19	0.73
Medium (calculation)	0.27	0.01	0.84
Hard (diagnosis)	0.22	0.12	0.41

inference.py produces the exact [START] / [STEP] / [END] stdout format required by OpenEnv and completes all 3 tasks in under 5 minutes on standard hardware.

Example Episode (Diagnosis - Hard)

> reset(options={"task": "hard"})
Patient: 45M, presenting with fever, rash, and joint pain
Type 'help' for available tools.

> chart.history                                        reward: +0.02
PMH: None significant
Medications: None
Social: Non-smoker, occasional alcohol

> chart.vitals                                         reward: +0.02
BP: 130/85 | HR: 102 | Temp: 39.2C | RR: 18 | SpO2: 98%

> chart.labs                                           reward: 0.00
Available lab panels: CBC, BMP, inflammatory_markers, LFTs

> chart.labs inflammatory_markers                      reward: +0.02
inflammatory_markers:
  ESR: 85 mm/hr
  CRP: 12.4 mg/dL
  Ferritin: 26,250 ng/mL

> reference.ranges ferritin                            reward: 0.00
FERRITIN: Normal range 12-300 ng/mL
  Female 12-150, Male 12-300. Very high in HLH, Still disease

> interpret ferritin 26250                             reward: 0.00
FERRITIN 26250.0 ng/mL: HIGH - critically elevated (normal 12-300)
  Female 12-150, Male 12-300. Very high in HLH, Still disease

> reference.criteria hlh                               reward: 0.00
HLH (HScore): Fever, organomegaly, cytopenias (2-3 lineages),
hypertriglyceridemia (>=265) or hypofibrinogenemia (<=150),
ferritin >=500 (often >10,000), elevated soluble CD25...

> ddx.add HLH                                         reward: 0.00
Added 'HLH'. Differential has 1 entry(ies).

> ddx.add Adult-onset Still disease                    reward: 0.00
Added 'Adult-onset Still disease'. Differential has 2 entry(ies).

> ddx.confirm Adult-onset Still disease                reward: +0.34
Diagnosis submitted: 'Adult-onset Still disease'. Score: 0.34

[STATUS] DDX: [HLH, Adult-onset Still disease] | Step: 10/50
Total episode score: 0.40

The agent earned intermediate rewards for each relevant chart section explored (+0.02 each), used reference tools to interpret the critically elevated ferritin (no reward, but informed its reasoning), built a 2-item differential (avoiding the -0.05 penalty), and got partial terminal credit for a close but not exact diagnosis match.

Setup

# Install
uv sync

# Run server
uv run uvicorn server.app:app --port 8000

# Run inference (set HF_TOKEN first)
export HF_TOKEN="your_token"
uv run python inference.py

Docker

docker build -t claude_code_for_health .
docker run -p 8000:8000 claude_code_for_health

Environment Variables

Variable	Description	Default
API_BASE_URL	LLM endpoint	https://router.huggingface.co/v1
MODEL_NAME	Model identifier	meta-llama/Llama-3.1-8B-Instruct
HF_TOKEN	HuggingFace API key	(required)
IMAGE_NAME	Docker image for from_docker_image()	(optional)

Project Structure

claude_code_for_health/
├── Dockerfile              # Container image definition
├── openenv.yaml            # OpenEnv manifest
├── pyproject.toml          # Dependencies
├── inference.py            # Baseline inference script
├── models.py               # MedAction, MedObservation, MedState
├── client.py               # EnvClient wrapper
├── __init__.py             # Module exports
├── data/
│   ├── MedCaseReasoning/   # Diagnosis cases (JSONL)
│   ├── MedCalcBench/       # Calculation cases (CSV)
│   ├── MEDEC/              # Note review cases (CSV)
│   └── reference/          # Lab ranges, criteria, drug info (JSON)
└── server/
    ├── app.py              # FastAPI application
    ├── claude_code_for_health_environment.py  # Core environment
    ├── command_parser.py   # CLI command parsing
    ├── data_loader.py      # Dataset loading
    ├── task_configs.py     # Difficulty tiers + case selection
    ├── graders.py          # Dense reward functions
    ├── constants.py        # Reference data loader
    └── ui.py               # Custom Gradio dashboard