data/INDEX.md

Ajay's HealthBench HF Spaces — local mirror

Mirror of every HealthBench-related space under huggingface.co/ajay-citadel. Inventory taken 2026-08-07. Per-run detail: MANIFEST.csv (68 runs), raw headers: headers_raw.json.

Space URL pattern: https://huggingface.co/spaces/ajay-citadel/<dir> Log URL pattern: .../resolve/main/logs/<file>.eval

dir	space	runs	date	what it is
healthbench-v1 (symlink → inspect_ai/sandbox/healthbench-v1)	healthbench-v1	7 (2 ok, 4 err/cancelled)	07-09	first attempt, gpt-4o-mini judge, no length adjustment
hs-non-professional	hs-non-professional	6	07-15..24	HealthBench full (n=5000), gpt-4.1 judge
hs-consensus	hs-consensus	6	07-16..24	HealthBench consensus (n=3671)
hs-hard	hs-hard	6	07-16..24	HealthBench hard (n=1000)
hs-original	hs-original	6	07-24..25	HealthBench Professional full (n=525, epochs=8)
hs-prof-subsets	hs-prof-subsets	26	07-24..25	Professional split by use-case + physician baseline
medgemma-healthbench-hard	medgemma-healthbench-hard	2	08-05	MedGemma hard re-run, gpt-4.1 judge
healthbench-hard-medgemma-v2	healthbench-hard-medgemma-v2	2	08-05	same as above via vllm/ provider prefix (near-duplicate)
healthbench-consensus-medgemma	healthbench-consensus-medgemma	2	08-05	MedGemma consensus re-run, gpt-4.1 judge
healthbench-professional-deepseek-v4-pro	healthbench-professional-deepseek-v4-pro	1	08-06	DeepSeek Professional, first genuinely fresh generation

Empty shells on HF (README/index.html only, no logs — nothing to mirror): healthbench-profes, healthbench-prof-part1.

Non-HealthBench spaces by the same author, not mirrored here: inspectevals, tevals, tmevals, medgemma_runs, igakuqaruns, igakuqaruns-v2, igakuqa-medgemma, situational-awareness-v1, anzen1, anzen-kimi-k3-safety-audit, pair-jbb-attacker-study-extended. (MedQA/IgakuQA sets are already local under inspect_ai/sandbox/MedQA/ and .../IgakuQA/.)

Traps

1. Judge model is not constant across the set — three judges are in play:

   • openai/gpt-4o-mini (scorer default): healthbench-v1 (07-09), all of hs-hard, all of hs-consensus
   • openai/gpt-4.1: all of hs-non-professional, and the 08-05 MedGemma re-runs
   • openai/gpt-5.4, reasoning_effort=low: every Professional run

   Raw healthbench_score is judge-dependent and is not touched by the length-adjustment args (scorer.py:524 — it's achieved/possible points). So a raw-score delta between two runs with different judges is a judge artifact. Measured, same model + same task code:

   	gpt-4o-mini	gpt-4.1	Δ
   healthbench full, GPT-5.5	0.4873	0.5687	+8.1
   healthbench full, Opus-4.7	0.4762	0.5344	+5.8
   consensus, MedGemma-27B	0.7758	0.9105	+13.5
   consensus, MedGemma-4B	0.7137	0.7583	+4.5
   hard, MedGemma-27B	0.2109	0.1415	−6.9
   hard, MedGemma-4B	0.1060	−0.0347	−14.1

   gpt-4.1 is more generous on consensus and harsher on hard. Never mix judges in one table.

2. cache=true throughout. Only 23 of 68 runs actually generated candidate responses. 27 are full cache replays (zero candidate tokens), 7 more are "mostly cached" (<200 candidate tokens/sample — e.g. hs-hard MedGemma-27B at 24 tok/sample, so its 0.2109 rests on cached completions from a run that was never published). Column provenance in MANIFEST.csv; stats.model_usage is the underlying evidence.

3. Professional epochs are inconsistent: 8 for GPT-5.5 / Opus-4-7 / PLaMo / MedGemma, 1 for DeepSeek. Do not put those in the same table without saying so.

4. DeepSeek Professional has two conflicting numbers. hs-original 07-25 = 0.3430 raw (but 10 candidate tok/sample → a replay), healthbench-professional-deepseek-v4-pro 08-06 = 0.3781 raw (fresh, 1.26M tokens). Same nominal config. Use the 08-06 one; treat the ~3.5-point gap as the reproducibility floor for a cached-vs-fresh comparison.

5. Professional subset metrics inside the log are wrong (discard length adjustment, clip per sample). Recompute from per-sample scores. See ../config-check-v2.html.

6. Physician baseline was judged once and replayed. The identical 0.4429/0.4387 across four models in hs-prof-subsets is one computation, not four. (0.4387 matches OpenAI's published 43.7 — this is the anchor that says the harness is correct.)

7. Professional use-case subsets don't partition the set. consult 236 + writing 142 + research 147 + red-teaming 191 = 716 > 525 total, so samples carry multiple use-case tags. The four subset scores cannot be weight-averaged back to the full-set score.

8. The subset matrix is incomplete. MedGemma-4B has only consult; DeepSeek has no epochs=8 consult. Don't render hs-prof-subsets as a full grid.

9. medgemma-healthbench-hard and healthbench-hard-medgemma-v2 are the same experiment run twice 2h apart, differing only in provider prefix (openai-api/vllm/... vs vllm/...). Score spread between them (0.1332 vs 0.1415) is vLLM sampling noise, ~0.8pt.