# Ajay's HealthBench HF Spaces — local mirror

Mirror of every HealthBench-related space under `huggingface.co/ajay-citadel`.
Inventory taken 2026-08-07. Per-run detail: `MANIFEST.csv` (68 runs), raw headers: `headers_raw.json`.

Space URL pattern: `https://huggingface.co/spaces/ajay-citadel/<dir>`
Log URL pattern: `.../resolve/main/logs/<file>.eval`

| dir | space | runs | date | what it is |
|---|---|---|---|---|
| `healthbench-v1` (symlink → `inspect_ai/sandbox/healthbench-v1`) | healthbench-v1 | 7 (2 ok, 4 err/cancelled) | 07-09 | first attempt, `gpt-4o-mini` judge, no length adjustment |
| `hs-non-professional` | hs-non-professional | 6 | 07-15..24 | HealthBench full (n=5000), `gpt-4.1` judge |
| `hs-consensus` | hs-consensus | 6 | 07-16..24 | HealthBench consensus (n=3671) |
| `hs-hard` | hs-hard | 6 | 07-16..24 | HealthBench hard (n=1000) |
| `hs-original` | hs-original | 6 | 07-24..25 | HealthBench **Professional** full (n=525, epochs=8) |
| `hs-prof-subsets` | hs-prof-subsets | 26 | 07-24..25 | Professional split by use-case + physician baseline |
| `medgemma-healthbench-hard` | medgemma-healthbench-hard | 2 | 08-05 | MedGemma hard re-run, `gpt-4.1` judge |
| `healthbench-hard-medgemma-v2` | healthbench-hard-medgemma-v2 | 2 | 08-05 | same as above via `vllm/` provider prefix (near-duplicate) |
| `healthbench-consensus-medgemma` | healthbench-consensus-medgemma | 2 | 08-05 | MedGemma consensus re-run, `gpt-4.1` judge |
| `healthbench-professional-deepseek-v4-pro` | healthbench-professional-deepseek-v4-pro | 1 | 08-06 | DeepSeek Professional, first genuinely fresh generation |

Empty shells on HF (README/index.html only, no logs — nothing to mirror):
`healthbench-profes`, `healthbench-prof-part1`.

Non-HealthBench spaces by the same author, **not** mirrored here:
`inspectevals`, `tevals`, `tmevals`, `medgemma_runs`, `igakuqaruns`, `igakuqaruns-v2`,
`igakuqa-medgemma`, `situational-awareness-v1`, `anzen1`, `anzen-kimi-k3-safety-audit`,
`pair-jbb-attacker-study-extended`.
(MedQA/IgakuQA sets are already local under `inspect_ai/sandbox/MedQA/` and `.../IgakuQA/`.)

## Traps

1. **Judge model is not constant across the set** — three judges are in play:
   - `openai/gpt-4o-mini` (scorer default): `healthbench-v1` (07-09), all of `hs-hard`, all of `hs-consensus`
   - `openai/gpt-4.1`: all of `hs-non-professional`, and the 08-05 MedGemma re-runs
   - `openai/gpt-5.4`, `reasoning_effort=low`: every Professional run

   Raw `healthbench_score` is judge-dependent and is *not* touched by the length-adjustment
   args (`scorer.py:524` — it's `achieved/possible` points). So a raw-score delta between two
   runs with different judges is a judge artifact. Measured, same model + same task code:

   | | gpt-4o-mini | gpt-4.1 | Δ |
   |---|---|---|---|
   | healthbench full, GPT-5.5 | 0.4873 | 0.5687 | **+8.1** |
   | healthbench full, Opus-4.7 | 0.4762 | 0.5344 | **+5.8** |
   | consensus, MedGemma-27B | 0.7758 | 0.9105 | **+13.5** |
   | consensus, MedGemma-4B | 0.7137 | 0.7583 | +4.5 |
   | hard, MedGemma-27B | 0.2109 | 0.1415 | **−6.9** |
   | hard, MedGemma-4B | 0.1060 | −0.0347 | **−14.1** |

   gpt-4.1 is *more* generous on consensus and *harsher* on hard. Never mix judges in one table.
2. **`cache=true` throughout.** Only 23 of 68 runs actually generated candidate responses.
   27 are full cache replays (zero candidate tokens), 7 more are "mostly cached"
   (<200 candidate tokens/sample — e.g. `hs-hard` MedGemma-27B at 24 tok/sample, so its 0.2109
   rests on cached completions from a run that was never published). Column `provenance`
   in `MANIFEST.csv`; `stats.model_usage` is the underlying evidence.
3. **Professional epochs are inconsistent**: 8 for GPT-5.5 / Opus-4-7 / PLaMo / MedGemma,
   1 for DeepSeek. Do not put those in the same table without saying so.
4. **DeepSeek Professional has two conflicting numbers.** `hs-original` 07-25 = 0.3430 raw
   (but 10 candidate tok/sample → a replay), `healthbench-professional-deepseek-v4-pro` 08-06
   = 0.3781 raw (fresh, 1.26M tokens). Same nominal config. Use the 08-06 one; treat the
   ~3.5-point gap as the reproducibility floor for a cached-vs-fresh comparison.
5. **Professional subset metrics inside the log are wrong** (discard length adjustment, clip
   per sample). Recompute from per-sample scores. See
   `../config-check-v2.html`.
6. **Physician baseline was judged once and replayed.** The identical 0.4429/0.4387 across
   four models in `hs-prof-subsets` is one computation, not four. (0.4387 matches OpenAI's
   published 43.7 — this is the anchor that says the harness is correct.)
7. **Professional use-case subsets don't partition the set.** consult 236 + writing 142 +
   research 147 + red-teaming 191 = 716 > 525 total, so samples carry multiple use-case tags.
   The four subset scores cannot be weight-averaged back to the full-set score.
8. **The subset matrix is incomplete.** MedGemma-4B has only `consult`; DeepSeek has no
   epochs=8 `consult`. Don't render `hs-prof-subsets` as a full grid.
9. `medgemma-healthbench-hard` and `healthbench-hard-medgemma-v2` are the same experiment run
   twice 2h apart, differing only in provider prefix (`openai-api/vllm/...` vs `vllm/...`).
   Score spread between them (0.1332 vs 0.1415) is vLLM sampling noise, ~0.8pt.