--- title: HealthBench Eval Logs emoji: 🩺 colorFrom: gray colorTo: red sdk: static pinned: false short_description: 68 Inspect eval logs, 6 models, 8 HealthBench variants --- # HealthBench eval logs Every HealthBench run we have, as Inspect `.eval` logs, in one browsable place. **6 models** GPT-5.5, Claude Opus 4.7, DeepSeek-V4-Pro, PLaMo-3.0-Prime, MedGemma-27B-text-it, MedGemma-4B-it **8 benches** HealthBench full / consensus / hard, and Professional plus its four use-case slices (consult, writing, research, red-teaming) **68 logs**, 2.3 GB, runs dated 2026-07-09 to 2026-08-06 ## Layout | Path | What it is | |---|---| | `index.html` | Start here. Every log in a filterable table with direct download links. | | `viewer/` | The Inspect log viewer, all 68 logs loaded. Click into samples and judge verdicts. | | `viewer/logs/*.eval` | The raw logs. Named `____.eval`. | | `matrix.html` | Model x bench coverage matrix: what is complete, what is off-config, what is missing. | | `config-check-v2.html` | Do our numbers reproduce OpenAI's published results? | | `config-check-v1.html` | Earlier pass, superseded by v2, kept for history. | | `data/log_mapping.csv` | Maps each renamed log back to its original space and filename. | | `data/MANIFEST.csv` | Per-run header dump: judge, epochs, token counts, package versions. | | `data/headers_raw.json` | The same, unflattened. | | `data/INDEX.md` | Short version of the traps list. | ## Reading a log directly ```python from inspect_ai.log import read_eval_log log = read_eval_log("professional__gpt-5.5__2026-07-24.eval") print(log.eval.model, log.results.scores[0].metrics["mean"].value) ``` Or fetch one without cloning the whole Space: ``` https://huggingface.co/spaces/kirby44/healthbench-eval-logs/resolve/main/viewer/logs/.eval ``` ## Four things that will bite you 1. **The judge is not constant.** `gpt-4o-mini` on hard and consensus, `gpt-4.1` on full and the Aug-05 MedGemma re-runs, `gpt-5.4` on all Professional. Swapping the judge moves a score by up to 14 points, and not always in the same direction. Only compare runs that share a judge. 2. **Professional epochs are inconsistent.** 8 for most models, 1 for DeepSeek. 3. **In-log subset metrics are wrong.** `use_case_*_score`, `specialty_*_score`, `difficulty_*_score` and `source_slice_*_score` discard the length adjustment and clip each sample to [0,1] before averaging. Errors reach +32 points, always upward. Use the standalone `professional-*` logs for the use-case slices; re-aggregate the other axes from per-sample scores. 4. **`cache=true` on every run.** A `-replay` or `-cached` suffix in the filename means responses came from Inspect's cache. An empty `stats.model_usage` is a replay, not a run. The pipeline itself is validated: the Professional physician baseline lands at 43.9 against OpenAI's published 43.7. ## Provenance Runs executed by Ajay (`ajay-citadel`) between 2026-07-09 and 2026-08-06, originally published as ten separate Spaces. This Space consolidates them into one viewer, renames the logs so the config is legible from the filename, and adds the provenance classification the raw logs do not carry. The logs are byte-identical to the originals; `data/log_mapping.csv` maps every file back to its source.