HealthBench config check: our four spaces vs OpenAI's published numbers

We have a real anchor for this check: OpenAI's GPT-5.6 system card publishes gpt-5.5 scores for all four HealthBench variants, and gpt-5.5 is one of the models we ran. Lining our numbers up against theirs gives a per-variant verdict rather than a guess.

The result splits cleanly. The two spaces where we used the correct grader reproduce OpenAI within a point. The full HealthBench run lands at 55.8 against their 56.5, and our HealthBench Professional physician baseline lands at 43.87 against their published 43.7. That is close enough to say the dataset, prompting, rubric scoring, length adjustment and aggregation are all wired up right.

The two spaces where we used the wrong grader are badly off. HealthBench Hard is 5.5 points low and HealthBench Consensus is 13.6 points low. Both were graded by gpt-4o-mini instead of GPT-4.1, not by choice but because the inspect_evals task wrappers for those two variants do not expose a judge_model argument and silently fall back to the package default. Consensus is the loudest signal: every frontier model in OpenAI's table sits in a 94 to 96 band, and ours sits at 82.

A separate, smaller gap shows up on the inference side. Our gpt-5.5 Professional score is 4 points below OpenAI's, even though the grader is correct there. The physician baseline matching to 0.2 points rules out the grading pipeline, which points the finger at reasoning effort: we pass none, while OpenAI evaluates "at the highest reasoning effort option available via each model's API." Claude was run with extended thinking off entirely.

The anchor check

All numbers are the length-adjusted score on the 0 to 100 scale, which is what OpenAI reports as the primary metric. Their figures come from the GPT-5.6 system card Table 6, given as length-adjusted (unadjusted, mean chars). Ours are recomputed from the downloaded .eval logs.

Our Professional overall is a sample-weighted mean of the three use-case subsets (consult 236, writing 142, research 147, total 525), since we ran them as separate tasks rather than one benchmark. Red teaming is a cross-cutting slice of those same 525 examples, not a fourth use case, so it is excluded from the weighting.

Two things worth noticing beyond the scores. First, mean response lengths track closely (2175 vs 2313, 2158 vs 2289, 2143 vs 2259, 3748 vs 3818), consistently about 5% shorter but never structurally different. Since length is the one output property that would move if our prompting or harness diverged, this is good evidence the request side is faithful. Second, the two correct grader rows and the two wrong grader rows separate perfectly, which is what makes the diagnosis confident rather than speculative.

The physician baseline is the cleanest check we have

HealthBench Professional ships a set of 525 physician-written responses. Scoring them involves no model inference at all: the responses are fixed text, and the only moving part is the grader plus the aggregation. So if our number matches OpenAI's, the entire scoring half of the pipeline is verified, and any remaining discrepancy on real models has to live on the inference side.

Every cell is within a point. Note that the per-use-case rows here are recomputed from per-sample scores, not read off the log's own subset metrics, which are inflated for a reason covered in issue 4.

Config matrix

What we set against what OpenAI's reference implementation sets, per variant.

The length adjustment is worth calling out as correct-by-verification, not just correct-by-declaration: we confirmed the arithmetic per sample against score − penalty × ((len − center) / 500), OpenAI's calculate_length_adjusted_score. On one Hard sample: raw 0.6102, completion 13,776 chars, giving 0.6102 − 0.0392 × 23.552 = −0.3131, which is exactly the stored value.

Issue 1: Hard and Consensus were graded by gpt-4o-mini

This is the largest error and it was not a deliberate choice. The inspect_evals task wrappers for these two variants accept only the length-adjustment arguments and pass nothing else through, so judge_model silently keeps the package default of openai/gpt-4o-mini:

def healthbench(
    ...
    judge_model: str | Model = "openai/gpt-4o-mini",   # package default
    ...
)

@task
def healthbench_hard(
    length_adjustment_center: float | None = None,
    length_adjustment_penalty_per_500_chars: float | None = None,
) -> Task:
    return healthbench(                      # judge_model never forwarded
        subset="hard",
        length_adjustment_center=length_adjustment_center,
        length_adjustment_penalty_per_500_chars=length_adjustment_penalty_per_500_chars,
    )

OpenAI never grades any variant with a mini-tier model. In simple_evals.py the same healthbench_grading_sampler is handed to healthbench, healthbench_hard and healthbench_consensus alike, so all three get GPT-4.1, or all three get GPT-5.4 low when the --healthbench-use-gpt-5-4-low-grader flag is set.

The HealthBench paper measured what a weaker grader costs. Its meta-evaluation ranks candidate graders by agreement with physicians (Macro-F1): GPT-4.1 0.709, o4-mini 0.692, o3 0.681, GPT-4.1 mini 0.661, GPT-4.1 nano 0.580, and calls the smaller variants "substantially worse". gpt-4o-mini is not in that table but sits below GPT-4.1 mini in capability, so 0.661 is the optimistic bound.

Issue 2: reasoning effort is never set, and the models diverge because of it

The HealthBench Professional paper states that models are "evaluated at the highest reasoning effort option available via each model's API (e.g., xhigh for GPT-5.4)", and measures a 5.6 to 7.3 point gain moving from low to xhigh. None of our runs pass a reasoning effort, so each provider's default applies, and the defaults are not comparable to each other:

Claude ran with extended thinking off on every sample of every log, while gpt-5.5 and DeepSeek reasoned by default. Whatever the intended comparison was, this is not it: the reasoning models get their default budget and Claude gets none. Claude is the most disadvantaged model in the set for a reason that has nothing to do with Claude.

This also explains the one gap the grader cannot account for. On Professional, where our grader is correct and the physician baseline matches to 0.2 points, gpt-5.5 still lands 4 points low. A default-effort versus highest-effort difference of that size is consistent with the 5.6 to 7.3 point low-to-xhigh delta OpenAI reports.

Issue 3: three spaces show the unadjusted score as the headline

OpenAI reports "length-adjusted score (unadjusted, mean response length in characters)", so the adjusted number is the primary one. In our listing.json the primary_metric for hs-non-professional, hs-hard and hs-consensus is the unadjusted bootstrap_score. Only hs-prof-subsets carries use_length_adjusted: true.

This is not cosmetic. Length adjustment reorders the Hard leaderboard almost completely, because our models differ enormously in verbosity:

Claude gains because it is the only model averaging under the 2000-character centre. medgemma-27b loses 16 points. Anyone reading the space's default metric is reading a different ranking from the one OpenAI's methodology produces.

Issue 4: the Professional per-use-case subscores are inflated

The custom healthbench_professional.py emits use_case_*_score, type_*_score and difficulty_*_score metrics alongside the headline. These do not use the same aggregation as the headline and should not be quoted. They are consistent with per-sample clipping to [0, 1] before averaging, where OpenAI clips only the final mean.

The physician baseline makes the error measurable, because OpenAI publishes the ground truth for exactly these three cells:

overall mean raw                    = 44.29   # matches reported 44.29
overall mean adj                    = 43.87   # matches reported 43.87, and OpenAI's 43.7
overall mean adj, clipped per-sample= 49.71
frac. of samples with adj < 0       = 0.164

by use case          n      mean_adj   clipped_per_sample
  consult          1888      42.54          48.09
  research         1176      56.60          57.47
  writing          1136      32.91          44.39

The headline bootstrap_score is unaffected and remains correct. Only the subset breakdowns are wrong, which matters because the by-use-case split is the most quoted view of Professional results.

Issue 5: five Professional scores are floored at zero

OpenAI clips the aggregate mean to [0, 1], and our implementation follows suit, so this is faithful behaviour rather than a bug. But it means five reported numbers are all displayed as 0.00 while their true values differ by 7 points, which hides real ranking information:

Worth reporting the unclipped value alongside the clipped one for these, or at least noting that a zero means "at or below zero" rather than "scored nothing".

Run hygiene

What to change

Concretely, for the two broken variants, stop using the wrapper tasks and call the parent task instead:

# broken: judge_model silently defaults to gpt-4o-mini
inspect eval inspect_evals/healthbench_hard \
  -T length_adjustment_center=2000 \
  -T length_adjustment_penalty_per_500_chars=0.0392

# works: call healthbench() directly with subset=
inspect eval inspect_evals/healthbench \
  -T subset=hard \
  -T judge_model=openai/gpt-4.1 \
  -T length_adjustment_center=2000 \
  -T length_adjustment_penalty_per_500_chars=0.0392

Re-running with the correct grader and highest reasoning effort should move Hard up by roughly 5 points and Consensus up by roughly 13 toward the published band. If it does not, that is the signal something else is wrong. If it does, the config is settled.

Full results

Main variants, all six models

Judge: gpt-4.1 for full, gpt-4o-mini for hard and consensus. Epochs 1 throughout. All runs completed with zero errors. Only the full column is grader-comparable to the published row.

Professional subsets, length-adjusted

Red teaming values are shown unclipped so the ordering is visible; the logs report these as 0.00. DeepSeek's consult figure comes from its epochs=1 run, the rest from epochs=8. Weighted overall uses consult 236, writing 142, research 147. Only gpt-5.5 and claude-opus-4-7 beat the physician baseline overall, and both do so narrowly.

How this was checked

The four HuggingFace spaces were cloned and their .eval logs pulled as real LFS blobs (the plain clone yields 133-byte pointers). Headers gave the declared config; per-sample model events gave the config actually sent, including which model served every grading call; per-sample scores were re-aggregated independently to confirm the reported metrics and to recompute the subset breakdowns.

Local copies at /Users/kazuki/Sandbox/hb-spaces/. The hs-original space was excluded, its link is broken. Reference sources: HealthBench (arXiv:2505.08775), HealthBench Professional (arXiv:2604.27470), the GPT-5.6 Preview system card Table 6, and openai/simple-evals.