Hugging Face's logo

Spaces:
kirby44
/
healthbench-eval-logs
Running

App Files Community
healthbench-eval-logs / matrix.html
kirby44's picture
kirby44
Hub index, coverage matrix, config-check reports, and a Professional-runs log viewer
c189142 verified
Raw
History Blame Contribute Delete
33.7 kB
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<link rel="icon" href="favicon.svg">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>HealthBench coverage: which model ran which bench</title>
<style>
 :root {
 --ivory: #FAF9F5;
 --slate: #141413;
 --clay: #D97757;
 --oat: #E3DACC;
 --olive: #788C5D;
 --gray-150:#F0EEE6;
 --gray-300:#D1CFC5;
 --gray-500:#87867F;
 --gray-700:#3D3D3A;
 --serif: ui-serif, Georgia, "Times New Roman", serif;
 --sans: system-ui, -apple-system, "Segoe UI", Roboto, sans-serif;
 --mono: ui-monospace, "SF Mono", Menlo, Consolas, monospace;
 }
 * { box-sizing: border-box; margin: 0; padding: 0; }
 html { scroll-behavior: smooth; }
 body {
 background: var(--ivory);
 color: var(--gray-700);
 font-family: var(--sans);
 font-size: 15px;
 line-height: 1.65;
 -webkit-font-smoothing: antialiased;
 padding: 56px 24px 120px;
 }
 .page { max-width: 1160px; margin: 0 auto; display: grid; grid-template-columns: 200px minmax(0, 1fr); gap: 48px; }
 @media (max-width: 920px) { .page { grid-template-columns: 1fr; } nav { display: none; } }
 nav { position: sticky; top: 32px; align-self: start; font-size: 13px; }
 nav .label { font-family: var(--mono); font-size: 10px; letter-spacing: 0.1em; text-transform: uppercase; color: var(--gray-500); margin-bottom: 12px; }
 nav a { display: block; padding: 5px 0 5px 12px; border-left: 2px solid var(--gray-300); color: var(--gray-700); text-decoration: none; }
 nav a:hover { color: var(--slate); border-color: var(--slate); }
 nav a.l2 { padding-left: 24px; font-size: 12.5px; color: var(--gray-500); }
 nav .files { margin-top: 28px; border-top: 1px solid var(--gray-300); padding-top: 16px; }
 nav .files a { text-decoration: none; }
 nav .files code { display: block; font-family: var(--mono); font-size: 11px; color: var(--gray-500); padding: 3px 0; }
 header { margin-bottom: 12px; }
 .eyebrow { font-family: var(--mono); font-size: 11px; letter-spacing: 0.08em; text-transform: uppercase; color: var(--gray-500); margin-bottom: 10px; }
 h1 { font-family: var(--serif); font-weight: 500; font-size: 32px; color: var(--slate); letter-spacing: -0.01em; margin-bottom: 14px; }
 .dateline { font-family: var(--mono); font-size: 11px; letter-spacing: 0.04em; color: var(--gray-500); margin-top: -4px; }
 h2 { font-family: var(--serif); font-weight: 500; font-size: 22px; color: var(--slate); margin: 40px 0 14px; scroll-margin-top: 24px; }
 h3 { font-family: var(--serif); font-weight: 500; font-size: 17px; color: var(--slate); margin: 22px 0 8px; }
 p { margin-bottom: 12px; max-width: 680px; }
 ul, ol { margin: 0 0 12px 22px; max-width: 680px; }
 li { margin-bottom: 6px; }
 code { font-family: var(--mono); font-size: 13px; }
 a { color: var(--slate); text-decoration: underline; text-decoration-color: var(--gray-300); }
 a:hover { text-decoration-color: var(--clay); }
 details { border: 1.5px solid var(--gray-300); border-radius: 10px; background: #fff; margin: 14px 0; overflow: hidden; }
 summary { list-style: none; cursor: pointer; padding: 14px 16px; font-family: var(--serif); font-size: 16px; color: var(--slate); display: flex; align-items: baseline; gap: 10px; }
 summary::-webkit-details-marker { display: none; }
 summary::before { content: "\25B8"; color: var(--clay); font-family: var(--sans); font-size: 12px; transition: transform 120ms; }
 details[open] summary::before { transform: rotate(90deg); }
 summary .where { font-family: var(--mono); font-size: 11px; color: var(--gray-500); margin-left: auto; }
 details .body { padding: 0 16px 16px; }
 details .body p { font-size: 14px; }
 details .body pre { background: var(--gray-150); padding: 12px 14px; border-radius: 6px; font-family: var(--mono); font-size: 12px; line-height: 1.55; overflow-x: auto; margin-top: 8px; }
 .callout { display: flex; gap: 12px; border: 1.5px solid var(--oat); background: rgba(227,218,204,0.35); border-radius: 10px; padding: 14px 16px; margin: 18px 0; font-size: 14px; max-width: 760px; }
 .callout .ico { color: var(--clay); font-weight: 600; }
 .callout p { margin-bottom: 6px; }
 .callout p:last-child { margin-bottom: 0; }
 table { border-collapse: collapse; width: 100%; max-width: 760px; margin: 14px 0; font-size: 14px; }
 th, td { padding: 8px 12px; border-bottom: 1px solid var(--gray-300); text-align: left; vertical-align: top; }
 th { font-family: var(--mono); font-size: 11px; letter-spacing: 0.06em; text-transform: uppercase; color: var(--gray-500); font-weight: 600; border-bottom: 1.5px solid var(--gray-300); }
 td.num { text-align: right; font-variant-numeric: tabular-nums; }
 /* the coverage matrix */
 .mwrap { overflow-x: auto; margin: 16px 0 6px; border: 1.5px solid var(--gray-300); border-radius: 10px; background: #fff; }
 table.matrix { max-width: none; width: 100%; margin: 0; font-size: 13px; }
 table.matrix th, table.matrix td { border-bottom: 1px solid var(--gray-300); padding: 6px 7px; }
 table.matrix thead th { background: var(--gray-150); text-align: right; font-size: 10px; letter-spacing: 0.02em; white-space: nowrap; }
 table.matrix thead th.rowhead { text-align: left; }
 table.matrix thead th .jd { display: block; font-family: var(--mono); font-size: 9px; letter-spacing: 0; text-transform: none; color: var(--gray-500); font-weight: 400; }
 table.matrix tbody th { font-family: var(--sans); font-size: 12.5px; font-weight: 500; color: var(--slate); text-transform: none; letter-spacing: 0; white-space: nowrap; }
 table.matrix td { text-align: right; font-family: var(--mono); font-variant-numeric: tabular-nums; font-size: 12px; white-space: nowrap; }
 table.matrix tbody tr:last-child th, table.matrix tbody tr:last-child td { border-bottom: none; }
 td.st-ok { background: rgba(120,140,93,0.11); color: var(--slate); }
 td.st-off { background: rgba(217,119,87,0.13); color: var(--slate); }
 td.st-gap { background: repeating-linear-gradient(135deg, transparent, transparent 4px, rgba(135,134,127,0.14) 4px, rgba(135,134,127,0.14) 5px); color: var(--gray-500); text-align: center; }
 td .mk { color: var(--clay); font-size: 10px; vertical-align: super; }
 td .mkc { color: var(--gray-500); font-size: 10px; vertical-align: super; }
 tr.baseline th, tr.baseline td { background: var(--gray-150); color: var(--gray-500); font-style: normal; }
 .mlegend { font-family: var(--mono); font-size: 11px; color: var(--gray-500); margin: 0 0 4px; max-width: none; line-height: 1.9; }
 .sw { display: inline-block; width: 10px; height: 10px; border-radius: 2px; vertical-align: -1px; margin-right: 4px; }
 .sw-ok { background: rgba(120,140,93,0.5); }
 .sw-off { background: rgba(217,119,87,0.55); }
 .sw-gap { background: repeating-linear-gradient(135deg, transparent, transparent 3px, rgba(135,134,127,0.5) 3px, rgba(135,134,127,0.5) 4px); border: 1px solid var(--gray-300); }
 .ok { color: var(--olive); font-weight: 600; }
 .warn { color: var(--clay); font-weight: 600; }
 .bad { color: #B0533A; font-weight: 600; }
 .ladder { display: grid; grid-template-columns: repeat(8, 1fr); gap: 6px; margin: 14px 0 4px; max-width: 760px; }
 .ladder .chip { border: 1px solid var(--gray-300); border-radius: 6px; background: #fff; padding: 8px 0; text-align: center; font-family: var(--mono); font-size: 10.5px; color: var(--gray-500); }
 .ladder .chip.hit { border-color: var(--olive); color: var(--slate); background: rgba(120,140,93,0.10); font-weight: 600; }
 .ladder-legend { font-family: var(--mono); font-size: 11px; color: var(--gray-500); margin-top: 2px; }
 @media (max-width: 720px) { .ladder { grid-template-columns: repeat(4, 1fr); } }
 dl.faq { margin-top: 8px; }
 dl.faq dt { font-family: var(--serif); font-size: 16px; color: var(--slate); margin-top: 18px; }
 dl.faq dd { font-size: 14px; margin: 4px 0 0; max-width: 640px; }
</style>
<style id="hbnav-css">
 .hbnav{position:sticky;top:0;z-index:50;display:flex;flex-wrap:wrap;align-items:center;gap:4px;
 padding:8px 16px;margin:-56px -24px 40px;background:rgba(250,249,245,.94);
 backdrop-filter:saturate(180%) blur(8px);border-bottom:1px solid #D1CFC5;
 font-family:ui-monospace,"SF Mono",Menlo,Consolas,monospace;font-size:11.5px}
 .hbnav .home{color:#87867F;margin-right:6px;text-decoration:none}
 .hbnav .home:hover{color:#141413}
 .hbnav a.pill{color:#3D3D3A;text-decoration:none;padding:3px 9px;border:1px solid #D1CFC5;border-radius:6px}
 .hbnav a.pill:hover{border-color:#D97757;color:#141413}
 .hbnav a.pill.here{background:#141413;border-color:#141413;color:#FAF9F5}
 .hbnav .sp{flex:1}
 @media(max-width:640px){.hbnav{margin:-56px -24px 28px}}
</style>
</head>
<body>
<div class="hbnav"><a class="home" href="index.html">&#8592; HealthBench logs</a><a class="pill here" href="matrix.html">Coverage matrix</a><a class="pill" href="config-check-v2.html">Config check v2</a><a class="pill" href="config-check-v1.html">Config check v1</a><span class="sp"></span><a class="pill" href="viewer/index.html">Log viewer &rarr;</a></div>
<div class="page">
<nav>
 <div class="label">On this page</div>
 <a href="#picture">The coverage picture</a>
 <a href="#matrix">Model &times; bench matrix</a>
 <a href="#matrix" class="l2">Reading the cells</a>
 <a href="#complete">What is already complete</a>
 <a href="#todo">What to run next</a>
 <a href="#todo" class="l2">5 required runs</a>
 <a href="#todo" class="l2">The gpt-4.1 decision</a>
 <a href="#axes">Slices with no clean data</a>
 <a href="#why">Why cells disagree</a>
 <a href="#noise">How big is a real gap</a>
 <a href="#files">Where everything lives</a>
 <a href="#faq">Questions</a>
 <div class="files">
 <div class="label">Run artifacts</div>
 <a href="data/MANIFEST.csv"><code>data/MANIFEST.csv</code></a>
 <a href="data/log_mapping.csv"><code>data/log_mapping.csv</code></a>
 <a href="data/INDEX.md"><code>data/INDEX.md</code></a>
 <a href="data/headers_raw.json"><code>data/headers_raw.json</code></a>
 <a href="viewer/index.html"><code>viewer/logs/*.eval</code></a>
 </div>
 </nav>
<main>
 <header>
 <div class="eyebrow">HealthBench &middot; model coverage audit</div>
 <h1>Which model ran which bench, and what is still missing</h1>
 <div class="dateline">Created 2026-08-07</div>
 </header>
<h2 id="picture">The coverage picture</h2>
<p><b>Nominally the matrix is nearly full: 45 of 48 model &times; bench cells have a run.
 Usably, only 3 of the 8 benches have a complete, apples-to-apples set of all 6 models.</b>
 The blockers are not missing runs, they are config drift: the grader model changes between
 benches, and <code>epochs</code> changes between models on the same bench.</p>
<p>Six models were evaluated: GPT-5.5, Claude Opus 4.7, DeepSeek-V4-Pro, PLaMo-3.0-Prime,
 MedGemma-27B-text-it and MedGemma-4B-it. Eight benches: HealthBench full, consensus and hard,
 plus HealthBench Professional whole-set and its four use-case slices (consult, writing,
 research, red-teaming). 68 logs across 10 HuggingFace spaces.</p>
<p>The good news for a comprehensive story: only <b>5 runs</b> are strictly required to close
 every real gap, and 3 of those 5 are grader-only passes that reuse cached model responses, so
 they are cheap. One further decision (what to do about a half-finished
 <code>gpt-4.1</code> re-grade) is worth 8 more runs or a deletion, your call.</p>
<h2 id="matrix">Model &times; bench matrix</h2>
<p>Scores are the <b>length-adjusted</b> HealthBench score &times; 100, which is the headline
 metric OpenAI reports. The last four columns are the Professional use-case slices. Each column
 header carries its judge model (<code>4.1</code> = <code>openai/gpt-4.1</code>,
 <code>4o-mini</code> = <code>openai/gpt-4o-mini</code>, <code>5.4</code> =
 <code>openai/gpt-5.4</code>) and sample count: columns are internally consistent but
 <b>not comparable to each other</b>.</p>
<div class="mwrap">
 <table class="matrix">
 <thead>
 <tr>
 <th class="rowhead">Model</th>
 <th>full<span class="jd">4.1 &middot; 5000</span></th>
 <th>consens.<span class="jd">4o-mini &middot; 3671</span></th>
 <th>hard<span class="jd">4o-mini &middot; 1000</span></th>
 <th>prof<span class="jd">5.4 &middot; 525</span></th>
 <th>consult<span class="jd">5.4 &middot; 236</span></th>
 <th>writing<span class="jd">5.4 &middot; 142</span></th>
 <th>research<span class="jd">5.4 &middot; 147</span></th>
 <th>red&#8209;team<span class="jd">5.4 &middot; 191</span></th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <th>GPT-5.5</th>
 <td class="st-ok">55.8<span class="mkc">c</span></td>
 <td class="st-ok">82.0<span class="mkc">c</span></td>
 <td class="st-ok">26.0<span class="mkc">c</span></td>
 <td class="st-ok">47.8</td>
 <td class="st-ok">48.6<span class="mkc">c</span></td>
 <td class="st-ok">36.0<span class="mkc">c</span></td>
 <td class="st-ok">57.9<span class="mkc">c</span></td>
 <td class="st-ok">28.2<span class="mkc">c</span></td>
 </tr>
 <tr>
 <th>Opus 4.7</th>
 <td class="st-ok">54.3<span class="mkc">c</span></td>
 <td class="st-ok">80.2<span class="mkc">c</span></td>
 <td class="st-ok">27.8<span class="mkc">c</span></td>
 <td class="st-ok">48.0</td>
 <td class="st-ok">47.0<span class="mkc">c</span></td>
 <td class="st-ok">36.2<span class="mkc">c</span></td>
 <td class="st-ok">61.1<span class="mkc">c</span></td>
 <td class="st-ok">26.7<span class="mkc">c</span></td>
 </tr>
 <tr>
 <th>DeepSeek-V4-Pro</th>
 <td class="st-ok">41.7</td>
 <td class="st-ok">78.5<span class="mkc">c</span></td>
 <td class="st-ok">13.8<span class="mkc">c</span></td>
 <td class="st-off">31.0<span class="mk">&#42;</span></td>
 <td class="st-off">25.6<span class="mk">&#42;</span><span class="mkc">c</span></td>
 <td class="st-ok">5.0</td>
 <td class="st-ok">52.9</td>
 <td class="st-ok">&minus;6.9</td>
 </tr>
 <tr>
 <th>PLaMo 3.0 Prime</th>
 <td class="st-ok">32.4</td>
 <td class="st-ok">74.8<span class="mkc">c</span></td>
 <td class="st-ok">9.6<span class="mkc">c</span></td>
 <td class="st-ok">13.7</td>
 <td class="st-ok">15.4<span class="mkc">c</span></td>
 <td class="st-ok">&minus;4.3<span class="mkc">c</span></td>
 <td class="st-ok">28.6<span class="mkc">c</span></td>
 <td class="st-ok">&minus;11.8<span class="mkc">c</span></td>
 </tr>
 <tr>
 <th>MedGemma 27B</th>
 <td class="st-ok">33.2</td>
 <td class="st-ok">76.6</td>
 <td class="st-ok">4.8<span class="mkc">p</span></td>
 <td class="st-ok">20.0</td>
 <td class="st-ok">17.8<span class="mkc">p</span></td>
 <td class="st-ok">9.1<span class="mkc">p</span></td>
 <td class="st-ok">34.4<span class="mkc">p</span></td>
 <td class="st-ok">&minus;6.8<span class="mkc">c</span></td>
 </tr>
 <tr>
 <th>MedGemma 4B</th>
 <td class="st-ok">18.3<span class="mkc">p</span></td>
 <td class="st-ok">70.8</td>
 <td class="st-ok">1.3</td>
 <td class="st-ok">9.0</td>
 <td class="st-ok">8.2<span class="mkc">p</span></td>
 <td class="st-gap">GAP</td>
 <td class="st-gap">GAP</td>
 <td class="st-gap">GAP</td>
 </tr>
 <tr class="baseline">
 <th>physician baseline</th>
 <td class="st-gap">&mdash;</td>
 <td class="st-gap">&mdash;</td>
 <td class="st-gap">&mdash;</td>
 <td>43.9</td>
 <td class="st-gap">&mdash;</td>
 <td class="st-gap">&mdash;</td>
 <td class="st-gap">&mdash;</td>
 <td class="st-gap">&mdash;</td>
 </tr>
 </tbody>
 </table>
 </div>
 <p class="mlegend">
 <span class="sw sw-ok"></span>in the comparable set for that column &nbsp;
 <span class="sw sw-off"></span>present but off-config &nbsp;
 <span class="sw sw-gap"></span>no run exists<br>
 <span class="mk">&#42;</span> ran at <code>epochs=1</code> where the rest of the column ran <code>epochs=8</code>
 &nbsp;&middot;&nbsp; <span class="mkc">c</span> score comes from a full cache replay (no model generation)
 &nbsp;&middot;&nbsp; <span class="mkc">p</span> mostly cached (&lt;200 candidate tokens per sample)
 </p>
<h3>Reading the cells</h3>
<p>The <code>c</code> and <code>p</code> markers matter less than they look. Caching a model's
 <em>response</em> and re-grading it is legitimate reuse: the response does not depend on the
 judge. What the markers tell you is that the number cannot be independently reproduced from
 the published log alone, because the generation happened in some earlier run. 27 of 68 logs
 are full replays; only 23 generated fresh candidate responses.</p>
<p>The <span class="warn">&#42;</span> marker is the one that breaks a table. DeepSeek's
 Professional runs used <code>epochs=1</code> (525 samples) while every other model used
 <code>epochs=8</code> (4200 samples). More epochs shrinks per-sample judge noise, so DeepSeek's
 31.0 carries visibly more error than GPT-5.5's 47.8 next to it.</p>
<h2 id="complete">What is already complete</h2>
<p>Three benches have all six models under one judge and one epoch setting. These you can
 publish today without caveats beyond the usual judge disclosure.</p>
<div class="ladder">
 <div class="chip hit">full 6/6</div>
 <div class="chip hit">consensus 6/6</div>
 <div class="chip hit">hard 6/6</div>
 <div class="chip">prof 5/6</div>
 <div class="chip">consult 5/6</div>
 <div class="chip">writing 5/6</div>
 <div class="chip">research 5/6</div>
 <div class="chip">red-team 5/6</div>
 </div>
 <div class="ladder-legend">olive = complete 6-model comparable set &middot; outline = incomplete</div>
<p>Model rankings are stable across every bench, which is the reassuring part: GPT-5.5 and
 Opus 4.7 lead and are within noise of each other, DeepSeek is third, then MedGemma-27B and
 PLaMo trade places depending on the bench, and MedGemma-4B is last everywhere. Whatever you
 fix in the gaps below is very unlikely to reorder the leaderboard.</p>
<div class="callout">
 <span class="ico">&#9733;</span>
 <div>
 <p>The harness itself is validated. The physician baseline on Professional comes out at
 <b>43.9</b> against OpenAI's published <b>43.7</b>. That is the anchor that says the
 scoring pipeline is correct, so the gaps below are coverage problems, not correctness
 problems.</p>
 </div>
 </div>
<h2 id="todo">What to run next</h2>
<h3>5 required runs</h3>
<p>These five close every genuine hole in the matrix. Ordered by how much they buy you.</p>
<details open>
 <summary>1 &middot; DeepSeek-V4-Pro, Professional whole-set at <code>epochs=8</code> <span class="where">fixes the &#42; on prof</span></summary>
 <div class="body">
 <p>The only run here that needs fresh generation, so it is the expensive one:
 roughly 1.3M candidate tokens &times; 8 epochs plus about 22M judge tokens, by analogy with
 the other <code>epochs=8</code> Professional runs. Everything else in the column already
 ran at 8 epochs, so this single run makes the Professional table uniform.</p>
 <pre>inspect eval healthbench_professional.py \
 --model openrouter/deepseek/deepseek-v4-pro \
 --epochs 8 \
 -T judge_model=openai/gpt-5.4 -T judge_reasoning_effort=low</pre>
 </div>
 </details>
<details>
 <summary>2 &middot; DeepSeek-V4-Pro, <code>consult</code> slice at <code>epochs=8</code> <span class="where">fixes the &#42; on consult</span></summary>
 <div class="body">
 <p>Once run 1 is in the cache, this is a grader-only pass over the same responses. It is the
 one slice DeepSeek never got at 8 epochs (writing, research and red-teaming already exist).</p>
 </div>
 </details>
<details>
 <summary>3&ndash;5 &middot; MedGemma-4B, the three missing Professional slices <span class="where">writing, research, red-teaming</span></summary>
 <div class="body">
 <p>MedGemma-4B has <code>consult</code> but not the other three. Its <code>epochs=8</code>
 whole-set Professional responses are already cached from the <code>hs-original</code> run on
 2026-07-25, so all three are grader-only passes. Cheapest wins in the list.</p>
 <pre>for uc in writing research red_teaming; do
 inspect eval healthbench_professional.py@healthbench_professional_$uc \
 --model vllm/google/medgemma-4b-it --epochs 8 \
 -T judge_model=openai/gpt-5.4 -T judge_reasoning_effort=low
done</pre>
 </div>
 </details>
<h3>The gpt-4.1 decision</h3>
<p>On 2026-08-05 the two MedGemma models were re-run on <code>hard</code> and
 <code>consensus</code> with the judge switched from <code>gpt-4o-mini</code> to
 <code>gpt-4.1</code>. Nothing else was re-run. That leaves a half-populated second column on
 both benches: 2 of 6 models have a gpt-4.1 number, 6 of 6 have a gpt-4o-mini number.</p>
<p>Two clean options, and you should pick one explicitly rather than let both sit in the folder:</p>
<table>
 <thead><tr><th>Option</th><th>Cost</th><th>Result</th></tr></thead>
 <tbody>
 <tr>
 <td><b>Drop the Aug-05 runs</b> from the reported set, keep gpt-4o-mini for hard and consensus.</td>
 <td>free</td>
 <td>Consistent today. But <code>full</code> stays on gpt-4.1 while hard and consensus stay on gpt-4o-mini, so the three non-professional benches still use two judges.</td>
 </tr>
 <tr>
 <td><b>Finish the gpt-4.1 column</b>: re-grade GPT-5.5, Opus 4.7, DeepSeek and PLaMo on hard and consensus.</td>
 <td>8 grader-only runs, roughly 140M judge tokens</td>
 <td>All three non-professional benches on gpt-4.1, matching <code>full</code>. This is the version worth publishing if the audience compares across benches.</td>
 </tr>
 </tbody>
 </table>
<p>The candidate responses for all 8 of those are already cached from July, so option 2 needs no
 model generation at all, only judge spend.</p>
<h2 id="axes">Slices with no clean data at all</h2>
<p>Beyond the four use-case slices, HealthBench Professional carries three more breakdown axes:
 <b>difficulty</b> (2 levels), <b>source slice</b> (3), and <b>specialty</b> (36). None of these
 has a usable number for any model, because they exist only as in-log metrics and those metrics
 are computed wrongly.</p>
<p>The in-log subset metrics (<code>use_case_*_score</code>, <code>type_*_score</code>,
 <code>difficulty_*_score</code>, <code>source_slice_*_score</code>, <code>specialty_*_score</code>)
 discard the length adjustment and clip each sample to [0, 1] before averaging. Comparing them
 against the standalone slice runs in <code>hs-prof-subsets</code> shows how far off that is:</p>
<table>
 <thead>
 <tr><th>Model &middot; slice</th><th class="num">in-log metric</th><th class="num">standalone run</th><th class="num">error</th></tr>
 </thead>
 <tbody>
 <tr><td>PLaMo &middot; red-teaming</td><td class="num">16.88</td><td class="num">&minus;11.82</td><td class="num bad">+28.7</td></tr>
 <tr><td>PLaMo &middot; writing</td><td class="num">19.77</td><td class="num">&minus;4.31</td><td class="num bad">+24.1</td></tr>
 <tr><td>MedGemma-27B &middot; red-teaming</td><td class="num">25.29</td><td class="num">&minus;6.80</td><td class="num bad">+32.1</td></tr>
 <tr><td>MedGemma-27B &middot; writing</td><td class="num">38.79</td><td class="num">9.13</td><td class="num bad">+29.7</td></tr>
 <tr><td>GPT-5.5 &middot; red-teaming</td><td class="num">45.80</td><td class="num">28.19</td><td class="num bad">+17.6</td></tr>
 <tr><td>GPT-5.5 &middot; consult</td><td class="num">60.58</td><td class="num">48.64</td><td class="num bad">+11.9</td></tr>
 <tr><td>Opus 4.7 &middot; research</td><td class="num">67.39</td><td class="num">61.07</td><td class="num warn">+6.3</td></tr>
 </tbody>
 </table>
<p>Every error is positive, and the worst cases are the slices where models score near zero,
 which is exactly where per-sample clipping does the most damage. The four use-case slices are
 safe only because Ajay built <code>hs-prof-subsets</code> to recompute them as standalone task
 runs. <b>Difficulty, source slice and specialty have no equivalent</b>, so if you need those
 breakdowns they must be re-aggregated from per-sample scores, or run as standalone filtered
 tasks the way the use-case slices were.</p>
<div class="callout">
 <span class="ico">&#9733;</span>
 <div>
 <p>Practical rule: never quote a <code>_scorer/&lt;axis&gt;_score</code> value out of a
 Professional log. Use <code>hs-prof-subsets</code> for the four use-case slices, and
 recompute anything else yourself.</p>
 </div>
 </div>
<h2 id="why">Why cells disagree</h2>
<p>Three sources of drift, in descending order of how much they move a number.</p>
<details>
 <summary>Judge model, worth up to 14 points <span class="where">3 judges across the set</span></summary>
 <div class="body">
 <p>Raw <code>healthbench_score</code> is just achieved points over possible points
 (<code>scorer.py:524</code>), so it is fully determined by the judge's rubric verdicts and is
 untouched by the length-adjustment arguments. That makes these clean single-variable
 comparisons: same model, same task code, judge swapped.</p>
 <pre> gpt-4o-mini gpt-4.1 delta
consensus, MedGemma-27B 0.7758 0.9105 +13.5
full, GPT-5.5 0.4873 0.5687 +8.1
full, Opus-4.7 0.4762 0.5344 +5.8
consensus, MedGemma-4B 0.7137 0.7583 +4.5
hard, MedGemma-27B 0.2109 0.1415 -6.9
hard, MedGemma-4B 0.1060 -0.0347 -14.1</pre>
 <p>gpt-4.1 is <em>more</em> generous than gpt-4o-mini on consensus and <em>harsher</em> on
 hard, so you cannot even predict the direction of the shift, let alone correct for it. Three
 judges are in play across the set: gpt-4o-mini (healthbench-v1, all of hs-hard, all of
 hs-consensus), gpt-4.1 (all of hs-non-professional, plus the Aug-05 re-runs), and gpt-5.4 at
 <code>reasoning_effort=low</code> for every Professional run.</p>
 </div>
 </details>
<details>
 <summary>Epochs, worth a couple of points of extra noise <span class="where">8 vs 1 on Professional</span></summary>
 <div class="body">
 <p>Professional ran at <code>epochs=8</code> for five models and <code>epochs=1</code> for
 DeepSeek. Every other bench ran at <code>epochs=1</code> for everyone, so no repeat-epoch
 variance estimate exists for full, consensus or hard at all.</p>
 </div>
 </details>
<details>
 <summary>Cache reuse, worth about 3.5 points on one measured pair <span class="where">cache=true everywhere</span></summary>
 <div class="body">
 <p>DeepSeek has two Professional runs with identical nominal config. The 2026-07-25 one in
 <code>hs-original</code> spent 10 candidate tokens per sample (a replay) and scored 34.30 raw;
 the 2026-08-06 one in <code>healthbench-professional-deepseek-v4-pro</code> generated 1.26M
 tokens fresh and scored 37.81 raw. Use the fresh one. The 3.5-point gap is about 1.5 standard
 errors, so it is plausibly just resampling, but it is the only direct measurement of
 cached-versus-fresh drift in the whole set.</p>
 </div>
 </details>
<h2 id="noise">How big does a gap have to be to matter</h2>
<p>Bootstrap standard errors from the logs, so you can tell a real difference from resampling
 noise before commissioning a re-run:</p>
<table>
 <thead><tr><th>Bench</th><th class="num">n</th><th class="num">std error</th><th>reads as</th></tr></thead>
 <tbody>
 <tr><td>full</td><td class="num">5000</td><td class="num">0.005</td><td>differences under 1 point are noise</td></tr>
 <tr><td>consensus</td><td class="num">3671</td><td class="num">0.005</td><td>same</td></tr>
 <tr><td>hard</td><td class="num">1000</td><td class="num">0.010</td><td>differences under 2 points are noise</td></tr>
 <tr><td>professional</td><td class="num">525</td><td class="num">0.023</td><td>differences under 4.5 points are noise</td></tr>
 </tbody>
 </table>
<p>This is why GPT-5.5 at 47.8 and Opus 4.7 at 48.0 on Professional should be reported as tied,
 and why the judge-swap effects above (up to 14 points) are unambiguously real rather than noise.</p>
<p>One footnote: the Aug-05 MedGemma-4B <code>hard</code> runs report a standard error of exactly
 <code>0.0</code>. That is a clipping artifact, not precision. The bootstrap clamps the mean to
 [0, 1], and since that run's mean is negative every resample clamps to 0, collapsing the spread.
 Treat the std error as unknown for any run with a negative mean.</p>
<h2 id="files">Where everything lives</h2>
<p>All ten HealthBench spaces are mirrored locally, 68 logs. <code>healthbench-v1</code> is a
 symlink into the inspect_ai sandbox where it already lived.</p>
<table>
 <thead><tr><th>Directory</th><th class="num">runs</th><th>Contents</th></tr></thead>
 <tbody>
 <tr><td><code>healthbench-v1</code></td><td class="num">7</td><td>first attempt, 2 usable, gpt-4o-mini judge, no length adjustment</td></tr>
 <tr><td><code>hs-non-professional</code></td><td class="num">6</td><td>the <code>full</code> column</td></tr>
 <tr><td><code>hs-consensus</code></td><td class="num">6</td><td>the <code>consensus</code> column</td></tr>
 <tr><td><code>hs-hard</code></td><td class="num">6</td><td>the <code>hard</code> column</td></tr>
 <tr><td><code>hs-original</code></td><td class="num">6</td><td>the <code>professional</code> column</td></tr>
 <tr><td><code>hs-prof-subsets</code></td><td class="num">26</td><td>the four use-case slices plus physician baseline</td></tr>
 <tr><td><code>medgemma-healthbench-hard</code></td><td class="num">2</td><td>Aug-05 gpt-4.1 re-grade, hard</td></tr>
 <tr><td><code>healthbench-hard-medgemma-v2</code></td><td class="num">2</td><td>same experiment, different provider prefix</td></tr>
 <tr><td><code>healthbench-consensus-medgemma</code></td><td class="num">2</td><td>Aug-05 gpt-4.1 re-grade, consensus</td></tr>
 <tr><td><code>healthbench-professional-deepseek-v4-pro</code></td><td class="num">1</td><td>the fresh DeepSeek Professional run</td></tr>
 </tbody>
 </table>
<p>Per-run detail with judge, epochs, token counts and a provenance classification is in
 <code>hb-spaces/MANIFEST.csv</code>, one row per log. <code>hb-spaces/INDEX.md</code> carries the
 trap list. On HuggingFace, <code>healthbench-profes</code> and
 <code>healthbench-prof-part1</code> exist but hold no logs, so there is nothing to mirror.</p>
<h2 id="faq">Questions this raises</h2>
<dl class="faq">
 <dt>If I only have budget for one thing, what is it?</dt>
 <dd>The DeepSeek Professional <code>epochs=8</code> run. It is the single cell that breaks the
 Professional table, and Professional is the bench with the physician baseline, so it is the one
 you would actually put in front of someone.</dd>
<dt>Can I merge the four use-case slices back into a whole-set Professional score?</dt>
 <dd>No. 236 + 142 + 147 + 191 = 716 against 525 total samples, so samples carry multiple
 use-case tags and the slices overlap. Use the whole-set run for the headline and the slices
 only as breakdowns.</dd>
<dt>Are the two MedGemma hard spaces duplicates?</dt>
 <dd>Effectively yes. <code>medgemma-healthbench-hard</code> and
 <code>healthbench-hard-medgemma-v2</code> are the same experiment run two hours apart, differing
 only in whether the model was addressed as <code>openai-api/vllm/google/&hellip;</code> or
 <code>vllm/google/&hellip;</code>. The 0.8-point spread between them is vLLM sampling noise, which
 incidentally gives you a free repeatability estimate for the local models.</dd>
<dt>Was gpt-5-nano ever evaluated?</dt>
 <dd>One attempt on 2026-07-09, which died on a 401. It is not part of the model set.</dd>
<dt>Why do so many runs show zero token usage?</dt>
 <dd><code>cache=true</code> was set on every run, and Inspect does not count cache hits toward
 <code>stats.model_usage</code>. An empty <code>model_usage</code> means every call was served
 from cache, which is the fastest way to spot a replay masquerading as a re-run.</dd>
 </dl>
</main>
</div>
</body>
</html>