Hub index, coverage matrix, config-check reports, and a Professional-runs log viewer

.gitattributes

@@ -33,3 +33,10 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+viewer/logs/professional__deepseek-v4-pro__2026-07-25-cached.eval filter=lfs diff=lfs merge=lfs -text
+viewer/logs/professional__deepseek-v4-pro__2026-08-06.eval filter=lfs diff=lfs merge=lfs -text
+viewer/logs/professional__gpt-5.5__2026-07-24.eval filter=lfs diff=lfs merge=lfs -text
+viewer/logs/professional__medgemma-27b__2026-07-24.eval filter=lfs diff=lfs merge=lfs -text
+viewer/logs/professional__medgemma-4b__2026-07-25.eval filter=lfs diff=lfs merge=lfs -text
+viewer/logs/professional__opus-4.7__2026-07-24.eval filter=lfs diff=lfs merge=lfs -text
+viewer/logs/professional__plamo-3.0-prime__2026-07-24.eval filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,10 +1,70 @@
 ---
-title: Healthbench Eval Logs
-emoji: 💻
-colorFrom: green
+title: HealthBench Eval Logs
+emoji: 🩺
+colorFrom: gray
 colorTo: red
 sdk: static
 pinned: false
+short_description: 68 Inspect eval logs, 6 models, 8 HealthBench variants
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# HealthBench eval logs
+
+Every HealthBench run we have, as Inspect `.eval` logs, in one browsable place.
+
+**6 models** GPT-5.5, Claude Opus 4.7, DeepSeek-V4-Pro, PLaMo-3.0-Prime, MedGemma-27B-text-it, MedGemma-4B-it
+**8 benches** HealthBench full / consensus / hard, and Professional plus its four use-case slices (consult, writing, research, red-teaming)
+**68 logs**, 2.3 GB, runs dated 2026-07-09 to 2026-08-06
+
+## Layout
+
+| Path | What it is |
+|---|---|
+| `index.html` | Start here. Every log in a filterable table with direct download links. |
+| `viewer/` | The Inspect log viewer, all 68 logs loaded. Click into samples and judge verdicts. |
+| `viewer/logs/*.eval` | The raw logs. Named `<bench>__<model>__<date>.eval`. |
+| `matrix.html` | Model x bench coverage matrix: what is complete, what is off-config, what is missing. |
+| `config-check-v2.html` | Do our numbers reproduce OpenAI's published results? |
+| `config-check-v1.html` | Earlier pass, superseded by v2, kept for history. |
+| `data/log_mapping.csv` | Maps each renamed log back to its original space and filename. |
+| `data/MANIFEST.csv` | Per-run header dump: judge, epochs, token counts, package versions. |
+| `data/headers_raw.json` | The same, unflattened. |
+| `data/INDEX.md` | Short version of the traps list. |
+
+## Reading a log directly
+
+```python
+from inspect_ai.log import read_eval_log
+
+log = read_eval_log("professional__gpt-5.5__2026-07-24.eval")
+print(log.eval.model, log.results.scores[0].metrics["mean"].value)
+```
+
+Or fetch one without cloning the whole Space:
+
+```
+https://huggingface.co/spaces/kirby44/healthbench-eval-logs/resolve/main/viewer/logs/<filename>.eval
+```
+
+## Four things that will bite you
+
+1. **The judge is not constant.** `gpt-4o-mini` on hard and consensus, `gpt-4.1` on full and the
+   Aug-05 MedGemma re-runs, `gpt-5.4` on all Professional. Swapping the judge moves a score by up to
+   14 points, and not always in the same direction. Only compare runs that share a judge.
+2. **Professional epochs are inconsistent.** 8 for most models, 1 for DeepSeek.
+3. **In-log subset metrics are wrong.** `use_case_*_score`, `specialty_*_score`,
+   `difficulty_*_score` and `source_slice_*_score` discard the length adjustment and clip each
+   sample to [0,1] before averaging. Errors reach +32 points, always upward. Use the standalone
+   `professional-*` logs for the use-case slices; re-aggregate the other axes from per-sample scores.
+4. **`cache=true` on every run.** A `-replay` or `-cached` suffix in the filename means responses
+   came from Inspect's cache. An empty `stats.model_usage` is a replay, not a run.
+
+The pipeline itself is validated: the Professional physician baseline lands at 43.9 against OpenAI's
+published 43.7.
+
+## Provenance
+
+Runs executed by Ajay (`ajay-citadel`) between 2026-07-09 and 2026-08-06, originally published as ten
+separate Spaces. This Space consolidates them into one viewer, renames the logs so the config is
+legible from the filename, and adds the provenance classification the raw logs do not carry. The logs
+are byte-identical to the originals; `data/log_mapping.csv` maps every file back to its source.

config-check-v1.html

@@ -0,0 +1,699 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<link rel="icon" href="favicon.svg">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>HealthBench config check: our four spaces vs OpenAI's published numbers</title>
+<style>
+  :root{
+    --ivory:#FAF9F5; --slate:#141413; --gray-700:#3D3D3A; --gray-500:#87867F;
+    --gray-300:#D1CFC5; --gray-150:#F0EEE6; --clay:#D97757; --oat:#E3DACC;
+    --olive:#788C5D;
+  }
+  *{box-sizing:border-box}
+  html{scroll-behavior:smooth}
+  body{
+    margin:0; background:var(--ivory); color:var(--gray-700);
+    font-family:system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;
+    font-size:15px; line-height:1.65; padding:56px 24px 120px;
+  }
+  .page{max-width:1100px; margin:0 auto; display:grid;
+    grid-template-columns:200px minmax(0,1fr); gap:48px}
+  h1,h2,h3,dt{font-family:ui-serif,Georgia,"Times New Roman",serif; font-weight:500; color:var(--slate)}
+  h1{font-size:32px; line-height:1.2; margin:0}
+  h2{font-size:22px; margin:40px 0 12px; scroll-margin-top:24px}
+  h3{font-size:17px; margin:26px 0 8px}
+  p{max-width:680px}
+  a{color:inherit}
+  code{font-family:ui-monospace,"SF Mono",Menlo,Consolas,monospace; font-size:13px}
+  .eyebrow{font-family:ui-monospace,"SF Mono",Menlo,Consolas,monospace;
+    text-transform:uppercase; letter-spacing:.08em; font-size:11px; color:var(--gray-500); margin:0 0 14px}
+  .dateline{font-family:ui-monospace,"SF Mono",Menlo,Consolas,monospace; font-size:11px; color:var(--gray-500); margin-top:10px}
+  header{margin-bottom:8px}
+  nav{position:sticky; top:32px; align-self:start; font-size:13px}
+  nav .navlinks{display:flex; flex-direction:column; border-left:2px solid var(--gray-300)}
+  nav a{display:block; padding:5px 0 5px 14px; margin-left:-2px;
+    border-left:2px solid transparent; color:var(--gray-700); text-decoration:none}
+  nav a:hover{border-left-color:var(--slate); color:var(--slate)}
+  nav a.l2{padding-left:26px; font-size:12px; color:var(--gray-500)}
+  nav .files{margin-top:22px}
+  nav .files .lbl{font-family:ui-monospace,Menlo,monospace; font-size:10px; text-transform:uppercase;
+    letter-spacing:.08em; color:var(--gray-500); margin-bottom:6px}
+  nav .files code{display:block; font-size:11px; color:var(--gray-500); word-break:break-all; margin-bottom:4px}
+  table{border-collapse:collapse; width:100%; max-width:860px; margin:14px 0; font-size:13.5px}
+  th,td{text-align:left; padding:8px 12px; border-bottom:1px solid var(--gray-300)}
+  th{font-family:ui-monospace,Menlo,monospace; font-size:11px; text-transform:uppercase;
+    letter-spacing:.05em; color:var(--gray-500); font-weight:600; border-bottom:1.5px solid var(--gray-300)}
+  td.num{font-family:ui-monospace,Menlo,monospace; text-align:right}
+  tr.ours{background:rgba(217,119,87,.09)}
+  tr.ours td{font-weight:600; color:var(--slate)}
+  tr.human td{color:var(--gray-500); font-style:italic}
+  .tag{font-family:ui-monospace,Menlo,monospace; font-size:10px; padding:1px 6px; border-radius:4px;
+    border:1px solid var(--gray-300); color:var(--gray-500); white-space:nowrap}
+  .tag.same{border-color:var(--olive); color:var(--olive)}
+  .tag.diff{border-color:var(--clay); color:var(--clay)}
+  .callout{border:1.5px solid var(--oat); background:rgba(227,218,204,.35); border-radius:10px;
+    padding:16px 18px; margin:20px 0; max-width:800px; display:flex; gap:14px}
+  .callout .star{color:var(--clay); font-size:18px; line-height:1.4}
+  .callout .ct{font-size:14px}
+  .callout .ct b{color:var(--slate)}
+  details{background:#fff; border:1px solid var(--gray-300); border-radius:10px; margin:10px 0; max-width:860px}
+  summary{cursor:pointer; padding:12px 16px; font-family:ui-serif,Georgia,serif; font-size:15px;
+    color:var(--slate); list-style:none; display:flex; align-items:center; gap:10px}
+  summary::-webkit-details-marker{display:none}
+  summary::before{content:"\25B8"; color:var(--clay); transition:transform .15s; font-size:13px}
+  details[open] summary::before{transform:rotate(90deg)}
+  summary .where{margin-left:auto; font-family:ui-monospace,Menlo,monospace; font-size:11px; color:var(--gray-500)}
+  details .body{padding:2px 16px 14px; font-size:14px}
+  details .body p{margin:8px 0}
+  details .body pre{background:var(--gray-150); border-radius:6px; padding:12px 14px; overflow-x:auto;
+    font-family:ui-monospace,Menlo,monospace; font-size:12px; line-height:1.5; margin:10px 0}
+  .hl{color:var(--clay)}
+  .ok{color:var(--olive); font-weight:600}
+  .warn{color:var(--clay); font-weight:600}
+  .bad{color:#B0533A; font-weight:600}
+  .muted{color:var(--gray-500)}
+  .note{font-size:12.5px; color:var(--gray-500); max-width:800px}
+  .ladder{display:grid; grid-template-columns:repeat(auto-fill,minmax(184px,1fr)); gap:7px; max-width:800px; margin:14px 0}
+  .chip{font-family:ui-monospace,Menlo,monospace; font-size:10.5px; border:1px solid var(--gray-300);
+    border-radius:6px; padding:6px 9px; color:var(--gray-500)}
+  .chip.hit{border-color:var(--olive); background:rgba(120,140,93,.10); color:var(--slate); font-weight:600}
+  .chip.miss{border-color:var(--clay); background:rgba(217,119,87,.08); color:var(--slate); font-weight:600}
+  .ladder-legend{font-family:ui-monospace,Menlo,monospace; font-size:10.5px; color:var(--gray-500); margin-top:6px}
+  .cmp-chart{width:100%; max-width:760px; height:auto; margin:10px 0 4px; font-family:system-ui}
+  .tc-ax{font-size:11px; fill:var(--gray-500); font-family:ui-monospace,Menlo,monospace}
+  .tc-ml{font-size:11px; fill:var(--slate); font-family:ui-monospace,Menlo,monospace}
+  .tc-lg{font-size:11px; fill:var(--gray-700)}
+  .tc-val{font-size:10px; fill:var(--slate); font-family:ui-monospace,Menlo,monospace}
+  .tc-dl{font-size:10.5px; font-family:ui-monospace,Menlo,monospace; font-weight:600}
+  @media (max-width:920px){
+    .page{grid-template-columns:1fr}
+    nav{display:none}
+  }
+</style>
+
+<style id="hbnav-css">
+  .hbnav{position:sticky;top:0;z-index:50;display:flex;flex-wrap:wrap;align-items:center;gap:4px;
+    padding:8px 16px;margin:-56px -24px 40px;background:rgba(250,249,245,.94);
+    backdrop-filter:saturate(180%) blur(8px);border-bottom:1px solid #D1CFC5;
+    font-family:ui-monospace,"SF Mono",Menlo,Consolas,monospace;font-size:11.5px}
+  .hbnav .home{color:#87867F;margin-right:6px;text-decoration:none}
+  .hbnav .home:hover{color:#141413}
+  .hbnav a.pill{color:#3D3D3A;text-decoration:none;padding:3px 9px;border:1px solid #D1CFC5;border-radius:6px}
+  .hbnav a.pill:hover{border-color:#D97757;color:#141413}
+  .hbnav a.pill.here{background:#141413;border-color:#141413;color:#FAF9F5}
+  .hbnav .sp{flex:1}
+  @media(max-width:640px){.hbnav{margin:-56px -24px 28px}}
+</style>
+</head>
+<body>
+<div class="hbnav"><a class="home" href="index.html">&#8592; HealthBench logs</a><a class="pill" href="matrix.html">Coverage matrix</a><a class="pill" href="config-check-v2.html">Config check v2</a><a class="pill here" href="config-check-v1.html">Config check v1</a><span class="sp"></span><a class="pill" href="viewer/index.html">Log viewer &rarr;</a></div>
+<div class="page">
+<nav>
+  <div class="navlinks">
+    <a href="#summary">Summary</a>
+    <a href="#anchor">The anchor check</a>
+    <a href="#baseline" class="l2">Physician baseline</a>
+    <a href="#config">Config matrix</a>
+    <a href="#judge">Issue 1: judge model</a>
+    <a href="#effort">Issue 2: reasoning effort</a>
+    <a href="#headline">Issue 3: headline metric</a>
+    <a href="#subscores">Issue 4: subscore clipping</a>
+    <a href="#floored">Issue 5: floored zeros</a>
+    <a href="#hygiene">Run hygiene</a>
+    <a href="#fixes">What to change</a>
+    <a href="#full">Full results</a>
+    <a href="#method">How this was checked</a>
+  </div>
+  <div class="files">
+    <div class="lbl">Run artifacts</div>
+    <code>hb-spaces/hs-non-professional/logs/</code>
+    <code>hb-spaces/hs-hard/logs/</code>
+    <code>hb-spaces/hs-consensus/logs/</code>
+    <code>hb-spaces/hs-prof-subsets/logs/</code>
+    <div class="lbl" style="margin-top:14px">Reference</div>
+    <code>arXiv:2505.08775 (HealthBench)</code>
+    <code>arXiv:2604.27470 (Professional)</code>
+    <code>GPT-5.6 system card, Table 6</code>
+    <code>openai/simple-evals</code>
+  </div>
+</nav>
+
+<main>
+<header>
+  <p class="eyebrow">HealthBench &middot; config sanity check</p>
+  <h1>HealthBench config check: our four spaces vs OpenAI's published numbers</h1>
+  <div class="dateline">Created 2026-07-27</div>
+</header>
+
+<section id="summary">
+<p>
+We have a real anchor for this check: OpenAI's GPT-5.6 system card publishes <b>gpt-5.5 scores for all four
+HealthBench variants</b>, and gpt-5.5 is one of the models we ran. Lining our numbers up against theirs
+gives a per-variant verdict rather than a guess.
+</p>
+<p>
+The result splits cleanly. <b>The two spaces where we used the correct grader reproduce OpenAI within a
+point.</b> The full HealthBench run lands at <b>55.8</b> against their <b>56.5</b>, and our HealthBench
+Professional physician baseline lands at <b>43.87</b> against their published <b>43.7</b>. That is close
+enough to say the dataset, prompting, rubric scoring, length adjustment and aggregation are all wired up
+right.
+</p>
+<p>
+<b>The two spaces where we used the wrong grader are badly off.</b> HealthBench Hard is <b>5.5 points low</b>
+and HealthBench Consensus is <b>13.6 points low</b>. Both were graded by <code>gpt-4o-mini</code> instead of
+GPT-4.1, not by choice but because the <code>inspect_evals</code> task wrappers for those two variants do not
+expose a <code>judge_model</code> argument and silently fall back to the package default. Consensus is the
+loudest signal: every frontier model in OpenAI's table sits in a 94 to 96 band, and ours sits at 82.
+</p>
+<p>
+A separate, smaller gap shows up on the inference side. Our gpt-5.5 Professional score is <b>4 points below</b>
+OpenAI's, even though the grader is correct there. The physician baseline matching to 0.2 points rules out
+the grading pipeline, which points the finger at <b>reasoning effort</b>: we pass none, while OpenAI evaluates
+"at the highest reasoning effort option available via each model's API." Claude was run with extended thinking
+off entirely.
+</p>
+
+<div class="callout">
+  <span class="star">&#9733;</span>
+  <span class="ct"><b>The one-line verdict.</b> The scoring machinery is correct and provably so. The
+  configuration around it is not: two variants use a mini-tier grader, no run sets reasoning effort, and three
+  of the four spaces show the unadjusted score as the headline where OpenAI shows the length-adjusted one.
+  None of this requires re-implementing anything, only re-running with four flags set.</span>
+</div>
+</section>
+
+<section id="anchor">
+<h2>The anchor check</h2>
+<p>
+All numbers are the <b>length-adjusted score on the 0 to 100 scale</b>, which is what OpenAI reports as the
+primary metric. Their figures come from the GPT-5.6 system card Table 6, given as
+<code>length-adjusted (unadjusted, mean chars)</code>. Ours are recomputed from the downloaded
+<code>.eval</code> logs.
+</p>
+
+<svg viewBox="0 0 760 430" xmlns="http://www.w3.org/2000/svg" class="cmp-chart" role="img"
+     aria-label="gpt-5.5 length-adjusted scores, OpenAI published versus our runs, across five checks">
+  <line x1="60" y1="40" x2="60" y2="340" stroke="#141413" stroke-width="1"/>
+  <line x1="60" y1="340" x2="740" y2="340" stroke="#141413" stroke-width="1"/>
+  <line x1="60" y1="280" x2="740" y2="280" stroke="#F0EEE6" stroke-width="1"/>
+  <line x1="60" y1="220" x2="740" y2="220" stroke="#F0EEE6" stroke-width="1"/>
+  <line x1="60" y1="160" x2="740" y2="160" stroke="#F0EEE6" stroke-width="1"/>
+  <line x1="60" y1="100" x2="740" y2="100" stroke="#F0EEE6" stroke-width="1"/>
+  <line x1="60" y1="40"  x2="740" y2="40"  stroke="#F0EEE6" stroke-width="1"/>
+  <text x="52" y="343.5" text-anchor="end" class="tc-ax">0</text>
+  <text x="52" y="283.5" text-anchor="end" class="tc-ax">20</text>
+  <text x="52" y="223.5" text-anchor="end" class="tc-ax">40</text>
+  <text x="52" y="163.5" text-anchor="end" class="tc-ax">60</text>
+  <text x="52" y="103.5" text-anchor="end" class="tc-ax">80</text>
+  <text x="52" y="43.5"  text-anchor="end" class="tc-ax">100</text>
+
+  <!-- group 0: physician baseline -->
+  <rect x="88"  y="208.9" width="40" height="131.1" fill="#87867F"/>
+  <rect x="132" y="208.4" width="40" height="131.6" fill="#D97757"/>
+  <text x="108" y="203" text-anchor="middle" class="tc-val">43.7</text>
+  <text x="152" y="203" text-anchor="middle" class="tc-val">43.9</text>
+  <text x="130" y="358" text-anchor="middle" class="tc-ml">Physician</text>
+  <text x="130" y="372" text-anchor="middle" class="tc-ml">baseline</text>
+  <text x="130" y="390" text-anchor="middle" class="tc-dl" fill="#788C5D">+0.2</text>
+
+  <!-- group 1: professional -->
+  <rect x="224" y="184.6" width="40" height="155.4" fill="#87867F"/>
+  <rect x="268" y="196.6" width="40" height="143.4" fill="#D97757"/>
+  <text x="244" y="178.6" text-anchor="middle" class="tc-val">51.8</text>
+  <text x="288" y="190.6" text-anchor="middle" class="tc-val">47.8</text>
+  <text x="266" y="358" text-anchor="middle" class="tc-ml">Professional</text>
+  <text x="266" y="372" text-anchor="middle" class="tc-ax">grader ok</text>
+  <text x="266" y="390" text-anchor="middle" class="tc-dl" fill="#D97757">&minus;4.0</text>
+
+  <!-- group 2: healthbench full -->
+  <rect x="360" y="170.5" width="40" height="169.5" fill="#87867F"/>
+  <rect x="404" y="172.5" width="40" height="167.5" fill="#D97757"/>
+  <text x="380" y="164.5" text-anchor="middle" class="tc-val">56.5</text>
+  <text x="424" y="166.5" text-anchor="middle" class="tc-val">55.8</text>
+  <text x="402" y="358" text-anchor="middle" class="tc-ml">HealthBench</text>
+  <text x="402" y="372" text-anchor="middle" class="tc-ax">grader ok</text>
+  <text x="402" y="390" text-anchor="middle" class="tc-dl" fill="#788C5D">&minus;0.7</text>
+
+  <!-- group 3: hard -->
+  <rect x="496" y="245.5" width="40" height="94.5" fill="#87867F"/>
+  <rect x="540" y="261.9" width="40" height="78.1" fill="#D97757"/>
+  <text x="516" y="239.5" text-anchor="middle" class="tc-val">31.5</text>
+  <text x="560" y="255.9" text-anchor="middle" class="tc-val">26.0</text>
+  <text x="538" y="358" text-anchor="middle" class="tc-ml">Hard</text>
+  <text x="538" y="372" text-anchor="middle" class="tc-ax">wrong grader</text>
+  <text x="538" y="390" text-anchor="middle" class="tc-dl" fill="#B0533A">&minus;5.5</text>
+
+  <!-- group 4: consensus -->
+  <rect x="632" y="53.2" width="40" height="286.8" fill="#87867F"/>
+  <rect x="676" y="93.9" width="40" height="246.1" fill="#D97757"/>
+  <text x="652" y="47.2" text-anchor="middle" class="tc-val">95.6</text>
+  <text x="696" y="87.9" text-anchor="middle" class="tc-val">82.0</text>
+  <text x="674" y="358" text-anchor="middle" class="tc-ml">Consensus</text>
+  <text x="674" y="372" text-anchor="middle" class="tc-ax">wrong grader</text>
+  <text x="674" y="390" text-anchor="middle" class="tc-dl" fill="#B0533A">&minus;13.6</text>
+
+  <rect x="60" y="408" width="12" height="12" fill="#87867F"/>
+  <text x="78" y="418" class="tc-lg">OpenAI published</text>
+  <rect x="200" y="408" width="12" height="12" fill="#D97757"/>
+  <text x="218" y="418" class="tc-lg">our run</text>
+  <text x="320" y="418" class="tc-ax">gpt-5.5, length-adjusted score (0 to 100)</text>
+</svg>
+
+<table>
+  <thead><tr><th>Check</th><th>Grader used</th><th class="num">OpenAI adj</th><th class="num">ours adj</th><th class="num">&Delta;</th><th class="num">OpenAI raw</th><th class="num">ours raw</th><th class="num">OpenAI chars</th><th class="num">ours chars</th></tr></thead>
+  <tbody>
+    <tr><td>Physician baseline (Prof)</td><td><span class="tag same">gpt-5.4 low</span></td><td class="num">43.7</td><td class="num">43.87</td><td class="num" style="color:#788C5D">+0.2</td><td class="num muted">n/a</td><td class="num">44.29</td><td class="num muted">n/a</td><td class="num muted">n/a</td></tr>
+    <tr><td>HealthBench (full)</td><td><span class="tag same">gpt-4.1</span></td><td class="num">56.5</td><td class="num">55.83</td><td class="num" style="color:#788C5D">&minus;0.7</td><td class="num">58.4</td><td class="num">56.87</td><td class="num">2313</td><td class="num">2175</td></tr>
+    <tr><td>HealthBench Professional</td><td><span class="tag same">gpt-5.4 low</span></td><td class="num">51.8</td><td class="num">47.81</td><td class="num" style="color:#D97757">&minus;4.0</td><td class="num">57.2</td><td class="num muted">n/a</td><td class="num">3818</td><td class="num">3748</td></tr>
+    <tr><td>HealthBench Hard</td><td><span class="tag diff">gpt-4o-mini</span></td><td class="num">31.5</td><td class="num">26.03</td><td class="num" style="color:#B0533A">&minus;5.5</td><td class="num">33.8</td><td class="num">27.27</td><td class="num">2289</td><td class="num">2158</td></tr>
+    <tr><td>HealthBench Consensus</td><td><span class="tag diff">gpt-4o-mini</span></td><td class="num">95.6</td><td class="num">82.02</td><td class="num" style="color:#B0533A">&minus;13.6</td><td class="num">95.7</td><td class="num">82.08</td><td class="num">2259</td><td class="num">2143</td></tr>
+  </tbody>
+</table>
+<p class="note">
+Our Professional overall is a sample-weighted mean of the three use-case subsets (consult 236, writing 142,
+research 147, total 525), since we ran them as separate tasks rather than one benchmark. Red teaming is a
+cross-cutting slice of those same 525 examples, not a fourth use case, so it is excluded from the weighting.
+</p>
+<p>
+Two things worth noticing beyond the scores. First, <b>mean response lengths track closely</b> (2175 vs 2313,
+2158 vs 2289, 2143 vs 2259, 3748 vs 3818), consistently about 5% shorter but never structurally different.
+Since length is the one output property that would move if our prompting or harness diverged, this is good
+evidence the request side is faithful. Second, the two <span class="tag same">correct grader</span> rows and
+the two <span class="tag diff">wrong grader</span> rows separate perfectly, which is what makes the diagnosis
+confident rather than speculative.
+</p>
+
+<h3 id="baseline">The physician baseline is the cleanest check we have</h3>
+<p>
+HealthBench Professional ships a set of <b>525 physician-written responses</b>. Scoring them involves no model
+inference at all: the responses are fixed text, and the only moving part is the grader plus the aggregation.
+So if our number matches OpenAI's, the entire scoring half of the pipeline is verified, and any remaining
+discrepancy on real models has to live on the inference side.
+</p>
+<table>
+  <thead><tr><th>Physician baseline</th><th class="num">OpenAI published</th><th class="num">ours (recomputed)</th><th class="num">&Delta;</th></tr></thead>
+  <tbody>
+    <tr class="ours"><td>Overall</td><td class="num">43.7</td><td class="num">43.87</td><td class="num">+0.17</td></tr>
+    <tr><td>Care consult</td><td class="num">42.7</td><td class="num">42.54</td><td class="num">&minus;0.16</td></tr>
+    <tr><td>Writing and documentation</td><td class="num">32.1</td><td class="num">32.91</td><td class="num">+0.81</td></tr>
+    <tr><td>Medical research</td><td class="num">56.3</td><td class="num">56.60</td><td class="num">+0.30</td></tr>
+  </tbody>
+</table>
+<p class="note">
+Every cell is within a point. Note that the per-use-case rows here are <b>recomputed from per-sample scores</b>,
+not read off the log's own subset metrics, which are inflated for a reason covered in
+<a href="#subscores">issue 4</a>.
+</p>
+<div class="callout">
+  <span class="star">&#9733;</span>
+  <span class="ct"><b>This is the finding that makes the rest interpretable.</b> Because the judge-only path
+  reproduces OpenAI to 0.2 points, the 4-point Professional gap on gpt-5.5 cannot be blamed on the grader,
+  the rubric, the length adjustment, or the aggregation. It is an inference-side difference, and reasoning
+  effort is the only inference-side knob that differs.</span>
+</div>
+</section>
+
+<section id="config">
+<h2>Config matrix</h2>
+<p>What we set against what OpenAI's reference implementation sets, per variant.</p>
+<table>
+  <thead><tr><th>Setting</th><th>OpenAI reference</th><th>hs-non-professional</th><th>hs-hard</th><th>hs-consensus</th><th>hs-prof-subsets</th></tr></thead>
+  <tbody>
+    <tr><td>Grader model</td><td>gpt-4.1 / gpt-5.4 low</td><td><span class="tag same">gpt-4.1</span></td><td><span class="tag diff">gpt-4o-mini</span></td><td><span class="tag diff">gpt-4o-mini</span></td><td><span class="tag same">gpt-5.4 low</span></td></tr>
+    <tr><td>Grader system msg</td><td>"You are a helpful assistant."</td><td><span class="tag same">same</span></td><td><span class="tag same">same</span></td><td><span class="tag same">same</span></td><td><span class="tag same">same</span></td></tr>
+    <tr><td>Length adj. center</td><td>2000 chars</td><td><span class="tag same">2000</span></td><td><span class="tag same">2000</span></td><td><span class="tag same">2000</span></td><td><span class="tag same">2000</span></td></tr>
+    <tr><td>Length adj. penalty</td><td>2.99 / 3.92 / 0.20 / 1.47</td><td><span class="tag same">0.0299</span></td><td><span class="tag same">0.0392</span></td><td><span class="tag same">0.002</span></td><td><span class="tag same">0.0147</span></td></tr>
+    <tr><td>Per-sample clipping</td><td>none (mean is clipped)</td><td><span class="tag same">none</span></td><td><span class="tag same">none</span></td><td><span class="tag same">none</span></td><td><span class="tag diff">subscores clipped</span></td></tr>
+    <tr><td>Dataset size</td><td>5000 / 1000 / 3671 / 525</td><td><span class="tag same">5000</span></td><td><span class="tag same">1000</span></td><td><span class="tag same">3671</span></td><td><span class="tag same">525</span></td></tr>
+    <tr><td>Samples per example</td><td>1 (main), 8 (Prof)</td><td><span class="tag same">1</span></td><td><span class="tag same">1</span></td><td><span class="tag same">1</span></td><td><span class="tag same">8</span> <span class="tag diff">1 for deepseek</span></td></tr>
+    <tr><td>Reasoning effort</td><td>highest available</td><td><span class="tag diff">unset</span></td><td><span class="tag diff">unset</span></td><td><span class="tag diff">unset</span></td><td><span class="tag diff">unset</span></td></tr>
+    <tr><td>Headline metric</td><td>length-adjusted</td><td><span class="tag diff">unadjusted</span></td><td><span class="tag diff">unadjusted</span></td><td><span class="tag diff">unadjusted</span></td><td><span class="tag same">length-adjusted</span></td></tr>
+    <tr><td>Bootstrap resamples</td><td>1000</td><td><span class="tag same">1000</span></td><td><span class="tag same">1000</span></td><td><span class="tag same">1000</span></td><td><span class="tag same">1000</span></td></tr>
+  </tbody>
+</table>
+<p class="note">
+The length adjustment is worth calling out as correct-by-verification, not just correct-by-declaration: we
+confirmed the arithmetic per sample against
+<code>score &minus; penalty &times; ((len &minus; center) / 500)</code>, OpenAI's
+<code>calculate_length_adjusted_score</code>. On one Hard sample: raw 0.6102, completion 13,776 chars, giving
+0.6102 &minus; 0.0392 &times; 23.552 = &minus;0.3131, which is exactly the stored value.
+</p>
+</section>
+
+<section id="judge">
+<h2>Issue 1: Hard and Consensus were graded by gpt-4o-mini</h2>
+<p>
+This is the largest error and it was not a deliberate choice. The <code>inspect_evals</code> task wrappers for
+these two variants accept only the length-adjustment arguments and pass nothing else through, so
+<code>judge_model</code> silently keeps the package default of <code>openai/gpt-4o-mini</code>:
+</p>
+<details>
+<summary>The wrapper that drops the judge argument <span class="where">inspect_evals/healthbench/healthbench.py</span></summary>
+<div class="body">
+<pre>def healthbench(
+    ...
+    judge_model: str | Model = "openai/gpt-4o-mini",   <span class="muted"># package default</span>
+    ...
+)
+
+@task
+def healthbench_hard(
+    length_adjustment_center: float | None = None,
+    length_adjustment_penalty_per_500_chars: float | None = None,
+) -&gt; Task:
+    return healthbench(                      <span class="muted"># judge_model never forwarded</span>
+        subset="hard",
+        length_adjustment_center=length_adjustment_center,
+        length_adjustment_penalty_per_500_chars=length_adjustment_penalty_per_500_chars,
+    )</pre>
+<p>Our <code>hs-non-professional</code> run passed <code>judge_model: "openai/gpt-4.1"</code> explicitly and got
+it. The Hard and Consensus runs declared no judge at all in <code>task_args</code>, and inspecting the model
+events in the logs confirms every grading call went to <code>gpt-4o-mini-2024-07-18</code>.</p>
+</div>
+</details>
+<p>
+OpenAI never grades any variant with a mini-tier model. In <code>simple_evals.py</code> the same
+<code>healthbench_grading_sampler</code> is handed to <code>healthbench</code>, <code>healthbench_hard</code>
+and <code>healthbench_consensus</code> alike, so all three get GPT-4.1, or all three get GPT-5.4 low when the
+<code>--healthbench-use-gpt-5-4-low-grader</code> flag is set.
+</p>
+<p>
+The HealthBench paper measured what a weaker grader costs. Its meta-evaluation ranks candidate graders by
+agreement with physicians (Macro-F1): <b>GPT-4.1 0.709</b>, o4-mini 0.692, o3 0.681, GPT-4.1 mini 0.661,
+GPT-4.1 nano 0.580, and calls the smaller variants "substantially worse". <code>gpt-4o-mini</code> is not in
+that table but sits below GPT-4.1 mini in capability, so 0.661 is the optimistic bound.
+</p>
+<div class="callout">
+  <span class="star">&#9733;</span>
+  <span class="ct"><b>Consensus is the smoking gun.</b> OpenAI's Consensus scores are 94 to 96 for every model
+  from gpt-5 through gpt-5.6, because the subset is deliberately low-noise and near-saturated. Our whole
+  Consensus space sits between <b>71 and 82</b>. That is not our models underperforming, that is the grader
+  failing to recognise criteria that a stronger grader marks as met.</span>
+</div>
+</section>
+
+<section id="effort">
+<h2>Issue 2: reasoning effort is never set, and the models diverge because of it</h2>
+<p>
+The HealthBench Professional paper states that models are "evaluated at the highest reasoning effort option
+available via each model's API (e.g., xhigh for GPT-5.4)", and measures a <b>5.6 to 7.3 point</b> gain moving
+from low to xhigh. None of our runs pass a reasoning effort, so each provider's default applies, and the
+defaults are not comparable to each other:
+</p>
+<table>
+  <thead><tr><th>Model (HealthBench Hard, 1000 samples)</th><th class="num">frac. with reasoning</th><th class="num">mean reasoning tokens</th><th>generate config sent</th></tr></thead>
+  <tbody>
+    <tr><td>openai/gpt-5.5</td><td class="num">0.97</td><td class="num">291</td><td><code>{cache: true}</code></td></tr>
+    <tr><td>openrouter/deepseek-v4-pro</td><td class="num">0.86</td><td class="num">489</td><td><code>{cache: true}</code></td></tr>
+    <tr class="ours"><td>anthropic/claude-opus-4-7</td><td class="num">0.00</td><td class="num">0</td><td><code>{cache: true, max_tokens: 32000}</code></td></tr>
+    <tr><td>plamo-3.0-prime</td><td class="num">0.00</td><td class="num">0</td><td><code>{cache: true}</code></td></tr>
+    <tr><td>medgemma-4b / 27b</td><td class="num">0.00</td><td class="num">0</td><td><code>{cache: true}</code></td></tr>
+  </tbody>
+</table>
+<p>
+Claude ran with <b>extended thinking off on every sample of every log</b>, while gpt-5.5 and DeepSeek reasoned
+by default. Whatever the intended comparison was, this is not it: the reasoning models get their default
+budget and Claude gets none. Claude is the most disadvantaged model in the set for a reason that has nothing
+to do with Claude.
+</p>
+<p>
+This also explains the one gap the grader cannot account for. On Professional, where our grader is correct and
+the physician baseline matches to 0.2 points, gpt-5.5 still lands 4 points low. A default-effort versus
+highest-effort difference of that size is consistent with the 5.6 to 7.3 point low-to-xhigh delta OpenAI
+reports.
+</p>
+</section>
+
+<section id="headline">
+<h2>Issue 3: three spaces show the unadjusted score as the headline</h2>
+<p>
+OpenAI reports "length-adjusted score (unadjusted, mean response length in characters)", so the adjusted
+number is the primary one. In our <code>listing.json</code> the <code>primary_metric</code> for
+<code>hs-non-professional</code>, <code>hs-hard</code> and <code>hs-consensus</code> is the
+<b>unadjusted</b> <code>bootstrap_score</code>. Only <code>hs-prof-subsets</code> carries
+<code>use_length_adjusted: true</code>.
+</p>
+<p>
+This is not cosmetic. Length adjustment reorders the Hard leaderboard almost completely, because our models
+differ enormously in verbosity:
+</p>
+<table>
+  <thead><tr><th>Model</th><th class="num">mean chars</th><th class="num">raw</th><th class="num">length-adj</th><th class="num">shift</th><th class="num">rank raw</th><th class="num">rank adj</th></tr></thead>
+  <tbody>
+    <tr><td>anthropic/claude-opus-4-7</td><td class="num">1849</td><td class="num">26.61</td><td class="num">27.80</td><td class="num" style="color:#788C5D">+1.2</td><td class="num">2</td><td class="num">1</td></tr>
+    <tr><td>openai/gpt-5.5</td><td class="num">2158</td><td class="num">27.27</td><td class="num">26.03</td><td class="num">&minus;1.2</td><td class="num">1</td><td class="num">2</td></tr>
+    <tr><td>openrouter/deepseek-v4-pro</td><td class="num">3408</td><td class="num">24.84</td><td class="num">13.80</td><td class="num" style="color:#B0533A">&minus;11.0</td><td class="num">3</td><td class="num">3</td></tr>
+    <tr><td>plamo-3.0-prime</td><td class="num">2997</td><td class="num">17.43</td><td class="num">9.61</td><td class="num" style="color:#B0533A">&minus;7.8</td><td class="num">5</td><td class="num">4</td></tr>
+    <tr><td>vllm/medgemma-27b</td><td class="num">4073</td><td class="num">21.09</td><td class="num">4.83</td><td class="num" style="color:#B0533A">&minus;16.3</td><td class="num">4</td><td class="num">5</td></tr>
+    <tr><td>vllm/medgemma-4b</td><td class="num">3192</td><td class="num">10.60</td><td class="num">1.26</td><td class="num" style="color:#B0533A">&minus;9.3</td><td class="num">6</td><td class="num">6</td></tr>
+  </tbody>
+</table>
+<p class="note">
+Claude gains because it is the only model averaging under the 2000-character centre. medgemma-27b loses 16
+points. Anyone reading the space's default metric is reading a different ranking from the one OpenAI's
+methodology produces.
+</p>
+</section>
+
+<section id="subscores">
+<h2>Issue 4: the Professional per-use-case subscores are inflated</h2>
+<p>
+The custom <code>healthbench_professional.py</code> emits <code>use_case_*_score</code>,
+<code>type_*_score</code> and <code>difficulty_*_score</code> metrics alongside the headline. <b>These do not
+use the same aggregation as the headline and should not be quoted.</b> They are consistent with per-sample
+clipping to [0, 1] before averaging, where OpenAI clips only the final mean.
+</p>
+<p>
+The physician baseline makes the error measurable, because OpenAI publishes the ground truth for exactly these
+three cells:
+</p>
+<table>
+  <thead><tr><th>Physician baseline by use case</th><th class="num">OpenAI</th><th class="num">recomputed correctly</th><th class="num">as reported in our log</th><th class="num">inflation</th></tr></thead>
+  <tbody>
+    <tr><td>Care consult</td><td class="num">42.7</td><td class="num">42.54</td><td class="num" style="color:#B0533A">48.4</td><td class="num">+5.9</td></tr>
+    <tr><td>Writing and documentation</td><td class="num">32.1</td><td class="num">32.91</td><td class="num" style="color:#B0533A">44.9</td><td class="num">+12.0</td></tr>
+    <tr><td>Medical research</td><td class="num">56.3</td><td class="num">56.60</td><td class="num" style="color:#B0533A">59.0</td><td class="num">+2.4</td></tr>
+  </tbody>
+</table>
+<details>
+<summary>The arithmetic that identifies the cause <span class="where">525 examples &times; 8 epochs = 4200</span></summary>
+<div class="body">
+<pre>overall mean raw                    = 44.29   <span class="muted"># matches reported 44.29</span>
+overall mean adj                    = 43.87   <span class="muted"># matches reported 43.87, and OpenAI's 43.7</span>
+overall mean adj, clipped per-sample= 49.71
+frac. of samples with adj &lt; 0       = 0.164
+
+by use case          n      mean_adj   clipped_per_sample
+  consult          1888      42.54          48.09
+  research         1176      56.60          57.47
+  writing          1136      32.91          44.39</pre>
+<p>16.4% of physician responses score below zero once the length penalty applies, mostly short writing-task
+answers that trip negative rubric criteria. Clipping those to zero before averaging is what lifts writing from
+32.9 to roughly 44. The residual of about 0.3 between the clipped figures and the reported ones is bootstrap
+resampling noise; the mechanism is the clipping.</p>
+</div>
+</details>
+<p>
+The headline <code>bootstrap_score</code> is unaffected and remains correct. Only the subset breakdowns are
+wrong, which matters because the by-use-case split is the most quoted view of Professional results.
+</p>
+</section>
+
+<section id="floored">
+<h2>Issue 5: five Professional scores are floored at zero</h2>
+<p>
+OpenAI clips the aggregate mean to [0, 1], and our implementation follows suit, so this is faithful behaviour
+rather than a bug. But it means five reported numbers are all displayed as <code>0.00</code> while their true
+values differ by 7 points, which hides real ranking information:
+</p>
+<table>
+  <thead><tr><th>Model</th><th>Subset</th><th class="num">reported</th><th class="num">true unclipped mean</th></tr></thead>
+  <tbody>
+    <tr><td>plamo-3.0-prime</td><td>red teaming</td><td class="num">0.00</td><td class="num">&minus;11.82</td></tr>
+    <tr><td>openrouter/deepseek-v4-pro</td><td>red teaming (ep 1)</td><td class="num">0.00</td><td class="num">&minus;8.29</td></tr>
+    <tr><td>openrouter/deepseek-v4-pro</td><td>red teaming (ep 8)</td><td class="num">0.00</td><td class="num">&minus;6.87</td></tr>
+    <tr><td>vllm/medgemma-27b</td><td>red teaming</td><td class="num">0.00</td><td class="num">&minus;6.80</td></tr>
+    <tr><td>plamo-3.0-prime</td><td>writing</td><td class="num">0.00</td><td class="num">&minus;4.31</td></tr>
+  </tbody>
+</table>
+<p class="note">
+Worth reporting the unclipped value alongside the clipped one for these, or at least noting that a zero means
+"at or below zero" rather than "scored nothing".
+</p>
+</section>
+
+<section id="hygiene">
+<h2>Run hygiene</h2>
+<details>
+<summary>DeepSeek's Professional runs are under-sampled, and its consult run is the only one <span class="where">hs-prof-subsets</span></summary>
+<div class="body">
+<p>Every model ran <code>epochs=8</code>, matching the paper's "8 samples per example". The 2026-07-25 DeepSeek
+batch ran <code>epochs=1</code>. For <code>writing</code>, <code>research</code>, <code>red_teaming</code> and
+<code>physician_baseline</code> both variants exist in <code>listing.json</code>, so the viewer shows duplicate
+conflicting entries. For <code>consult</code> the epochs=1 run is the <b>only</b> DeepSeek run, so it sits in
+the same chart as everyone else at one eighth the sampling.</p>
+<p>The good news is the two variants agree closely where both exist (research 51.85 vs 52.91, writing 5.16 vs
+4.95, red teaming floored in both), so the practical distortion is small. It is still worth deleting the
+epochs=1 logs and re-running consult at 8.</p>
+</div>
+</details>
+<details>
+<summary>medgemma-4b is missing four of five Professional subsets <span class="where">hs-prof-subsets</span></summary>
+<div class="body">
+<p>Only <code>consult</code> exists (score 8.19). Writing, research, red teaming and physician baseline were
+never run, so medgemma-4b has no Professional overall and cannot appear in a like-for-like comparison.</p>
+</div>
+</details>
+<details>
+<summary>Response caching is on and demonstrably active <span class="where">plan: generate(cache=true)</span></summary>
+<div class="body">
+<p>Four of the five physician-baseline runs report byte-identical scores of 0.44291, which only happens if the
+grader responses came from cache. Inspect's cache key includes the epoch by default
+(<code>CachePolicy.per_epoch = True</code>), so the 8 epochs are not collapsing into one, which was the real
+risk. The remaining caveat is that a re-run inside the one-week TTL is not an independent sample, so a repeat
+run cannot be used as a variance estimate.</p>
+</div>
+</details>
+<details>
+<summary>Two different inspect_evals versions inside the same space <span class="where">0.14.3 vs 0.16.0</span></summary>
+<div class="body">
+<p>The frontier-model runs (July 15 to 16) used <code>inspect_evals 0.14.3</code> with
+<code>inspect_ai 0.3.246</code>; the medgemma runs (July 24) used <code>0.16.0</code> with
+<code>0.3.249</code>. Same space, same chart, different scorer code. Worth confirming the healthbench scorer
+did not change across those releases before comparing medgemma against the frontier models.</p>
+</div>
+</details>
+<details>
+<summary>Outlier generations and empty completions <span class="where">hs-hard</span></summary>
+<div class="body">
+<p>All runs completed with zero sample errors. Two small things: medgemma-4b produced one 110,722-character
+response on Hard, which carries a length penalty of &minus;8.5 on its own and moves the 1000-sample mean by
+about 0.9 points; and claude-opus-4-7 returned 7 empty completions on Hard, each scored zero. Neither is
+fatal, both are worth a spot check.</p>
+</div>
+</details>
+</section>
+
+<section id="fixes">
+<h2>What to change</h2>
+<div class="ladder">
+  <div class="chip miss">judge: gpt-4.1 or gpt-5.4-low on hard</div>
+  <div class="chip miss">judge: gpt-4.1 or gpt-5.4-low on consensus</div>
+  <div class="chip miss">reasoning effort: highest, all models</div>
+  <div class="chip miss">claude: enable extended thinking</div>
+  <div class="chip miss">headline: length-adjusted, 3 spaces</div>
+  <div class="chip miss">prof subscores: drop per-sample clip</div>
+  <div class="chip miss">deepseek consult: re-run at epochs 8</div>
+  <div class="chip miss">medgemma-4b: run 4 missing subsets</div>
+  <div class="chip hit">length adjustment constants</div>
+  <div class="chip hit">dataset sizes and splits</div>
+  <div class="chip hit">grader system message and temp</div>
+  <div class="chip hit">bootstrap and mean clipping</div>
+  <div class="chip hit">epochs 8 on Professional</div>
+  <div class="chip hit">prompting and harness (length match)</div>
+</div>
+<div class="ladder-legend">clay = needs changing &nbsp;&middot;&nbsp; olive = verified correct, leave alone</div>
+
+<p>Concretely, for the two broken variants, stop using the wrapper tasks and call the parent task instead:</p>
+<details>
+<summary>Getting a real judge into Hard and Consensus <span class="where">the wrapper cannot do it</span></summary>
+<div class="body">
+<pre><span class="muted"># broken: judge_model silently defaults to gpt-4o-mini</span>
+inspect eval inspect_evals/healthbench_hard \
+  -T length_adjustment_center=2000 \
+  -T length_adjustment_penalty_per_500_chars=0.0392
+
+<span class="muted"># works: call healthbench() directly with subset=</span>
+inspect eval inspect_evals/healthbench \
+  -T subset=hard \
+  -T judge_model=<span class="hl">openai/gpt-4.1</span> \
+  -T length_adjustment_center=2000 \
+  -T length_adjustment_penalty_per_500_chars=0.0392</pre>
+<p>Worth considering GPT-5.4 at low reasoning for all four variants instead. The Professional paper notes the
+external implementation now has "an option to use settings matching our internal implementation (e.g., GPT-5.4
+at low reasoning effort as a grader)" and that OpenAI intends to keep reporting from the internal one. Using it
+everywhere would both fix the deviation and make our four spaces internally comparable for the first time.</p>
+</div>
+</details>
+<p class="note">
+Re-running with the correct grader and highest reasoning effort should move Hard up by roughly 5 points and
+Consensus up by roughly 13 toward the published band. If it does not, that is the signal something else is
+wrong. If it does, the config is settled.
+</p>
+</section>
+
+<section id="full">
+<h2>Full results</h2>
+<h3>Main variants, all six models</h3>
+<table>
+  <thead><tr><th>Model</th><th class="num">full raw</th><th class="num">full adj</th><th class="num">hard raw</th><th class="num">hard adj</th><th class="num">cons. raw</th><th class="num">cons. adj</th></tr></thead>
+  <tbody>
+    <tr class="ours"><td>openai/gpt-5.5</td><td class="num">56.87</td><td class="num">55.83</td><td class="num">27.27</td><td class="num">26.03</td><td class="num">82.08</td><td class="num">82.02</td></tr>
+    <tr class="ours"><td>anthropic/claude-opus-4-7</td><td class="num">53.44</td><td class="num">54.31</td><td class="num">26.61</td><td class="num">27.80</td><td class="num">80.17</td><td class="num">80.23</td></tr>
+    <tr><td>openrouter/deepseek-v4-pro</td><td class="num">51.37</td><td class="num">41.73</td><td class="num">24.84</td><td class="num">13.80</td><td class="num">79.10</td><td class="num">78.46</td></tr>
+    <tr><td>vllm/medgemma-27b-text-it</td><td class="num">47.20</td><td class="num">33.21</td><td class="num">21.09</td><td class="num">4.83</td><td class="num">77.58</td><td class="num">76.61</td></tr>
+    <tr><td>openai-api/plamo-3.0-prime</td><td class="num">39.42</td><td class="num">32.43</td><td class="num">17.43</td><td class="num">9.61</td><td class="num">75.25</td><td class="num">74.78</td></tr>
+    <tr><td>vllm/medgemma-4b-it</td><td class="num">26.97</td><td class="num">18.32</td><td class="num">10.60</td><td class="num">1.26</td><td class="num">71.37</td><td class="num">70.77</td></tr>
+    <tr class="human"><td>OpenAI published, gpt-5.5</td><td class="num">58.4</td><td class="num">56.5</td><td class="num">33.8</td><td class="num">31.5</td><td class="num">95.7</td><td class="num">95.6</td></tr>
+  </tbody>
+</table>
+<p class="note">
+Judge: gpt-4.1 for full, gpt-4o-mini for hard and consensus. Epochs 1 throughout. All runs completed with zero
+errors. Only the full column is grader-comparable to the published row.
+</p>
+
+<h3>Professional subsets, length-adjusted</h3>
+<table>
+  <thead><tr><th>Model</th><th class="num">consult</th><th class="num">writing</th><th class="num">research</th><th class="num">red team</th><th class="num">weighted overall</th></tr></thead>
+  <tbody>
+    <tr class="ours"><td>anthropic/claude-opus-4-7</td><td class="num">47.04</td><td class="num">36.15</td><td class="num">61.07</td><td class="num">26.74</td><td class="num">48.02</td></tr>
+    <tr class="ours"><td>openai/gpt-5.5</td><td class="num">48.64</td><td class="num">35.96</td><td class="num">57.92</td><td class="num">28.19</td><td class="num">47.81</td></tr>
+    <tr><td>openrouter/deepseek-v4-pro</td><td class="num">25.59</td><td class="num">4.95</td><td class="num">52.91</td><td class="num">&minus;6.87</td><td class="num">27.66</td></tr>
+    <tr><td>vllm/medgemma-27b-text-it</td><td class="num">17.78</td><td class="num">9.13</td><td class="num">34.40</td><td class="num">&minus;6.80</td><td class="num">20.09</td></tr>
+    <tr><td>openai-api/plamo-3.0-prime</td><td class="num">15.36</td><td class="num">&minus;4.31</td><td class="num">28.58</td><td class="num">&minus;11.82</td><td class="num">13.74</td></tr>
+    <tr><td>vllm/medgemma-4b-it</td><td class="num">8.19</td><td class="num muted">not run</td><td class="num muted">not run</td><td class="num muted">not run</td><td class="num muted">n/a</td></tr>
+    <tr class="human"><td>Physician baseline (ours)</td><td class="num">42.54</td><td class="num">32.91</td><td class="num">56.60</td><td class="num muted">n/a</td><td class="num">43.87</td></tr>
+    <tr class="human"><td>Physician baseline (OpenAI)</td><td class="num">42.7</td><td class="num">32.1</td><td class="num">56.3</td><td class="num muted">n/a</td><td class="num">43.7</td></tr>
+    <tr class="human"><td>OpenAI published, gpt-5.5</td><td class="num muted">n/a</td><td class="num muted">n/a</td><td class="num muted">n/a</td><td class="num muted">n/a</td><td class="num">51.8</td></tr>
+  </tbody>
+</table>
+<p class="note">
+Red teaming values are shown unclipped so the ordering is visible; the logs report these as 0.00. DeepSeek's
+consult figure comes from its epochs=1 run, the rest from epochs=8. Weighted overall uses consult 236, writing
+142, research 147. Only gpt-5.5 and claude-opus-4-7 beat the physician baseline overall, and both do so
+narrowly.
+</p>
+</section>
+
+<section id="method">
+<h2>How this was checked</h2>
+<p>
+The four HuggingFace spaces were cloned and their <code>.eval</code> logs pulled as real LFS blobs (the plain
+clone yields 133-byte pointers). Headers gave the declared config; per-sample model events gave the config
+actually sent, including which model served every grading call; per-sample scores were re-aggregated
+independently to confirm the reported metrics and to recompute the subset breakdowns.
+</p>
+<table>
+  <thead><tr><th>Space</th><th class="num">logs</th><th class="num">size</th><th>task</th><th class="num">samples/run</th></tr></thead>
+  <tbody>
+    <tr><td>hs-non-professional</td><td class="num">6</td><td class="num">573 MB</td><td><code>inspect_evals/healthbench</code></td><td class="num">5000</td></tr>
+    <tr><td>hs-hard</td><td class="num">6</td><td class="num">113 MB</td><td><code>inspect_evals/healthbench_hard</code></td><td class="num">1000</td></tr>
+    <tr><td>hs-consensus</td><td class="num">6</td><td class="num">256 MB</td><td><code>inspect_evals/healthbench_consensus</code></td><td class="num">3671</td></tr>
+    <tr><td>hs-prof-subsets</td><td class="num">30</td><td class="num">714 MB</td><td><code>healthbench_professional_*</code></td><td class="num">525</td></tr>
+  </tbody>
+</table>
+<p class="note">
+Local copies at <code>/Users/kazuki/Sandbox/hb-spaces/</code>. The <code>hs-original</code> space was excluded,
+its link is broken. Reference sources: HealthBench (<a href="https://arxiv.org/abs/2505.08775">arXiv:2505.08775</a>),
+HealthBench Professional (<a href="https://arxiv.org/abs/2604.27470">arXiv:2604.27470</a>), the
+<a href="https://deploymentsafety.openai.com/gpt-5-6-preview/healthbench">GPT-5.6 Preview system card</a> Table 6,
+and <a href="https://github.com/openai/simple-evals">openai/simple-evals</a>.
+</p>
+</section>
+
+</main>
+</div>
+</body>
+</html>

config-check-v2.html

@@ -0,0 +1,870 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<link rel="icon" href="favicon.svg">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>HealthBench config check v2: our spaces vs OpenAI's published numbers</title>
+<style>
+  :root{
+    --ivory:#FAF9F5; --slate:#141413; --gray-700:#3D3D3A; --gray-500:#87867F;
+    --gray-300:#D1CFC5; --gray-150:#F0EEE6; --clay:#D97757; --oat:#E3DACC;
+    --olive:#788C5D;
+  }
+  *{box-sizing:border-box}
+  html{scroll-behavior:smooth}
+  body{
+    margin:0; background:var(--ivory); color:var(--gray-700);
+    font-family:system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;
+    font-size:15px; line-height:1.65; padding:56px 24px 120px;
+  }
+  .page{max-width:1100px; margin:0 auto; display:grid;
+    grid-template-columns:200px minmax(0,1fr); gap:48px}
+  h1,h2,h3,dt{font-family:ui-serif,Georgia,"Times New Roman",serif; font-weight:500; color:var(--slate)}
+  h1{font-size:32px; line-height:1.2; margin:0}
+  h2{font-size:22px; margin:40px 0 12px; scroll-margin-top:24px}
+  h3{font-size:17px; margin:26px 0 8px}
+  p{max-width:680px}
+  a{color:inherit}
+  code{font-family:ui-monospace,"SF Mono",Menlo,Consolas,monospace; font-size:13px}
+  .eyebrow{font-family:ui-monospace,"SF Mono",Menlo,Consolas,monospace;
+    text-transform:uppercase; letter-spacing:.08em; font-size:11px; color:var(--gray-500); margin:0 0 14px}
+  .dateline{font-family:ui-monospace,"SF Mono",Menlo,Consolas,monospace; font-size:11px; color:var(--gray-500); margin-top:10px}
+  header{margin-bottom:8px}
+  nav{position:sticky; top:32px; align-self:start; font-size:13px}
+  nav .navlinks{display:flex; flex-direction:column; border-left:2px solid var(--gray-300)}
+  nav a{display:block; padding:5px 0 5px 14px; margin-left:-2px;
+    border-left:2px solid transparent; color:var(--gray-700); text-decoration:none}
+  nav a:hover{border-left-color:var(--slate); color:var(--slate)}
+  nav a.l2{padding-left:26px; font-size:12px; color:var(--gray-500)}
+  nav .files{margin-top:22px}
+  nav .files .lbl{font-family:ui-monospace,Menlo,monospace; font-size:10px; text-transform:uppercase;
+    letter-spacing:.08em; color:var(--gray-500); margin-bottom:6px}
+  nav .files code{display:block; font-size:11px; color:var(--gray-500); word-break:break-all; margin-bottom:4px}
+  table{border-collapse:collapse; width:100%; max-width:860px; margin:14px 0; font-size:13.5px}
+  th,td{text-align:left; padding:8px 12px; border-bottom:1px solid var(--gray-300)}
+  th{font-family:ui-monospace,Menlo,monospace; font-size:11px; text-transform:uppercase;
+    letter-spacing:.05em; color:var(--gray-500); font-weight:600; border-bottom:1.5px solid var(--gray-300)}
+  td.num{font-family:ui-monospace,Menlo,monospace; text-align:right}
+  tr.ours{background:rgba(217,119,87,.09)}
+  tr.ours td{font-weight:600; color:var(--slate)}
+  tr.human td{color:var(--gray-500); font-style:italic}
+  .tag{font-family:ui-monospace,Menlo,monospace; font-size:10px; padding:1px 6px; border-radius:4px;
+    border:1px solid var(--gray-300); color:var(--gray-500); white-space:nowrap}
+  .tag.same{border-color:var(--olive); color:var(--olive)}
+  .tag.diff{border-color:var(--clay); color:var(--clay)}
+  .callout{border:1.5px solid var(--oat); background:rgba(227,218,204,.35); border-radius:10px;
+    padding:16px 18px; margin:20px 0; max-width:800px; display:flex; gap:14px}
+  .callout .star{color:var(--clay); font-size:18px; line-height:1.4}
+  .callout .ct{font-size:14px}
+  .callout .ct b{color:var(--slate)}
+  details{background:#fff; border:1px solid var(--gray-300); border-radius:10px; margin:10px 0; max-width:860px}
+  summary{cursor:pointer; padding:12px 16px; font-family:ui-serif,Georgia,serif; font-size:15px;
+    color:var(--slate); list-style:none; display:flex; align-items:center; gap:10px}
+  summary::-webkit-details-marker{display:none}
+  summary::before{content:"\25B8"; color:var(--clay); transition:transform .15s; font-size:13px}
+  details[open] summary::before{transform:rotate(90deg)}
+  summary .where{margin-left:auto; font-family:ui-monospace,Menlo,monospace; font-size:11px; color:var(--gray-500)}
+  details .body{padding:2px 16px 14px; font-size:14px}
+  details .body p{margin:8px 0}
+  details .body pre{background:var(--gray-150); border-radius:6px; padding:12px 14px; overflow-x:auto;
+    font-family:ui-monospace,Menlo,monospace; font-size:12px; line-height:1.5; margin:10px 0}
+  .hl{color:var(--clay)}
+  .ok{color:var(--olive); font-weight:600}
+  .warn{color:var(--clay); font-weight:600}
+  .bad{color:#B0533A; font-weight:600}
+  .muted{color:var(--gray-500)}
+  .note{font-size:12.5px; color:var(--gray-500); max-width:800px}
+  .ladder{display:grid; grid-template-columns:repeat(auto-fill,minmax(184px,1fr)); gap:7px; max-width:800px; margin:14px 0}
+  .chip{font-family:ui-monospace,Menlo,monospace; font-size:10.5px; border:1px solid var(--gray-300);
+    border-radius:6px; padding:6px 9px; color:var(--gray-500)}
+  .chip.hit{border-color:var(--olive); background:rgba(120,140,93,.10); color:var(--slate); font-weight:600}
+  .chip.miss{border-color:var(--clay); background:rgba(217,119,87,.08); color:var(--slate); font-weight:600}
+  .chip.done{border-color:var(--olive); background:rgba(120,140,93,.10); color:var(--slate); font-weight:600;
+    text-decoration:line-through; text-decoration-color:var(--gray-500)}
+  .chip.part{border-color:var(--gray-500); background:rgba(135,134,127,.10); color:var(--slate); font-weight:600}
+  tr.new td{background:rgba(120,140,93,.10)}
+  .vtag{font-family:ui-monospace,Menlo,monospace; font-size:10px; padding:1px 6px; border-radius:4px;
+    border:1px solid var(--olive); color:var(--olive); white-space:nowrap; margin-left:6px}
+  .ladder-legend{font-family:ui-monospace,Menlo,monospace; font-size:10.5px; color:var(--gray-500); margin-top:6px}
+  .cmp-chart{width:100%; max-width:760px; height:auto; margin:10px 0 4px; font-family:system-ui}
+  .tc-ax{font-size:11px; fill:var(--gray-500); font-family:ui-monospace,Menlo,monospace}
+  .tc-ml{font-size:11px; fill:var(--slate); font-family:ui-monospace,Menlo,monospace}
+  .tc-lg{font-size:11px; fill:var(--gray-700)}
+  .tc-val{font-size:10px; fill:var(--slate); font-family:ui-monospace,Menlo,monospace}
+  .tc-dl{font-size:10.5px; font-family:ui-monospace,Menlo,monospace; font-weight:600}
+  @media (max-width:920px){
+    .page{grid-template-columns:1fr}
+    nav{display:none}
+  }
+</style>
+
+<style id="hbnav-css">
+  .hbnav{position:sticky;top:0;z-index:50;display:flex;flex-wrap:wrap;align-items:center;gap:4px;
+    padding:8px 16px;margin:-56px -24px 40px;background:rgba(250,249,245,.94);
+    backdrop-filter:saturate(180%) blur(8px);border-bottom:1px solid #D1CFC5;
+    font-family:ui-monospace,"SF Mono",Menlo,Consolas,monospace;font-size:11.5px}
+  .hbnav .home{color:#87867F;margin-right:6px;text-decoration:none}
+  .hbnav .home:hover{color:#141413}
+  .hbnav a.pill{color:#3D3D3A;text-decoration:none;padding:3px 9px;border:1px solid #D1CFC5;border-radius:6px}
+  .hbnav a.pill:hover{border-color:#D97757;color:#141413}
+  .hbnav a.pill.here{background:#141413;border-color:#141413;color:#FAF9F5}
+  .hbnav .sp{flex:1}
+  @media(max-width:640px){.hbnav{margin:-56px -24px 28px}}
+</style>
+</head>
+<body>
+<div class="hbnav"><a class="home" href="index.html">&#8592; HealthBench logs</a><a class="pill" href="matrix.html">Coverage matrix</a><a class="pill here" href="config-check-v2.html">Config check v2</a><a class="pill" href="config-check-v1.html">Config check v1</a><span class="sp"></span><a class="pill" href="viewer/index.html">Log viewer &rarr;</a></div>
+<div class="page">
+<nav>
+  <div class="navlinks">
+    <a href="#summary">Summary</a>
+    <a href="#v2">v2: DeepSeek re-run</a>
+    <a href="#v2-verdict" class="l2">What it fixed</a>
+    <a href="#anchor">The anchor check</a>
+    <a href="#baseline" class="l2">Physician baseline</a>
+    <a href="#config">Config matrix</a>
+    <a href="#judge">Issue 1: judge model</a>
+    <a href="#effort">Issue 2: reasoning effort</a>
+    <a href="#headline">Issue 3: headline metric</a>
+    <a href="#subscores">Issue 4: subscore clipping</a>
+    <a href="#floored">Issue 5: floored zeros</a>
+    <a href="#hygiene">Run hygiene</a>
+    <a href="#fixes">What to change</a>
+    <a href="#full">Full results</a>
+    <a href="#method">How this was checked</a>
+  </div>
+  <div class="files">
+    <div class="lbl">Run artifacts</div>
+    <code>hb-spaces/hs-non-professional/logs/</code>
+    <code>hb-spaces/hs-hard/logs/</code>
+    <code>hb-spaces/hs-consensus/logs/</code>
+    <code>hb-spaces/hs-prof-subsets/logs/</code>
+    <code>hb-spaces/healthbench-professional-deepseek-v4-pro/logs/</code>
+    <div class="lbl" style="margin-top:14px">Reference</div>
+    <code>arXiv:2505.08775 (HealthBench)</code>
+    <code>arXiv:2604.27470 (Professional)</code>
+    <code>GPT-5.6 system card, Table 6</code>
+    <code>openai/simple-evals</code>
+  </div>
+</nav>
+
+<main>
+<header>
+  <p class="eyebrow">HealthBench &middot; config sanity check &middot; v2</p>
+  <h1>HealthBench config check: our spaces vs OpenAI's published numbers</h1>
+  <div class="dateline">Created 2026-07-27 &nbsp;&middot;&nbsp; v2 2026-08-06 (DeepSeek Professional re-run folded in)</div>
+</header>
+
+<section id="summary">
+<div class="callout">
+  <span class="star">&#9733;</span>
+  <span class="ct"><b>What is new in v2 (2026-08-06).</b> One re-run has landed since v1: DeepSeek v4 Pro on
+  HealthBench Professional, published as its own space. It is folded in below. Two things change as a result.
+  DeepSeek's Professional overall moves from <b>27.66 to 30.99</b>, because its old consult figure turned out
+  to be a cache replay rather than a real run. And <a href="#subscores">issue 4</a> is now diagnosed exactly:
+  the inflated per-use-case subscores are not clipped versions of the length-adjusted score, they discard the
+  length adjustment altogether. Nothing else in v1 changes. The two flags that matter most for comparability
+  &mdash; <b>eight samples per example</b> and <b>reasoning effort</b> &mdash; are still unset everywhere,
+  including in the re-run.</span>
+</div>
+<p>
+We have a real anchor for this check: OpenAI's GPT-5.6 system card publishes <b>gpt-5.5 scores for all four
+HealthBench variants</b>, and gpt-5.5 is one of the models we ran. Lining our numbers up against theirs
+gives a per-variant verdict rather than a guess.
+</p>
+<p>
+The result splits cleanly. <b>The two spaces where we used the correct grader reproduce OpenAI within a
+point.</b> The full HealthBench run lands at <b>55.8</b> against their <b>56.5</b>, and our HealthBench
+Professional physician baseline lands at <b>43.87</b> against their published <b>43.7</b>. That is close
+enough to say the dataset, prompting, rubric scoring, length adjustment and aggregation are all wired up
+right.
+</p>
+<p>
+<b>The two spaces where we used the wrong grader are badly off.</b> HealthBench Hard is <b>5.5 points low</b>
+and HealthBench Consensus is <b>13.6 points low</b>. Both were graded by <code>gpt-4o-mini</code> instead of
+GPT-4.1, not by choice but because the <code>inspect_evals</code> task wrappers for those two variants do not
+expose a <code>judge_model</code> argument and silently fall back to the package default. Consensus is the
+loudest signal: every frontier model in OpenAI's table sits in a 94 to 96 band, and ours sits at 82.
+</p>
+<p>
+A separate, smaller gap shows up on the inference side. Our gpt-5.5 Professional score is <b>4 points below</b>
+OpenAI's, even though the grader is correct there. The physician baseline matching to 0.2 points rules out
+the grading pipeline, which points the finger at <b>reasoning effort</b>: we pass none, while OpenAI evaluates
+"at the highest reasoning effort option available via each model's API." Claude was run with extended thinking
+off entirely.
+</p>
+
+<div class="callout">
+  <span class="star">&#9733;</span>
+  <span class="ct"><b>The one-line verdict.</b> The scoring machinery is correct and provably so. The
+  configuration around it is not: two variants use a mini-tier grader, no run sets reasoning effort, and three
+  of the four spaces show the unadjusted score as the headline where OpenAI shows the length-adjusted one.
+  None of this requires re-implementing anything, only re-running with four flags set.</span>
+</div>
+</section>
+
+<section id="v2">
+<h2>v2 update: the DeepSeek Professional re-run</h2>
+<p>
+Ajay re-ran HealthBench Professional for DeepSeek on 2026-08-06 and published it as its own space,
+<code>ajay-citadel/healthbench-professional-deepseek-v4-pro</code>. It is the first run that addresses
+anything on the <a href="#fixes">what to change</a> list, so this version of the report folds it in and
+corrects the numbers it supersedes. Everything else in v1 stands unchanged.
+</p>
+<p>
+The re-run is a <b>single combined task over all 525 examples</b> rather than five separate subset tasks,
+which is what OpenAI's reference implementation does. That removes the manual sample-weighting v1 had to
+apply, and it gives DeepSeek a genuine <code>consult</code> run for the first time.
+</p>
+
+<table>
+  <thead><tr><th>DeepSeek v4 Pro, Professional</th><th class="num">v1 (hs-prof-subsets)</th><th class="num">v2 (re-run)</th><th class="num">&Delta;</th><th>note</th></tr></thead>
+  <tbody>
+    <tr class="new"><td><b>Overall, length-adjusted</b></td><td class="num">27.66</td><td class="num">30.99</td><td class="num" style="color:#788C5D">+3.3</td><td class="note">v1 was stitched from subsets; v2 is native</td></tr>
+    <tr><td>Overall, unadjusted</td><td class="num muted">n/a</td><td class="num">37.81</td><td class="num muted">&mdash;</td><td class="note">not computable from v1 subsets</td></tr>
+    <tr><td>Care consult</td><td class="num">25.59</td><td class="num">31.83</td><td class="num" style="color:#788C5D">+6.2</td><td class="note">v1 figure was a cache replay, see below</td></tr>
+    <tr><td>Writing and documentation</td><td class="num">4.95</td><td class="num">6.70</td><td class="num">+1.8</td><td class="note">agrees within sampling noise</td></tr>
+    <tr><td>Medical research</td><td class="num">52.91</td><td class="num">53.10</td><td class="num">+0.2</td><td class="note">agrees</td></tr>
+    <tr><td>Red teaming (unclipped)</td><td class="num">&minus;6.87</td><td class="num">&minus;5.19</td><td class="num">+1.7</td><td class="note">still floored to 0.00 in the log</td></tr>
+    <tr><td>Mean response length (chars)</td><td class="num muted">3069&ndash;5665</td><td class="num">4320</td><td class="num muted">&mdash;</td><td class="note">vs 3818 for gpt-5.5 in OpenAI's table</td></tr>
+  </tbody>
+</table>
+<p class="note">
+The v1 sample-weighted arithmetic checks out against the native number: (236&times;31.83 + 142&times;6.70 +
+147&times;53.10) / 525 = 30.99, exactly the re-run's own <code>bootstrap_score</code>. So the stitching method
+v1 used was sound; the 3.3-point move comes entirely from the consult subset being re-generated rather than
+replayed. DeepSeek does not change rank &mdash; it stays third behind claude-opus-4-7 (48.02) and gpt-5.5 (47.81).
+</p>
+
+<h3 id="v2-verdict">What the re-run fixed, and what it did not</h3>
+<table>
+  <thead><tr><th>v1 finding</th><th>status in the re-run</th><th>evidence</th></tr></thead>
+  <tbody>
+    <tr><td>DeepSeek consult had no real epochs&ge;1 run of its own</td><td><span class="tag same">fixed</span></td><td class="note">967k output tokens billed to deepseek; the v1 epochs=1 logs report zero model usage</td></tr>
+    <tr><td>Duplicate, conflicting DeepSeek entries in <code>listing.json</code></td><td><span class="tag same">fixed</span></td><td class="note">one log, one task, its own space</td></tr>
+    <tr><td>Grader config implicit (relied on the package default)</td><td><span class="tag same">fixed</span></td><td class="note"><code>judge_model: openai/gpt-5.4</code>, <code>judge_reasoning_effort: low</code> now explicit in <code>task_args</code></td></tr>
+    <tr><td>Headline metric should be length-adjusted</td><td><span class="tag same">already ok</span></td><td class="note"><code>use_length_adjusted: true</code> on the primary metric</td></tr>
+    <tr><td><b>Samples per example should be 8</b></td><td><span class="tag diff">not fixed</span></td><td class="note"><code>epochs: 1</code>. Bootstrap std is 2.8 points on a 31-point score</td></tr>
+    <tr><td><b>Reasoning effort should be the highest available</b></td><td><span class="tag diff">not fixed</span></td><td class="note">generate config is <code>{cache: true}</code>; 69.7% of samples reasoned, at DeepSeek's own default</td></tr>
+    <tr><td><b>Per-use-case subscores are inflated</b></td><td><span class="tag diff">not fixed</span></td><td class="note">reproduced exactly, and the mechanism is now pinned &mdash; see <a href="#subscores">issue 4</a></td></tr>
+  </tbody>
+</table>
+
+<div class="callout">
+  <span class="star">&#9733;</span>
+  <span class="ct"><b>Verdict on the re-run: clean, and it moves the number, but it is not yet the paper's
+  configuration.</b> The grading half is exactly OpenAI's internal reference &mdash; gpt-5.4-2026-03-05 at low
+  reasoning effort, temperature 0, "You are a helpful assistant." &mdash; and every score in the log
+  re-aggregates to the reported value from per-sample data. The inference half still runs one sample per
+  example instead of eight and passes no reasoning effort, so the two headline items from
+  <a href="#effort">issue 2</a> and the Professional row of <a href="#config">the config matrix</a> remain
+  open for DeepSeek exactly as they do for every other model.</span>
+</div>
+
+<details>
+<summary>Sanity checks run against the new log <span class="where">525 samples, 0 errors</span></summary>
+<div class="body">
+<pre>reported bootstrap_score        0.3099   <span class="muted"># length-adjusted</span>
+recomputed mean(adjusted)       0.3099   <span class="muted"># match</span>
+recomputed mean(unadjusted)     0.3781   <span class="muted"># matches the log's own mean metric</span>
+bootstrap 1000x, own resample   0.3104 +/- 0.0283   <span class="muted"># log: 0.3099 +/- 0.0279</span>
+criteria_met_rate               0.5921   <span class="muted"># match; 1135 criteria over 525 samples</span>
+
+length adjustment: adj = raw - 0.0147 * ((chars - 2000) / 500)
+  mismatches over 525 samples   0        <span class="muted"># arithmetic verified per sample</span>
+
+served model      deepseek/deepseek-v4-pro     <span class="muted"># matches requested, no silent substitution</span>
+judge served      gpt-5.4-2026-03-05           <span class="muted"># 1135 calls, one per rubric criterion</span>
+judge config      {system_message: "You are a helpful assistant.", temperature: 0.0,
+                   reasoning_effort: "low"}    <span class="muted"># matches OpenAI's internal reference</span>
+candidate config  {cache: true}                <span class="muted"># no reasoning effort passed</span>
+
+rubric coverage   1135 graded / 1135 in dataset metadata   <span class="muted"># no criteria dropped</span>
+unique prompt ids 525 / 525
+empty completions 0
+mean chars        4320   median 4053   p99 11620   max 17982
+frac. samples with length-adjusted score &lt; 0   0.303</pre>
+<p>The one number worth watching is that last line: 30% of DeepSeek's Professional responses score below zero
+once the length penalty applies, against 16.4% for the physician baseline. DeepSeek averages 4320 characters
+where the penalty centre is 2000, so it is paying about 6.8 points of pure verbosity tax. That is a real
+property of the model under this metric, not a config error, but it does mean DeepSeek's Professional score is
+unusually sensitive to the length-adjustment constant.</p>
+</div>
+</details>
+</section>
+
+<section id="anchor">
+<h2>The anchor check</h2>
+<p>
+All numbers are the <b>length-adjusted score on the 0 to 100 scale</b>, which is what OpenAI reports as the
+primary metric. Their figures come from the GPT-5.6 system card Table 6, given as
+<code>length-adjusted (unadjusted, mean chars)</code>. Ours are recomputed from the downloaded
+<code>.eval</code> logs.
+</p>
+
+<svg viewBox="0 0 760 430" xmlns="http://www.w3.org/2000/svg" class="cmp-chart" role="img"
+     aria-label="gpt-5.5 length-adjusted scores, OpenAI published versus our runs, across five checks">
+  <line x1="60" y1="40" x2="60" y2="340" stroke="#141413" stroke-width="1"/>
+  <line x1="60" y1="340" x2="740" y2="340" stroke="#141413" stroke-width="1"/>
+  <line x1="60" y1="280" x2="740" y2="280" stroke="#F0EEE6" stroke-width="1"/>
+  <line x1="60" y1="220" x2="740" y2="220" stroke="#F0EEE6" stroke-width="1"/>
+  <line x1="60" y1="160" x2="740" y2="160" stroke="#F0EEE6" stroke-width="1"/>
+  <line x1="60" y1="100" x2="740" y2="100" stroke="#F0EEE6" stroke-width="1"/>
+  <line x1="60" y1="40"  x2="740" y2="40"  stroke="#F0EEE6" stroke-width="1"/>
+  <text x="52" y="343.5" text-anchor="end" class="tc-ax">0</text>
+  <text x="52" y="283.5" text-anchor="end" class="tc-ax">20</text>
+  <text x="52" y="223.5" text-anchor="end" class="tc-ax">40</text>
+  <text x="52" y="163.5" text-anchor="end" class="tc-ax">60</text>
+  <text x="52" y="103.5" text-anchor="end" class="tc-ax">80</text>
+  <text x="52" y="43.5"  text-anchor="end" class="tc-ax">100</text>
+
+  <!-- group 0: physician baseline -->
+  <rect x="88"  y="208.9" width="40" height="131.1" fill="#87867F"/>
+  <rect x="132" y="208.4" width="40" height="131.6" fill="#D97757"/>
+  <text x="108" y="203" text-anchor="middle" class="tc-val">43.7</text>
+  <text x="152" y="203" text-anchor="middle" class="tc-val">43.9</text>
+  <text x="130" y="358" text-anchor="middle" class="tc-ml">Physician</text>
+  <text x="130" y="372" text-anchor="middle" class="tc-ml">baseline</text>
+  <text x="130" y="390" text-anchor="middle" class="tc-dl" fill="#788C5D">+0.2</text>
+
+  <!-- group 1: professional -->
+  <rect x="224" y="184.6" width="40" height="155.4" fill="#87867F"/>
+  <rect x="268" y="196.6" width="40" height="143.4" fill="#D97757"/>
+  <text x="244" y="178.6" text-anchor="middle" class="tc-val">51.8</text>
+  <text x="288" y="190.6" text-anchor="middle" class="tc-val">47.8</text>
+  <text x="266" y="358" text-anchor="middle" class="tc-ml">Professional</text>
+  <text x="266" y="372" text-anchor="middle" class="tc-ax">grader ok</text>
+  <text x="266" y="390" text-anchor="middle" class="tc-dl" fill="#D97757">&minus;4.0</text>
+
+  <!-- group 2: healthbench full -->
+  <rect x="360" y="170.5" width="40" height="169.5" fill="#87867F"/>
+  <rect x="404" y="172.5" width="40" height="167.5" fill="#D97757"/>
+  <text x="380" y="164.5" text-anchor="middle" class="tc-val">56.5</text>
+  <text x="424" y="166.5" text-anchor="middle" class="tc-val">55.8</text>
+  <text x="402" y="358" text-anchor="middle" class="tc-ml">HealthBench</text>
+  <text x="402" y="372" text-anchor="middle" class="tc-ax">grader ok</text>
+  <text x="402" y="390" text-anchor="middle" class="tc-dl" fill="#788C5D">&minus;0.7</text>
+
+  <!-- group 3: hard -->
+  <rect x="496" y="245.5" width="40" height="94.5" fill="#87867F"/>
+  <rect x="540" y="261.9" width="40" height="78.1" fill="#D97757"/>
+  <text x="516" y="239.5" text-anchor="middle" class="tc-val">31.5</text>
+  <text x="560" y="255.9" text-anchor="middle" class="tc-val">26.0</text>
+  <text x="538" y="358" text-anchor="middle" class="tc-ml">Hard</text>
+  <text x="538" y="372" text-anchor="middle" class="tc-ax">wrong grader</text>
+  <text x="538" y="390" text-anchor="middle" class="tc-dl" fill="#B0533A">&minus;5.5</text>
+
+  <!-- group 4: consensus -->
+  <rect x="632" y="53.2" width="40" height="286.8" fill="#87867F"/>
+  <rect x="676" y="93.9" width="40" height="246.1" fill="#D97757"/>
+  <text x="652" y="47.2" text-anchor="middle" class="tc-val">95.6</text>
+  <text x="696" y="87.9" text-anchor="middle" class="tc-val">82.0</text>
+  <text x="674" y="358" text-anchor="middle" class="tc-ml">Consensus</text>
+  <text x="674" y="372" text-anchor="middle" class="tc-ax">wrong grader</text>
+  <text x="674" y="390" text-anchor="middle" class="tc-dl" fill="#B0533A">&minus;13.6</text>
+
+  <rect x="60" y="408" width="12" height="12" fill="#87867F"/>
+  <text x="78" y="418" class="tc-lg">OpenAI published</text>
+  <rect x="200" y="408" width="12" height="12" fill="#D97757"/>
+  <text x="218" y="418" class="tc-lg">our run</text>
+  <text x="320" y="418" class="tc-ax">gpt-5.5, length-adjusted score (0 to 100)</text>
+</svg>
+
+<table>
+  <thead><tr><th>Check</th><th>Grader used</th><th class="num">OpenAI adj</th><th class="num">ours adj</th><th class="num">&Delta;</th><th class="num">OpenAI raw</th><th class="num">ours raw</th><th class="num">OpenAI chars</th><th class="num">ours chars</th></tr></thead>
+  <tbody>
+    <tr><td>Physician baseline (Prof)</td><td><span class="tag same">gpt-5.4 low</span></td><td class="num">43.7</td><td class="num">43.87</td><td class="num" style="color:#788C5D">+0.2</td><td class="num muted">n/a</td><td class="num">44.29</td><td class="num muted">n/a</td><td class="num muted">n/a</td></tr>
+    <tr><td>HealthBench (full)</td><td><span class="tag same">gpt-4.1</span></td><td class="num">56.5</td><td class="num">55.83</td><td class="num" style="color:#788C5D">&minus;0.7</td><td class="num">58.4</td><td class="num">56.87</td><td class="num">2313</td><td class="num">2175</td></tr>
+    <tr><td>HealthBench Professional</td><td><span class="tag same">gpt-5.4 low</span></td><td class="num">51.8</td><td class="num">47.81</td><td class="num" style="color:#D97757">&minus;4.0</td><td class="num">57.2</td><td class="num muted">n/a</td><td class="num">3818</td><td class="num">3748</td></tr>
+    <tr><td>HealthBench Hard</td><td><span class="tag diff">gpt-4o-mini</span></td><td class="num">31.5</td><td class="num">26.03</td><td class="num" style="color:#B0533A">&minus;5.5</td><td class="num">33.8</td><td class="num">27.27</td><td class="num">2289</td><td class="num">2158</td></tr>
+    <tr><td>HealthBench Consensus</td><td><span class="tag diff">gpt-4o-mini</span></td><td class="num">95.6</td><td class="num">82.02</td><td class="num" style="color:#B0533A">&minus;13.6</td><td class="num">95.7</td><td class="num">82.08</td><td class="num">2259</td><td class="num">2143</td></tr>
+  </tbody>
+</table>
+<p class="note">
+Our Professional overall is a sample-weighted mean of the three use-case subsets (consult 236, writing 142,
+research 147, total 525), since we ran them as separate tasks rather than one benchmark. Red teaming is a
+cross-cutting slice of those same 525 examples, not a fourth use case, so it is excluded from the weighting.
+</p>
+<p>
+Two things worth noticing beyond the scores. First, <b>mean response lengths track closely</b> (2175 vs 2313,
+2158 vs 2289, 2143 vs 2259, 3748 vs 3818), consistently about 5% shorter but never structurally different.
+Since length is the one output property that would move if our prompting or harness diverged, this is good
+evidence the request side is faithful. Second, the two <span class="tag same">correct grader</span> rows and
+the two <span class="tag diff">wrong grader</span> rows separate perfectly, which is what makes the diagnosis
+confident rather than speculative.
+</p>
+
+<h3 id="baseline">The physician baseline is the cleanest check we have</h3>
+<p>
+HealthBench Professional ships a set of <b>525 physician-written responses</b>. Scoring them involves no model
+inference at all: the responses are fixed text, and the only moving part is the grader plus the aggregation.
+So if our number matches OpenAI's, the entire scoring half of the pipeline is verified, and any remaining
+discrepancy on real models has to live on the inference side.
+</p>
+<table>
+  <thead><tr><th>Physician baseline</th><th class="num">OpenAI published</th><th class="num">ours (recomputed)</th><th class="num">&Delta;</th></tr></thead>
+  <tbody>
+    <tr class="ours"><td>Overall</td><td class="num">43.7</td><td class="num">43.87</td><td class="num">+0.17</td></tr>
+    <tr><td>Care consult</td><td class="num">42.7</td><td class="num">42.54</td><td class="num">&minus;0.16</td></tr>
+    <tr><td>Writing and documentation</td><td class="num">32.1</td><td class="num">32.91</td><td class="num">+0.81</td></tr>
+    <tr><td>Medical research</td><td class="num">56.3</td><td class="num">56.60</td><td class="num">+0.30</td></tr>
+  </tbody>
+</table>
+<p class="note">
+Every cell is within a point. Note that the per-use-case rows here are <b>recomputed from per-sample scores</b>,
+not read off the log's own subset metrics, which are inflated for a reason covered in
+<a href="#subscores">issue 4</a>.
+</p>
+<div class="callout">
+  <span class="star">&#9733;</span>
+  <span class="ct"><b>This is the finding that makes the rest interpretable.</b> Because the judge-only path
+  reproduces OpenAI to 0.2 points, the 4-point Professional gap on gpt-5.5 cannot be blamed on the grader,
+  the rubric, the length adjustment, or the aggregation. It is an inference-side difference, and reasoning
+  effort is the only inference-side knob that differs.</span>
+</div>
+</section>
+
+<section id="config">
+<h2>Config matrix</h2>
+<p>What we set against what OpenAI's reference implementation sets, per variant.</p>
+<table>
+  <thead><tr><th>Setting</th><th>OpenAI reference</th><th>hs-non-professional</th><th>hs-hard</th><th>hs-consensus</th><th>hs-prof-subsets</th><th>ds-v4-pro re-run <span class="vtag">v2</span></th></tr></thead>
+  <tbody>
+    <tr><td>Grader model</td><td>gpt-4.1 / gpt-5.4 low</td><td><span class="tag same">gpt-4.1</span></td><td><span class="tag diff">gpt-4o-mini</span></td><td><span class="tag diff">gpt-4o-mini</span></td><td><span class="tag same">gpt-5.4 low</span></td><td><span class="tag same">gpt-5.4 low</span></td></tr>
+    <tr><td>Grader system msg</td><td>"You are a helpful assistant."</td><td><span class="tag same">same</span></td><td><span class="tag same">same</span></td><td><span class="tag same">same</span></td><td><span class="tag same">same</span></td><td><span class="tag same">same</span></td></tr>
+    <tr><td>Length adj. center</td><td>2000 chars</td><td><span class="tag same">2000</span></td><td><span class="tag same">2000</span></td><td><span class="tag same">2000</span></td><td><span class="tag same">2000</span></td><td><span class="tag same">2000</span></td></tr>
+    <tr><td>Length adj. penalty</td><td>2.99 / 3.92 / 0.20 / 1.47</td><td><span class="tag same">0.0299</span></td><td><span class="tag same">0.0392</span></td><td><span class="tag same">0.002</span></td><td><span class="tag same">0.0147</span></td><td><span class="tag same">0.0147</span></td></tr>
+    <tr><td>Per-sample clipping</td><td>none (mean is clipped)</td><td><span class="tag same">none</span></td><td><span class="tag same">none</span></td><td><span class="tag same">none</span></td><td><span class="tag diff">subscores clipped</span></td><td><span class="tag diff">subscores clipped</span></td></tr>
+    <tr><td>Dataset size</td><td>5000 / 1000 / 3671 / 525</td><td><span class="tag same">5000</span></td><td><span class="tag same">1000</span></td><td><span class="tag same">3671</span></td><td><span class="tag same">525</span></td><td><span class="tag same">525</span></td></tr>
+    <tr><td>Samples per example</td><td>1 (main), 8 (Prof)</td><td><span class="tag same">1</span></td><td><span class="tag same">1</span></td><td><span class="tag same">1</span></td><td><span class="tag same">8</span> <span class="tag diff">1 for deepseek</span></td><td><span class="tag diff">1</span></td></tr>
+    <tr><td>Reasoning effort</td><td>highest available</td><td><span class="tag diff">unset</span></td><td><span class="tag diff">unset</span></td><td><span class="tag diff">unset</span></td><td><span class="tag diff">unset</span></td><td><span class="tag diff">unset</span></td></tr>
+    <tr><td>Headline metric</td><td>length-adjusted</td><td><span class="tag diff">unadjusted</span></td><td><span class="tag diff">unadjusted</span></td><td><span class="tag diff">unadjusted</span></td><td><span class="tag same">length-adjusted</span></td><td><span class="tag same">length-adjusted</span></td></tr>
+    <tr><td>Task shape</td><td>one benchmark</td><td><span class="tag same">one task</span></td><td><span class="tag same">one task</span></td><td><span class="tag same">one task</span></td><td><span class="tag diff">5 subset tasks</span></td><td><span class="tag same">one task</span></td></tr>
+    <tr><td>Bootstrap resamples</td><td>1000</td><td><span class="tag same">1000</span></td><td><span class="tag same">1000</span></td><td><span class="tag same">1000</span></td><td><span class="tag same">1000</span></td><td><span class="tag same">1000</span></td></tr>
+  </tbody>
+</table>
+<p class="note">
+<span class="ok">v2:</span> the re-run column is DeepSeek only. It closes the task-shape and explicit-grader
+gaps and leaves the two that matter most for score comparability &mdash; <b>samples per example</b> and
+<b>reasoning effort</b> &mdash; exactly where they were.
+</p>
+<p class="note">
+The length adjustment is worth calling out as correct-by-verification, not just correct-by-declaration: we
+confirmed the arithmetic per sample against
+<code>score &minus; penalty &times; ((len &minus; center) / 500)</code>, OpenAI's
+<code>calculate_length_adjusted_score</code>. On one Hard sample: raw 0.6102, completion 13,776 chars, giving
+0.6102 &minus; 0.0392 &times; 23.552 = &minus;0.3131, which is exactly the stored value.
+</p>
+</section>
+
+<section id="judge">
+<h2>Issue 1: Hard and Consensus were graded by gpt-4o-mini</h2>
+<p>
+This is the largest error and it was not a deliberate choice. The <code>inspect_evals</code> task wrappers for
+these two variants accept only the length-adjustment arguments and pass nothing else through, so
+<code>judge_model</code> silently keeps the package default of <code>openai/gpt-4o-mini</code>:
+</p>
+<details>
+<summary>The wrapper that drops the judge argument <span class="where">inspect_evals/healthbench/healthbench.py</span></summary>
+<div class="body">
+<pre>def healthbench(
+    ...
+    judge_model: str | Model = "openai/gpt-4o-mini",   <span class="muted"># package default</span>
+    ...
+)
+
+@task
+def healthbench_hard(
+    length_adjustment_center: float | None = None,
+    length_adjustment_penalty_per_500_chars: float | None = None,
+) -&gt; Task:
+    return healthbench(                      <span class="muted"># judge_model never forwarded</span>
+        subset="hard",
+        length_adjustment_center=length_adjustment_center,
+        length_adjustment_penalty_per_500_chars=length_adjustment_penalty_per_500_chars,
+    )</pre>
+<p>Our <code>hs-non-professional</code> run passed <code>judge_model: "openai/gpt-4.1"</code> explicitly and got
+it. The Hard and Consensus runs declared no judge at all in <code>task_args</code>, and inspecting the model
+events in the logs confirms every grading call went to <code>gpt-4o-mini-2024-07-18</code>.</p>
+</div>
+</details>
+<p>
+OpenAI never grades any variant with a mini-tier model. In <code>simple_evals.py</code> the same
+<code>healthbench_grading_sampler</code> is handed to <code>healthbench</code>, <code>healthbench_hard</code>
+and <code>healthbench_consensus</code> alike, so all three get GPT-4.1, or all three get GPT-5.4 low when the
+<code>--healthbench-use-gpt-5-4-low-grader</code> flag is set.
+</p>
+<p>
+The HealthBench paper measured what a weaker grader costs. Its meta-evaluation ranks candidate graders by
+agreement with physicians (Macro-F1): <b>GPT-4.1 0.709</b>, o4-mini 0.692, o3 0.681, GPT-4.1 mini 0.661,
+GPT-4.1 nano 0.580, and calls the smaller variants "substantially worse". <code>gpt-4o-mini</code> is not in
+that table but sits below GPT-4.1 mini in capability, so 0.661 is the optimistic bound.
+</p>
+<div class="callout">
+  <span class="star">&#9733;</span>
+  <span class="ct"><b>Consensus is the smoking gun.</b> OpenAI's Consensus scores are 94 to 96 for every model
+  from gpt-5 through gpt-5.6, because the subset is deliberately low-noise and near-saturated. Our whole
+  Consensus space sits between <b>71 and 82</b>. That is not our models underperforming, that is the grader
+  failing to recognise criteria that a stronger grader marks as met.</span>
+</div>
+</section>
+
+<section id="effort">
+<h2>Issue 2: reasoning effort is never set, and the models diverge because of it</h2>
+<p>
+The HealthBench Professional paper states that models are "evaluated at the highest reasoning effort option
+available via each model's API (e.g., xhigh for GPT-5.4)", and measures a <b>5.6 to 7.3 point</b> gain moving
+from low to xhigh. None of our runs pass a reasoning effort, so each provider's default applies, and the
+defaults are not comparable to each other:
+</p>
+<table>
+  <thead><tr><th>Model (HealthBench Hard, 1000 samples)</th><th class="num">frac. with reasoning</th><th class="num">mean reasoning tokens</th><th>generate config sent</th></tr></thead>
+  <tbody>
+    <tr><td>openai/gpt-5.5</td><td class="num">0.97</td><td class="num">291</td><td><code>{cache: true}</code></td></tr>
+    <tr><td>openrouter/deepseek-v4-pro</td><td class="num">0.86</td><td class="num">489</td><td><code>{cache: true}</code></td></tr>
+    <tr class="ours"><td>anthropic/claude-opus-4-7</td><td class="num">0.00</td><td class="num">0</td><td><code>{cache: true, max_tokens: 32000}</code></td></tr>
+    <tr><td>plamo-3.0-prime</td><td class="num">0.00</td><td class="num">0</td><td><code>{cache: true}</code></td></tr>
+    <tr><td>medgemma-4b / 27b</td><td class="num">0.00</td><td class="num">0</td><td><code>{cache: true}</code></td></tr>
+  </tbody>
+</table>
+<p>
+Claude ran with <b>extended thinking off on every sample of every log</b>, while gpt-5.5 and DeepSeek reasoned
+by default. Whatever the intended comparison was, this is not it: the reasoning models get their default
+budget and Claude gets none. Claude is the most disadvantaged model in the set for a reason that has nothing
+to do with Claude.
+</p>
+<p>
+This also explains the one gap the grader cannot account for. On Professional, where our grader is correct and
+the physician baseline matches to 0.2 points, gpt-5.5 still lands 4 points low. A default-effort versus
+highest-effort difference of that size is consistent with the 5.6 to 7.3 point low-to-xhigh delta OpenAI
+reports.
+</p>
+</section>
+
+<section id="headline">
+<h2>Issue 3: three spaces show the unadjusted score as the headline</h2>
+<p>
+OpenAI reports "length-adjusted score (unadjusted, mean response length in characters)", so the adjusted
+number is the primary one. In our <code>listing.json</code> the <code>primary_metric</code> for
+<code>hs-non-professional</code>, <code>hs-hard</code> and <code>hs-consensus</code> is the
+<b>unadjusted</b> <code>bootstrap_score</code>. Only <code>hs-prof-subsets</code> carries
+<code>use_length_adjusted: true</code>.
+</p>
+<p>
+This is not cosmetic. Length adjustment reorders the Hard leaderboard almost completely, because our models
+differ enormously in verbosity:
+</p>
+<table>
+  <thead><tr><th>Model</th><th class="num">mean chars</th><th class="num">raw</th><th class="num">length-adj</th><th class="num">shift</th><th class="num">rank raw</th><th class="num">rank adj</th></tr></thead>
+  <tbody>
+    <tr><td>anthropic/claude-opus-4-7</td><td class="num">1849</td><td class="num">26.61</td><td class="num">27.80</td><td class="num" style="color:#788C5D">+1.2</td><td class="num">2</td><td class="num">1</td></tr>
+    <tr><td>openai/gpt-5.5</td><td class="num">2158</td><td class="num">27.27</td><td class="num">26.03</td><td class="num">&minus;1.2</td><td class="num">1</td><td class="num">2</td></tr>
+    <tr><td>openrouter/deepseek-v4-pro</td><td class="num">3408</td><td class="num">24.84</td><td class="num">13.80</td><td class="num" style="color:#B0533A">&minus;11.0</td><td class="num">3</td><td class="num">3</td></tr>
+    <tr><td>plamo-3.0-prime</td><td class="num">2997</td><td class="num">17.43</td><td class="num">9.61</td><td class="num" style="color:#B0533A">&minus;7.8</td><td class="num">5</td><td class="num">4</td></tr>
+    <tr><td>vllm/medgemma-27b</td><td class="num">4073</td><td class="num">21.09</td><td class="num">4.83</td><td class="num" style="color:#B0533A">&minus;16.3</td><td class="num">4</td><td class="num">5</td></tr>
+    <tr><td>vllm/medgemma-4b</td><td class="num">3192</td><td class="num">10.60</td><td class="num">1.26</td><td class="num" style="color:#B0533A">&minus;9.3</td><td class="num">6</td><td class="num">6</td></tr>
+  </tbody>
+</table>
+<p class="note">
+Claude gains because it is the only model averaging under the 2000-character centre. medgemma-27b loses 16
+points. Anyone reading the space's default metric is reading a different ranking from the one OpenAI's
+methodology produces.
+</p>
+</section>
+
+<section id="subscores">
+<h2>Issue 4: the Professional per-use-case subscores are inflated</h2>
+<p>
+The custom <code>healthbench_professional.py</code> emits <code>use_case_*_score</code>,
+<code>type_*_score</code> and <code>difficulty_*_score</code> metrics alongside the headline. <b>These do not
+use the same aggregation as the headline and should not be quoted.</b>
+</p>
+<div class="callout">
+  <span class="star">&#9733;</span>
+  <span class="ct"><b>v2 correction: the mechanism is now pinned exactly, and it is worse than v1 said.</b>
+  v1 described these subscores as "consistent with per-sample clipping" of the length-adjusted score, with a
+  residual attributed to bootstrap noise. The DeepSeek re-run separates the two candidate formulas cleanly,
+  because DeepSeek is verbose enough that adjusted and unadjusted scores diverge sharply. Every one of the ten
+  subset metrics in that log reproduces to six decimal places as
+  <code>mean(clip(<b>unadjusted</b> score, 0, 1))</code>. So the subscores are not a clipped version of the
+  headline metric &mdash; <b>they ignore the length adjustment entirely</b>, despite the run declaring
+  <code>use_length_adjusted: true</code>, and then clip per sample on top. Both errors push the same way.</span>
+</div>
+<table>
+  <thead><tr><th>DeepSeek re-run, by slice</th><th class="num">as reported in the log</th><th class="num">mean(clip(raw))</th><th class="num">mean(clip(adj))</th><th class="num">correct (mean adj)</th><th class="num">inflation</th></tr></thead>
+  <tbody>
+    <tr><td>Care consult</td><td class="num" style="color:#B0533A">46.05</td><td class="num">46.05</td><td class="num muted">42.70</td><td class="num">31.83</td><td class="num">+14.2</td></tr>
+    <tr><td>Writing and documentation</td><td class="num" style="color:#B0533A">35.43</td><td class="num">35.43</td><td class="num muted">32.42</td><td class="num">6.70</td><td class="num">+28.7</td></tr>
+    <tr><td>Medical research</td><td class="num" style="color:#B0533A">66.60</td><td class="num">66.60</td><td class="num muted">57.00</td><td class="num">53.10</td><td class="num">+13.5</td></tr>
+    <tr class="new"><td>Red teaming</td><td class="num" style="color:#B0533A">24.94</td><td class="num">24.94</td><td class="num muted">23.98</td><td class="num">&minus;5.19</td><td class="num">+30.1</td></tr>
+    <tr><td>Difficult</td><td class="num" style="color:#B0533A">28.67</td><td class="num">28.67</td><td class="num muted">27.38</td><td class="num">3.51</td><td class="num">+25.2</td></tr>
+    <tr><td>Typical</td><td class="num" style="color:#B0533A">70.23</td><td class="num">70.23</td><td class="num muted">61.30</td><td class="num">59.86</td><td class="num">+10.4</td></tr>
+  </tbody>
+</table>
+<p class="note">
+The <code>mean(clip(raw))</code> column matches the reported column exactly, to six decimals, on all ten subset
+metrics the log emits &mdash; there is no bootstrap noise in these numbers at all. Red teaming is the clearest
+illustration of the damage: the log presents it as <b>24.94</b> when the correctly aggregated value is
+<b>&minus;5.19</b>, a 30-point swing that flips the slice from mediocre to negative.
+</p>
+<p>
+The same error was present in the v1 spaces; it was simply harder to characterise there, because the
+physician-baseline responses are short enough that the adjusted and unadjusted scores nearly coincide.
+</p>
+<p>
+The physician baseline makes the error measurable, because OpenAI publishes the ground truth for exactly these
+three cells:
+</p>
+<table>
+  <thead><tr><th>Physician baseline by use case</th><th class="num">OpenAI</th><th class="num">recomputed correctly</th><th class="num">as reported in our log</th><th class="num">inflation</th></tr></thead>
+  <tbody>
+    <tr><td>Care consult</td><td class="num">42.7</td><td class="num">42.54</td><td class="num" style="color:#B0533A">48.4</td><td class="num">+5.9</td></tr>
+    <tr><td>Writing and documentation</td><td class="num">32.1</td><td class="num">32.91</td><td class="num" style="color:#B0533A">44.9</td><td class="num">+12.0</td></tr>
+    <tr><td>Medical research</td><td class="num">56.3</td><td class="num">56.60</td><td class="num" style="color:#B0533A">59.0</td><td class="num">+2.4</td></tr>
+  </tbody>
+</table>
+<details>
+<summary>The arithmetic that identifies the cause <span class="where">525 examples &times; 8 epochs = 4200</span></summary>
+<div class="body">
+<pre>overall mean raw                    = 44.29   <span class="muted"># matches reported 44.29</span>
+overall mean adj                    = 43.87   <span class="muted"># matches reported 43.87, and OpenAI's 43.7</span>
+overall mean adj, clipped per-sample= 49.71
+frac. of samples with adj &lt; 0       = 0.164
+
+by use case          n      mean_adj   clipped_per_sample
+  consult          1888      42.54          48.09
+  research         1176      56.60          57.47
+  writing          1136      32.91          44.39</pre>
+<p>16.4% of physician responses score below zero once the length penalty applies, mostly short writing-task
+answers that trip negative rubric criteria. Clipping those to zero before averaging is what lifts writing from
+32.9 to roughly 44. <span class="ok">v2:</span> on the physician baseline the clipped-raw and clipped-adjusted
+columns are within 0.002 of each other, which is why v1 could not tell them apart and read the ~0.3 residual as
+bootstrap noise. The DeepSeek re-run resolves it: the formula is clipped <b>raw</b>.</p>
+</div>
+</details>
+<p>
+The headline <code>bootstrap_score</code> is unaffected and remains correct. Only the subset breakdowns are
+wrong, which matters because the by-use-case split is the most quoted view of Professional results.
+</p>
+</section>
+
+<section id="floored">
+<h2>Issue 5: five Professional scores are floored at zero</h2>
+<p>
+OpenAI clips the aggregate mean to [0, 1], and our implementation follows suit, so this is faithful behaviour
+rather than a bug. But it means five reported numbers are all displayed as <code>0.00</code> while their true
+values differ by 7 points, which hides real ranking information:
+</p>
+<table>
+  <thead><tr><th>Model</th><th>Subset</th><th class="num">reported</th><th class="num">true unclipped mean</th></tr></thead>
+  <tbody>
+    <tr><td>plamo-3.0-prime</td><td>red teaming</td><td class="num">0.00</td><td class="num">&minus;11.82</td></tr>
+    <tr><td>openrouter/deepseek-v4-pro</td><td>red teaming (ep 1)</td><td class="num">0.00</td><td class="num">&minus;8.29</td></tr>
+    <tr><td>openrouter/deepseek-v4-pro</td><td>red teaming (ep 8)</td><td class="num">0.00</td><td class="num">&minus;6.87</td></tr>
+    <tr><td>vllm/medgemma-27b</td><td>red teaming</td><td class="num">0.00</td><td class="num">&minus;6.80</td></tr>
+    <tr><td>plamo-3.0-prime</td><td>writing</td><td class="num">0.00</td><td class="num">&minus;4.31</td></tr>
+  </tbody>
+</table>
+<p class="note">
+Worth reporting the unclipped value alongside the clipped one for these, or at least noting that a zero means
+"at or below zero" rather than "scored nothing".
+</p>
+</section>
+
+<section id="hygiene">
+<h2>Run hygiene</h2>
+<details>
+<summary>DeepSeek's Professional runs are under-sampled, and its consult run is the only one <span class="where">hs-prof-subsets &middot; superseded in v2</span></summary>
+<div class="body">
+<p>Every model ran <code>epochs=8</code>, matching the paper's "8 samples per example". The 2026-07-25 DeepSeek
+batch ran <code>epochs=1</code>. For <code>writing</code>, <code>research</code>, <code>red_teaming</code> and
+<code>physician_baseline</code> both variants exist in <code>listing.json</code>, so the viewer shows duplicate
+conflicting entries. For <code>consult</code> the epochs=1 run is the <b>only</b> DeepSeek run, so it sits in
+the same chart as everyone else at one eighth the sampling.</p>
+<p>The good news is the two variants agree closely where both exist (research 51.85 vs 52.91, writing 5.16 vs
+4.95, red teaming floored in both), so the practical distortion is small. It is still worth deleting the
+epochs=1 logs and re-running consult at 8.</p>
+<p><span class="ok">v2 update, and one thing v1 got wrong.</span> The 2026-07-25 epochs=1 batch was worse than
+under-sampled: <b>it made no model calls at all</b>. Every one of those five logs reports an empty
+<code>model_usage</code>, meaning both the candidate generations and the grader responses were served from
+Inspect's response cache. They are cache replays of an earlier run, not independent runs, which is why they
+completed in seconds. v1 read the close agreement between the epochs=1 and epochs=8 variants as reassuring;
+it was tautological. The <a href="#v2">2026-08-06 re-run</a> is a genuine run &mdash; 967k output tokens
+billed to DeepSeek, 476 cached input tokens &mdash; and it moves consult from 25.59 to 31.83. The
+duplicate-entry and no-real-consult-run problems are resolved. <b>Epochs is still 1</b>, so the
+under-sampling itself is not.</p>
+</div>
+</details>
+<details>
+<summary>medgemma-4b is missing four of five Professional subsets <span class="where">hs-prof-subsets</span></summary>
+<div class="body">
+<p>Only <code>consult</code> exists (score 8.19). Writing, research, red teaming and physician baseline were
+never run, so medgemma-4b has no Professional overall and cannot appear in a like-for-like comparison.</p>
+</div>
+</details>
+<details>
+<summary>Response caching is on and demonstrably active <span class="where">plan: generate(cache=true)</span></summary>
+<div class="body">
+<p>Four of the five physician-baseline runs report byte-identical scores of 0.44291, which only happens if the
+grader responses came from cache. Inspect's cache key includes the epoch by default
+(<code>CachePolicy.per_epoch = True</code>), so the 8 epochs are not collapsing into one, which was the real
+risk. The remaining caveat is that a re-run inside the one-week TTL is not an independent sample, so a repeat
+run cannot be used as a variance estimate.</p>
+</div>
+</details>
+<details>
+<summary>Two different inspect_evals versions inside the same space <span class="where">0.14.3 vs 0.16.0</span></summary>
+<div class="body">
+<p>The frontier-model runs (July 15 to 16) used <code>inspect_evals 0.14.3</code> with
+<code>inspect_ai 0.3.246</code>; the medgemma runs (July 24) used <code>0.16.0</code> with
+<code>0.3.249</code>. Same space, same chart, different scorer code. Worth confirming the healthbench scorer
+did not change across those releases before comparing medgemma against the frontier models.</p>
+</div>
+</details>
+<details>
+<summary>Outlier generations and empty completions <span class="where">hs-hard</span></summary>
+<div class="body">
+<p>All runs completed with zero sample errors. Two small things: medgemma-4b produced one 110,722-character
+response on Hard, which carries a length penalty of &minus;8.5 on its own and moves the 1000-sample mean by
+about 0.9 points; and claude-opus-4-7 returned 7 empty completions on Hard, each scored zero. Neither is
+fatal, both are worth a spot check.</p>
+</div>
+</details>
+</section>
+
+<section id="fixes">
+<h2>What to change</h2>
+<div class="ladder">
+  <div class="chip miss">judge: gpt-4.1 or gpt-5.4-low on hard</div>
+  <div class="chip miss">judge: gpt-4.1 or gpt-5.4-low on consensus</div>
+  <div class="chip miss">reasoning effort: highest, all models</div>
+  <div class="chip miss">claude: enable extended thinking</div>
+  <div class="chip miss">headline: length-adjusted, 3 spaces</div>
+  <div class="chip miss">prof subscores: use adj score, no clip</div>
+  <div class="chip part">deepseek consult: real run &#10003;, epochs 8 &#10007;</div>
+  <div class="chip miss">medgemma-4b: run 4 missing subsets</div>
+  <div class="chip done">deepseek: duplicate listing entries</div>
+  <div class="chip done">prof: one task, not 5 subset tasks</div>
+  <div class="chip hit">length adjustment constants</div>
+  <div class="chip hit">dataset sizes and splits</div>
+  <div class="chip hit">grader system message and temp</div>
+  <div class="chip hit">bootstrap and mean clipping</div>
+  <div class="chip hit">epochs 8 on Professional</div>
+  <div class="chip hit">prompting and harness (length match)</div>
+</div>
+<div class="ladder-legend">clay = needs changing &nbsp;&middot;&nbsp; olive = verified correct, leave alone &nbsp;&middot;&nbsp;
+struck through = fixed by the v2 re-run &nbsp;&middot;&nbsp; grey = partly fixed</div>
+
+<p>Concretely, for the two broken variants, stop using the wrapper tasks and call the parent task instead:</p>
+<details>
+<summary>Getting a real judge into Hard and Consensus <span class="where">the wrapper cannot do it</span></summary>
+<div class="body">
+<pre><span class="muted"># broken: judge_model silently defaults to gpt-4o-mini</span>
+inspect eval inspect_evals/healthbench_hard \
+  -T length_adjustment_center=2000 \
+  -T length_adjustment_penalty_per_500_chars=0.0392
+
+<span class="muted"># works: call healthbench() directly with subset=</span>
+inspect eval inspect_evals/healthbench \
+  -T subset=hard \
+  -T judge_model=<span class="hl">openai/gpt-4.1</span> \
+  -T length_adjustment_center=2000 \
+  -T length_adjustment_penalty_per_500_chars=0.0392</pre>
+<p>Worth considering GPT-5.4 at low reasoning for all four variants instead. The Professional paper notes the
+external implementation now has "an option to use settings matching our internal implementation (e.g., GPT-5.4
+at low reasoning effort as a grader)" and that OpenAI intends to keep reporting from the internal one. Using it
+everywhere would both fix the deviation and make our four spaces internally comparable for the first time.</p>
+</div>
+</details>
+<p class="note">
+Re-running with the correct grader and highest reasoning effort should move Hard up by roughly 5 points and
+Consensus up by roughly 13 toward the published band. If it does not, that is the signal something else is
+wrong. If it does, the config is settled.
+</p>
+</section>
+
+<section id="full">
+<h2>Full results</h2>
+<h3>Main variants, all six models</h3>
+<table>
+  <thead><tr><th>Model</th><th class="num">full raw</th><th class="num">full adj</th><th class="num">hard raw</th><th class="num">hard adj</th><th class="num">cons. raw</th><th class="num">cons. adj</th></tr></thead>
+  <tbody>
+    <tr class="ours"><td>openai/gpt-5.5</td><td class="num">56.87</td><td class="num">55.83</td><td class="num">27.27</td><td class="num">26.03</td><td class="num">82.08</td><td class="num">82.02</td></tr>
+    <tr class="ours"><td>anthropic/claude-opus-4-7</td><td class="num">53.44</td><td class="num">54.31</td><td class="num">26.61</td><td class="num">27.80</td><td class="num">80.17</td><td class="num">80.23</td></tr>
+    <tr><td>openrouter/deepseek-v4-pro</td><td class="num">51.37</td><td class="num">41.73</td><td class="num">24.84</td><td class="num">13.80</td><td class="num">79.10</td><td class="num">78.46</td></tr>
+    <tr><td>vllm/medgemma-27b-text-it</td><td class="num">47.20</td><td class="num">33.21</td><td class="num">21.09</td><td class="num">4.83</td><td class="num">77.58</td><td class="num">76.61</td></tr>
+    <tr><td>openai-api/plamo-3.0-prime</td><td class="num">39.42</td><td class="num">32.43</td><td class="num">17.43</td><td class="num">9.61</td><td class="num">75.25</td><td class="num">74.78</td></tr>
+    <tr><td>vllm/medgemma-4b-it</td><td class="num">26.97</td><td class="num">18.32</td><td class="num">10.60</td><td class="num">1.26</td><td class="num">71.37</td><td class="num">70.77</td></tr>
+    <tr class="human"><td>OpenAI published, gpt-5.5</td><td class="num">58.4</td><td class="num">56.5</td><td class="num">33.8</td><td class="num">31.5</td><td class="num">95.7</td><td class="num">95.6</td></tr>
+  </tbody>
+</table>
+<p class="note">
+Judge: gpt-4.1 for full, gpt-4o-mini for hard and consensus. Epochs 1 throughout. All runs completed with zero
+errors. Only the full column is grader-comparable to the published row.
+</p>
+
+<h3>Professional subsets, length-adjusted</h3>
+<table>
+  <thead><tr><th>Model</th><th class="num">consult</th><th class="num">writing</th><th class="num">research</th><th class="num">red team</th><th class="num">weighted overall</th></tr></thead>
+  <tbody>
+    <tr class="ours"><td>anthropic/claude-opus-4-7</td><td class="num">47.04</td><td class="num">36.15</td><td class="num">61.07</td><td class="num">26.74</td><td class="num">48.02</td></tr>
+    <tr class="ours"><td>openai/gpt-5.5</td><td class="num">48.64</td><td class="num">35.96</td><td class="num">57.92</td><td class="num">28.19</td><td class="num">47.81</td></tr>
+    <tr class="new"><td>openrouter/deepseek-v4-pro <span class="vtag">v2 re-run</span></td><td class="num">31.83</td><td class="num">6.70</td><td class="num">53.10</td><td class="num">&minus;5.19</td><td class="num">30.99</td></tr>
+    <tr><td class="muted">openrouter/deepseek-v4-pro <span class="note">(v1, superseded)</span></td><td class="num muted">25.59</td><td class="num muted">4.95</td><td class="num muted">52.91</td><td class="num muted">&minus;6.87</td><td class="num muted">27.66</td></tr>
+    <tr><td>vllm/medgemma-27b-text-it</td><td class="num">17.78</td><td class="num">9.13</td><td class="num">34.40</td><td class="num">&minus;6.80</td><td class="num">20.09</td></tr>
+    <tr><td>openai-api/plamo-3.0-prime</td><td class="num">15.36</td><td class="num">&minus;4.31</td><td class="num">28.58</td><td class="num">&minus;11.82</td><td class="num">13.74</td></tr>
+    <tr><td>vllm/medgemma-4b-it</td><td class="num">8.19</td><td class="num muted">not run</td><td class="num muted">not run</td><td class="num muted">not run</td><td class="num muted">n/a</td></tr>
+    <tr class="human"><td>Physician baseline (ours)</td><td class="num">42.54</td><td class="num">32.91</td><td class="num">56.60</td><td class="num muted">n/a</td><td class="num">43.87</td></tr>
+    <tr class="human"><td>Physician baseline (OpenAI)</td><td class="num">42.7</td><td class="num">32.1</td><td class="num">56.3</td><td class="num muted">n/a</td><td class="num">43.7</td></tr>
+    <tr class="human"><td>OpenAI published, gpt-5.5</td><td class="num muted">n/a</td><td class="num muted">n/a</td><td class="num muted">n/a</td><td class="num muted">n/a</td><td class="num">51.8</td></tr>
+  </tbody>
+</table>
+<p class="note">
+Red teaming values are shown unclipped so the ordering is visible; the logs report these as 0.00. Weighted
+overall uses consult 236, writing 142, research 147. Only gpt-5.5 and claude-opus-4-7 beat the physician
+baseline overall, and both do so narrowly.
+</p>
+<p class="note">
+<span class="ok">v2:</span> the DeepSeek row now comes from the 2026-08-06 single-task re-run, whose 30.99 is a
+native overall rather than a weighted stitch. Its subset figures are recomputed from per-sample data, not read
+off the log's own subset metrics, which are inflated &mdash; see <a href="#subscores">issue 4</a>. The
+superseded v1 row is kept for traceability: its consult and writing figures came from the 2026-07-25 cache
+replay, and its research and red-teaming figures from the 2026-07-24 epochs=8 runs. Every other model's row is
+still epochs=8 and unchanged, so DeepSeek is the only single-sample row in this table.
+</p>
+</section>
+
+<section id="method">
+<h2>How this was checked</h2>
+<p>
+The four HuggingFace spaces were cloned and their <code>.eval</code> logs pulled as real LFS blobs (the plain
+clone yields 133-byte pointers). Headers gave the declared config; per-sample model events gave the config
+actually sent, including which model served every grading call; per-sample scores were re-aggregated
+independently to confirm the reported metrics and to recompute the subset breakdowns.
+</p>
+<table>
+  <thead><tr><th>Space</th><th class="num">logs</th><th class="num">size</th><th>task</th><th class="num">samples/run</th></tr></thead>
+  <tbody>
+    <tr><td>hs-non-professional</td><td class="num">6</td><td class="num">573 MB</td><td><code>inspect_evals/healthbench</code></td><td class="num">5000</td></tr>
+    <tr><td>hs-hard</td><td class="num">6</td><td class="num">113 MB</td><td><code>inspect_evals/healthbench_hard</code></td><td class="num">1000</td></tr>
+    <tr><td>hs-consensus</td><td class="num">6</td><td class="num">256 MB</td><td><code>inspect_evals/healthbench_consensus</code></td><td class="num">3671</td></tr>
+    <tr><td>hs-prof-subsets</td><td class="num">30</td><td class="num">714 MB</td><td><code>healthbench_professional_*</code></td><td class="num">525</td></tr>
+    <tr class="new"><td>healthbench-professional-deepseek-v4-pro <span class="vtag">v2</span></td><td class="num">1</td><td class="num">10 MB</td><td><code>healthbench_professional</code></td><td class="num">525</td></tr>
+  </tbody>
+</table>
+<p class="note">
+Local copies at <code>/Users/kazuki/Sandbox/hb-spaces/</code>. The <code>hs-original</code> space was excluded,
+its link is broken. The v2 space's <code>.eval</code> was fetched over the HuggingFace
+<code>resolve/main</code> endpoint rather than <code>git lfs pull</code>, since git-lfs is not installed on
+this machine; the file is a valid zstd-compressed Inspect log and reads cleanly. Reference sources: HealthBench (<a href="https://arxiv.org/abs/2505.08775">arXiv:2505.08775</a>),
+HealthBench Professional (<a href="https://arxiv.org/abs/2604.27470">arXiv:2604.27470</a>), the
+<a href="https://deploymentsafety.openai.com/gpt-5-6-preview/healthbench">GPT-5.6 Preview system card</a> Table 6,
+and <a href="https://github.com/openai/simple-evals">openai/simple-evals</a>.
+</p>
+</section>
+
+</main>
+</div>
+</body>
+</html>

data/INDEX.md

@@ -0,0 +1,76 @@
+# Ajay's HealthBench HF Spaces — local mirror
+
+Mirror of every HealthBench-related space under `huggingface.co/ajay-citadel`.
+Inventory taken 2026-08-07. Per-run detail: `MANIFEST.csv` (68 runs), raw headers: `headers_raw.json`.
+
+Space URL pattern: `https://huggingface.co/spaces/ajay-citadel/<dir>`
+Log URL pattern: `.../resolve/main/logs/<file>.eval`
+
+| dir | space | runs | date | what it is |
+|---|---|---|---|---|
+| `healthbench-v1` (symlink → `inspect_ai/sandbox/healthbench-v1`) | healthbench-v1 | 7 (2 ok, 4 err/cancelled) | 07-09 | first attempt, `gpt-4o-mini` judge, no length adjustment |
+| `hs-non-professional` | hs-non-professional | 6 | 07-15..24 | HealthBench full (n=5000), `gpt-4.1` judge |
+| `hs-consensus` | hs-consensus | 6 | 07-16..24 | HealthBench consensus (n=3671) |
+| `hs-hard` | hs-hard | 6 | 07-16..24 | HealthBench hard (n=1000) |
+| `hs-original` | hs-original | 6 | 07-24..25 | HealthBench **Professional** full (n=525, epochs=8) |
+| `hs-prof-subsets` | hs-prof-subsets | 26 | 07-24..25 | Professional split by use-case + physician baseline |
+| `medgemma-healthbench-hard` | medgemma-healthbench-hard | 2 | 08-05 | MedGemma hard re-run, `gpt-4.1` judge |
+| `healthbench-hard-medgemma-v2` | healthbench-hard-medgemma-v2 | 2 | 08-05 | same as above via `vllm/` provider prefix (near-duplicate) |
+| `healthbench-consensus-medgemma` | healthbench-consensus-medgemma | 2 | 08-05 | MedGemma consensus re-run, `gpt-4.1` judge |
+| `healthbench-professional-deepseek-v4-pro` | healthbench-professional-deepseek-v4-pro | 1 | 08-06 | DeepSeek Professional, first genuinely fresh generation |
+
+Empty shells on HF (README/index.html only, no logs — nothing to mirror):
+`healthbench-profes`, `healthbench-prof-part1`.
+
+Non-HealthBench spaces by the same author, **not** mirrored here:
+`inspectevals`, `tevals`, `tmevals`, `medgemma_runs`, `igakuqaruns`, `igakuqaruns-v2`,
+`igakuqa-medgemma`, `situational-awareness-v1`, `anzen1`, `anzen-kimi-k3-safety-audit`,
+`pair-jbb-attacker-study-extended`.
+(MedQA/IgakuQA sets are already local under `inspect_ai/sandbox/MedQA/` and `.../IgakuQA/`.)
+
+## Traps
+
+1. **Judge model is not constant across the set** — three judges are in play:
+   - `openai/gpt-4o-mini` (scorer default): `healthbench-v1` (07-09), all of `hs-hard`, all of `hs-consensus`
+   - `openai/gpt-4.1`: all of `hs-non-professional`, and the 08-05 MedGemma re-runs
+   - `openai/gpt-5.4`, `reasoning_effort=low`: every Professional run
+
+   Raw `healthbench_score` is judge-dependent and is *not* touched by the length-adjustment
+   args (`scorer.py:524` — it's `achieved/possible` points). So a raw-score delta between two
+   runs with different judges is a judge artifact. Measured, same model + same task code:
+
+   | | gpt-4o-mini | gpt-4.1 | Δ |
+   |---|---|---|---|
+   | healthbench full, GPT-5.5 | 0.4873 | 0.5687 | **+8.1** |
+   | healthbench full, Opus-4.7 | 0.4762 | 0.5344 | **+5.8** |
+   | consensus, MedGemma-27B | 0.7758 | 0.9105 | **+13.5** |
+   | consensus, MedGemma-4B | 0.7137 | 0.7583 | +4.5 |
+   | hard, MedGemma-27B | 0.2109 | 0.1415 | **−6.9** |
+   | hard, MedGemma-4B | 0.1060 | −0.0347 | **−14.1** |
+
+   gpt-4.1 is *more* generous on consensus and *harsher* on hard. Never mix judges in one table.
+2. **`cache=true` throughout.** Only 23 of 68 runs actually generated candidate responses.
+   27 are full cache replays (zero candidate tokens), 7 more are "mostly cached"
+   (<200 candidate tokens/sample — e.g. `hs-hard` MedGemma-27B at 24 tok/sample, so its 0.2109
+   rests on cached completions from a run that was never published). Column `provenance`
+   in `MANIFEST.csv`; `stats.model_usage` is the underlying evidence.
+3. **Professional epochs are inconsistent**: 8 for GPT-5.5 / Opus-4-7 / PLaMo / MedGemma,
+   1 for DeepSeek. Do not put those in the same table without saying so.
+4. **DeepSeek Professional has two conflicting numbers.** `hs-original` 07-25 = 0.3430 raw
+   (but 10 candidate tok/sample → a replay), `healthbench-professional-deepseek-v4-pro` 08-06
+   = 0.3781 raw (fresh, 1.26M tokens). Same nominal config. Use the 08-06 one; treat the
+   ~3.5-point gap as the reproducibility floor for a cached-vs-fresh comparison.
+5. **Professional subset metrics inside the log are wrong** (discard length adjustment, clip
+   per sample). Recompute from per-sample scores. See
+   `../config-check-v2.html`.
+6. **Physician baseline was judged once and replayed.** The identical 0.4429/0.4387 across
+   four models in `hs-prof-subsets` is one computation, not four. (0.4387 matches OpenAI's
+   published 43.7 — this is the anchor that says the harness is correct.)
+7. **Professional use-case subsets don't partition the set.** consult 236 + writing 142 +
+   research 147 + red-teaming 191 = 716 > 525 total, so samples carry multiple use-case tags.
+   The four subset scores cannot be weight-averaged back to the full-set score.
+8. **The subset matrix is incomplete.** MedGemma-4B has only `consult`; DeepSeek has no
+   epochs=8 `consult`. Don't render `hs-prof-subsets` as a full grid.
+9. `medgemma-healthbench-hard` and `healthbench-hard-medgemma-v2` are the same experiment run
+   twice 2h apart, differing only in provider prefix (`openai-api/vllm/...` vs `vllm/...`).
+   Score spread between them (0.1332 vs 0.1415) is vLLM sampling noise, ~0.8pt.

data/MANIFEST.csv

@@ -0,0 +1,69 @@
+space,created,task,task_version,model,epochs,total_samples,status,judge_model,judge_reasoning_effort,len_adj_center,len_adj_penalty,raw_score,len_adj_score,candidate_tokens,cand_tok_per_sample,judge_tokens,provenance,inspect_ai,inspect_evals,task_file,file
+healthbench-v1,2026-07-09T08:07:45,inspect_evals/healthbench_hard,2,openai/gpt-5-nano,1,,error,,,,,,,0,,0,not-a-run,0.3.244,0.14.2,,2026-07-09T08-07-45-00-00_healthbench-hard_VLbsW5zXdzSbfgeNMziFZ2.eval
+healthbench-v1,2026-07-09T08:10:01,inspect_evals/healthbench_hard,2,openai/gpt-5.5,1,,error,,,,,,,0,,0,not-a-run,0.3.244,0.14.2,,2026-07-09T08-10-01-00-00_healthbench-hard_jofngniV8jDhpdvbXSg9S4.eval
+healthbench-v1,2026-07-09T08:11:34,inspect_evals/healthbench_hard,2,anthropic/claude-opus-4-7,1,,cancelled,,,,,,,46358,,0,not-a-run,0.3.244,0.14.2,,2026-07-09T08-11-34-00-00_healthbench-hard_CzHRcCD6CoJpUFoxxWadFn.eval
+healthbench-v1,2026-07-09T09:03:02,inspect_evals/healthbench,2,openai/gpt-5.5,1,,error,openai/gpt-4o-mini,,,,,,0,,0,not-a-run,0.3.245,0.14.3,,2026-07-09T09-03-02-00-00_healthbench_EVLdZCZJQYG5EuWtS2TLsK.eval
+healthbench-v1,2026-07-09T09:05:32,inspect_evals/healthbench,2,openai/gpt-5.5,1,,error,openai/gpt-4o-mini,,,,,,0,,0,not-a-run,0.3.245,0.14.3,,2026-07-09T09-05-32-00-00_healthbench_Aucfe2K5BgdpYrkukVNJcf.eval
+healthbench-v1,2026-07-09T09:09:45,inspect_evals/healthbench,2,openai/gpt-5.5,1,5000,success,openai/gpt-4o-mini,,,,0.4873,,4699836,940.0,84688797,fresh,0.3.245,0.14.3,,2026-07-09T09-09-45-00-00_healthbench_gGjNAEJ4UezLEQvJPVi7Zi.eval
+healthbench-v1,2026-07-09T10:33:03,inspect_evals/healthbench,2,anthropic/claude-opus-4-7,1,5000,success,openai/gpt-4o-mini,,,,0.4762,,4923085,984.6,80274748,fresh,0.3.245,0.14.3,,2026-07-09T10-33-03-00-00_healthbench_QeHrUjumcLfkEWL5T7HaUa.eval
+hs-non-professional,2026-07-15T06:51:58,inspect_evals/healthbench,2,openai/gpt-5.5,1,5000,success,openai/gpt-4.1,,2000,0.0299,0.5687,0.5583,0,0.0,0,FULL CACHE REPLAY,0.3.246,0.14.3,,2026-07-15T06-51-58-00-00_healthbench_WJDhL9LgZJKMGgYizSLYpL.eval
+hs-non-professional,2026-07-15T07:01:24,inspect_evals/healthbench,2,anthropic/claude-opus-4-7,1,5000,success,openai/gpt-4.1,,2000,0.0299,0.5344,0.5431,0,0.0,0,FULL CACHE REPLAY,0.3.246,0.14.3,,2026-07-15T07-01-24-00-00_healthbench_Hah9CRDdSrTz7K5EFgRWF8.eval
+hs-non-professional,2026-07-16T04:29:43,inspect_evals/healthbench,2,openrouter/deepseek/deepseek-v4-pro,1,5000,success,openai/gpt-4.1,,2000,0.0299,0.5137,0.4173,7415173,1483.0,105583667,fresh,0.3.246,0.14.3,,2026-07-16T04-29-43-00-00_healthbench_jBnP8VSE7rSvrMpzRStQFK.eval
+hs-non-professional,2026-07-16T05:23:21,inspect_evals/healthbench,2,openai-api/plamo/plamo-3.0-prime,1,5000,success,openai/gpt-4.1,,2000,0.0299,0.3942,0.3243,2449482,489.9,49156082,fresh,0.3.246,0.14.3,,2026-07-16T05-23-21-00-00_healthbench_iexipStaaxm8E5ni3Gg96Z.eval
+hs-hard,2026-07-16T07:28:45,inspect_evals/healthbench_hard,2,openai/gpt-5.5,1,1000,success,"openai/gpt-4o-mini (default, inferred)",,2000,0.0392,0.2727,0.2603,0,0.0,17512569,FULL CACHE REPLAY,0.3.246,0.14.3,,2026-07-16T07-28-45-00-00_healthbench-hard_dxTyk6Wmosx3kJAp4iULs9.eval
+hs-hard,2026-07-16T18:52:11,inspect_evals/healthbench_hard,2,anthropic/claude-opus-4-7,1,1000,success,"openai/gpt-4o-mini (default, inferred)",,2000,0.0392,0.2661,0.278,0,0.0,16665282,FULL CACHE REPLAY,0.3.246,0.14.3,,2026-07-16T18-52-11-00-00_healthbench-hard_Z9QDknMd4SukiAP7Q4FSPz.eval
+hs-hard,2026-07-16T19:07:31,inspect_evals/healthbench_hard,2,openrouter/deepseek/deepseek-v4-pro,1,1000,success,"openai/gpt-4o-mini (default, inferred)",,2000,0.0392,0.2484,0.138,0,0.0,21037790,FULL CACHE REPLAY,0.3.246,0.14.3,,2026-07-16T19-07-31-00-00_healthbench-hard_SABwaQjc3Xon5iDMpWFQV3.eval
+hs-hard,2026-07-16T19:23:40,inspect_evals/healthbench_hard,2,openai-api/plamo/plamo-3.0-prime,1,1000,success,"openai/gpt-4o-mini (default, inferred)",,2000,0.0392,0.1743,0.0961,0,0.0,20067380,FULL CACHE REPLAY,0.3.246,0.14.3,,2026-07-16T19-23-40-00-00_healthbench-hard_mEs7dJHDapQ4JVqHQtfTJc.eval
+hs-consensus,2026-07-16T20:09:58,inspect_evals/healthbench_consensus,2,openai-api/plamo/plamo-3.0-prime,1,3671,success,"openai/gpt-4o-mini (default, inferred)",,2000,0.002,0.7525,0.7478,0,0.0,12531224,FULL CACHE REPLAY,0.3.246,0.14.3,,2026-07-16T20-09-58-00-00_healthbench-consensus_87a5hkWgWY5rrHyoygzDVx.eval
+hs-consensus,2026-07-16T20:20:37,inspect_evals/healthbench_consensus,2,openai/gpt-5.5,1,3671,success,"openai/gpt-4o-mini (default, inferred)",,2000,0.002,0.8208,0.8202,0,0.0,10709027,FULL CACHE REPLAY,0.3.246,0.14.3,,2026-07-16T20-20-37-00-00_healthbench-consensus_gmfrK88PFGpuvRWqnk7uc5.eval
+hs-consensus,2026-07-16T20:44:07,inspect_evals/healthbench_consensus,2,openrouter/deepseek/deepseek-v4-pro,1,3671,success,"openai/gpt-4o-mini (default, inferred)",,2000,0.002,0.791,0.7846,0,0.0,13025085,FULL CACHE REPLAY,0.3.246,0.14.3,,2026-07-16T20-44-07-00-00_healthbench-consensus_EbuNesZCwbaRVAqRedBwjP.eval
+hs-consensus,2026-07-16T21:11:43,inspect_evals/healthbench_consensus,2,anthropic/claude-opus-4-7,1,3671,success,"openai/gpt-4o-mini (default, inferred)",,2000,0.002,0.8017,0.8023,0,0.0,10263280,FULL CACHE REPLAY,0.3.246,0.14.3,,2026-07-16T21-11-43-00-00_healthbench-consensus_k2nEohAktRtjKfZ5BNGJSG.eval
+hs-consensus,2026-07-24T09:07:55,inspect_evals/healthbench_consensus,2,vllm/google/medgemma-4b-it,1,3671,success,"openai/gpt-4o-mini (default, inferred)",,2000,0.002,0.7137,0.7077,3440824,937.3,14762331,fresh,0.3.249,0.16.0,,2026-07-24T09-07-55-00-00_healthbench-consensus_QX6zAfF9oDZDAX4p6k6yb9.eval
+hs-hard,2026-07-24T09:18:15,inspect_evals/healthbench_hard,2,vllm/google/medgemma-4b-it,1,1000,success,"openai/gpt-4o-mini (default, inferred)",,2000,0.0392,0.106,0.0126,402479,402.5,17583141,fresh,0.3.249,0.16.0,,2026-07-24T09-18-15-00-00_healthbench-hard_WkGg4UV64qpccys2fbViHY.eval
+hs-non-professional,2026-07-24T09:32:53,inspect_evals/healthbench,2,vllm/google/medgemma-4b-it,1,5000,success,openai/gpt-4.1,,2000,0.0299,0.2697,0.1832,877814,175.6,100167550,MOSTLY CACHED,0.3.249,0.16.0,,2026-07-24T09-32-53-00-00_healthbench_6Co8iMuXsKxGNrMeAkRbdp.eval
+hs-consensus,2026-07-24T11:39:48,inspect_evals/healthbench_consensus,2,vllm/google/medgemma-27b-text-it,1,3671,success,"openai/gpt-4o-mini (default, inferred)",,2000,0.002,0.7758,0.7661,4303064,1172.2,16607219,fresh,0.3.249,0.16.0,,2026-07-24T11-39-48-00-00_healthbench-consensus_7b4ZMKGYfFkgFNSJXJs53z.eval
+hs-non-professional,2026-07-24T12:08:49,inspect_evals/healthbench,2,vllm/google/medgemma-27b-text-it,1,5000,success,openai/gpt-4.1,,2000,0.0299,0.472,0.3321,1556281,311.3,113164489,fresh,0.3.249,0.16.0,,2026-07-24T12-08-49-00-00_healthbench_cuBWJL4bbU32vR5cxHsjEC.eval
+hs-hard,2026-07-24T14:36:42,inspect_evals/healthbench_hard,2,vllm/google/medgemma-27b-text-it,1,1000,success,"openai/gpt-4o-mini (default, inferred)",,2000,0.0392,0.2109,0.0483,24135,24.1,20040174,MOSTLY CACHED,0.3.249,0.16.0,,2026-07-24T14-36-42-00-00_healthbench-hard_BU7wDz79JMGuziKwJrRrpN.eval
+hs-original,2026-07-24T15:57:49,healthbench_professional,0,openai/gpt-5.5,8,4200,success,openai/gpt-5.4,low,2000.0,0.0147,0.5295,0.4781,8994839,2141.6,21393751,fresh,0.3.246,,healthbench_professional.py,2026-07-24T15-57-49-00-00_healthbench-professional_7PZFsBJYTZRCdXZkJPfki7.eval
+hs-prof-subsets,2026-07-24T16:36:59,healthbench_professional_consult,0,openai/gpt-5.5,8,1888,success,openai/gpt-5.4,low,,,0.51,0.4864,0,0.0,0,FULL CACHE REPLAY,0.3.246,,healthbench_professional.py,2026-07-24T16-36-59-00-00_healthbench-professional-consult_D8Z93ZWFtkR4CDbDBLZvBT.eval
+hs-prof-subsets,2026-07-24T16:37:37,healthbench_professional_writing,0,openai/gpt-5.5,8,1136,success,openai/gpt-5.4,low,,,0.4062,0.3596,0,0.0,0,FULL CACHE REPLAY,0.3.246,,healthbench_professional.py,2026-07-24T16-37-37-00-00_healthbench-professional-writing_joZQNJ9ZAZidpUaSpEqyvS.eval
+hs-prof-subsets,2026-07-24T16:37:58,healthbench_professional_research,0,openai/gpt-5.5,8,1176,success,openai/gpt-5.4,low,,,0.6799,0.5792,0,0.0,0,FULL CACHE REPLAY,0.3.246,,healthbench_professional.py,2026-07-24T16-37-58-00-00_healthbench-professional-research_3Vd4jhz6oiMQTsC2Mhw849.eval
+hs-prof-subsets,2026-07-24T16:38:22,healthbench_professional_red_teaming,0,openai/gpt-5.5,8,1528,success,openai/gpt-5.4,low,,,0.2992,0.2819,0,0.0,0,FULL CACHE REPLAY,0.3.246,,healthbench_professional.py,2026-07-24T16-38-22-00-00_healthbench-professional-red-teaming_Xund3mwQvSqTJfxno7icGe.eval
+hs-prof-subsets,2026-07-24T16:38:52,healthbench_professional_physician_baseline,0,openai/gpt-5.5,8,4200,success,openai/gpt-5.4,low,,,0.4429,0.4387,0,0.0,17204312,baseline(no model gen by design),0.3.246,,healthbench_professional.py,2026-07-24T16-38-52-00-00_healthbench-professional-physician-baseline_XADueRF5xwkSmDPWc4LtNf.eval
+hs-original,2026-07-24T16:55:39,healthbench_professional,0,anthropic/claude-opus-4-7,8,4200,success,openai/gpt-5.4,low,2000.0,0.0147,0.5083,0.4803,9075443,2160.8,19817644,fresh,0.3.246,,healthbench_professional.py,2026-07-24T16-55-39-00-00_healthbench-professional_boiooCVNG7ZkBVsnEpKLNT.eval
+hs-prof-subsets,2026-07-24T17:19:14,healthbench_professional_consult,0,anthropic/claude-opus-4-7,8,1888,success,openai/gpt-5.4,low,,,0.4906,0.4704,0,0.0,0,FULL CACHE REPLAY,0.3.246,,healthbench_professional.py,2026-07-24T17-19-14-00-00_healthbench-professional-consult_hZgDXu6oT2MuZrn4L7msXN.eval
+hs-prof-subsets,2026-07-24T17:19:42,healthbench_professional_writing,0,anthropic/claude-opus-4-7,8,1136,success,openai/gpt-5.4,low,,,0.3947,0.3615,0,0.0,0,FULL CACHE REPLAY,0.3.246,,healthbench_professional.py,2026-07-24T17-19-42-00-00_healthbench-professional-writing_haJJesaYyVNeNRLHnAi8HD.eval
+hs-prof-subsets,2026-07-24T17:19:59,healthbench_professional_research,0,anthropic/claude-opus-4-7,8,1176,success,openai/gpt-5.4,low,,,0.6464,0.6107,0,0.0,0,FULL CACHE REPLAY,0.3.246,,healthbench_professional.py,2026-07-24T17-19-59-00-00_healthbench-professional-research_7RUfdNFLnH7EpAC92NeL7e.eval
+hs-prof-subsets,2026-07-24T17:20:17,healthbench_professional_red_teaming,0,anthropic/claude-opus-4-7,8,1528,success,openai/gpt-5.4,low,,,0.2828,0.2674,0,0.0,0,FULL CACHE REPLAY,0.3.246,,healthbench_professional.py,2026-07-24T17-20-17-00-00_healthbench-professional-red-teaming_DLPPa6kbs9VeKS6cfPDhdv.eval
+hs-prof-subsets,2026-07-24T17:20:42,healthbench_professional_physician_baseline,0,anthropic/claude-opus-4-7,8,4200,success,openai/gpt-5.4,low,,,0.4429,0.4387,0,0.0,0,baseline(no model gen by design),0.3.246,,healthbench_professional.py,2026-07-24T17-20-42-00-00_healthbench-professional-physician-baseline_98adhjPPBGUtnnuJVmWXjn.eval
+hs-original,2026-07-24T17:30:37,healthbench_professional,0,openai-api/plamo/plamo-3.0-prime,8,4200,success,openai/gpt-5.4,low,2000.0,0.0147,0.2078,0.1374,8130263,1935.8,23382174,fresh,0.3.246,,healthbench_professional.py,2026-07-24T17-30-37-00-00_healthbench-professional_c9RnXrEaoi4T33bvwXEA5h.eval
+hs-prof-subsets,2026-07-24T19:33:04,healthbench_professional_consult,0,openai-api/plamo/plamo-3.0-prime,8,1888,success,openai/gpt-5.4,low,,,0.2184,0.1536,0,0.0,0,FULL CACHE REPLAY,0.3.246,,healthbench_professional.py,2026-07-24T19-33-04-00-00_healthbench-professional-consult_dgWnf9y2uVjcWoDcyTm3Mm.eval
+hs-prof-subsets,2026-07-24T19:33:35,healthbench_professional_writing,0,openai-api/plamo/plamo-3.0-prime,8,1136,success,openai/gpt-5.4,low,,,-0.0276,-0.0431,0,0.0,0,FULL CACHE REPLAY,0.3.246,,healthbench_professional.py,2026-07-24T19-33-35-00-00_healthbench-professional-writing_eAmnviGLop8QfjxGLMp9Dz.eval
+hs-prof-subsets,2026-07-24T19:33:52,healthbench_professional_research,0,openai-api/plamo/plamo-3.0-prime,8,1176,success,openai/gpt-5.4,low,,,0.4183,0.2858,0,0.0,0,FULL CACHE REPLAY,0.3.246,,healthbench_professional.py,2026-07-24T19-33-52-00-00_healthbench-professional-research_WF5VpCWo3K285bwUiVdCUm.eval
+hs-prof-subsets,2026-07-24T19:34:11,healthbench_professional_red_teaming,0,openai-api/plamo/plamo-3.0-prime,8,1528,success,openai/gpt-5.4,low,,,-0.0986,-0.1182,0,0.0,0,FULL CACHE REPLAY,0.3.246,,healthbench_professional.py,2026-07-24T19-34-11-00-00_healthbench-professional-red-teaming_4db2cTWDv6VCVoFSFqWpmW.eval
+hs-prof-subsets,2026-07-24T19:34:35,healthbench_professional_physician_baseline,0,openai-api/plamo/plamo-3.0-prime,8,4200,success,openai/gpt-5.4,low,,,0.4429,0.4387,0,0.0,0,baseline(no model gen by design),0.3.246,,healthbench_professional.py,2026-07-24T19-34-35-00-00_healthbench-professional-physician-baseline_S4NuFZgLXeK7RAgHpRKqTy.eval
+hs-prof-subsets,2026-07-24T19:59:01,healthbench_professional_writing,0,openrouter/deepseek/deepseek-v4-pro,8,1136,success,openai/gpt-5.4,low,,,0.0954,0.0495,1479207,1302.1,3316215,fresh,0.3.246,,healthbench_professional.py,2026-07-24T19-59-01-00-00_healthbench-professional-writing_933975rUF6cdWKJECk44bJ.eval
+hs-prof-subsets,2026-07-24T20:07:30,healthbench_professional_research,0,openrouter/deepseek/deepseek-v4-pro,8,1176,success,openai/gpt-5.4,low,,,0.6368,0.5291,2021029,1718.6,4316530,fresh,0.3.246,,healthbench_professional.py,2026-07-24T20-07-30-00-00_healthbench-professional-research_7gKGNrErBwsumd7oKKUcuN.eval
+hs-prof-subsets,2026-07-24T20:18:24,healthbench_professional_red_teaming,0,openrouter/deepseek/deepseek-v4-pro,8,1528,success,openai/gpt-5.4,low,,,-0.0372,-0.0687,1062156,695.1,2312783,fresh,0.3.246,,healthbench_professional.py,2026-07-24T20-18-24-00-00_healthbench-professional-red-teaming_HmwUcZnfLNqmUkyZkHDTdJ.eval
+hs-prof-subsets,2026-07-24T20:28:47,healthbench_professional_physician_baseline,0,openrouter/deepseek/deepseek-v4-pro,8,4200,success,openai/gpt-5.4,low,,,0.4429,0.4387,0,0.0,0,baseline(no model gen by design),0.3.246,,healthbench_professional.py,2026-07-24T20-28-47-00-00_healthbench-professional-physician-baseline_kjY8x2CXBcbHZUSHs55G7D.eval
+hs-original,2026-07-24T22:44:03,healthbench_professional,0,vllm/google/medgemma-27b-text-it,8,4200,success,openai/gpt-5.4,low,2000.0,0.0147,0.3125,0.2003,8383935,1996.2,25383913,fresh,0.3.249,,healthbench_professional.py,2026-07-24T22-44-03-00-00_healthbench-professional_6AYXiEDgGuzNtAcyWPDGrR.eval
+hs-prof-subsets,2026-07-24T23:41:11,healthbench_professional_consult,0,vllm/google/medgemma-27b-text-it,8,1888,success,openai/gpt-5.4,low,,,0.2851,0.1778,23572,12.5,61467,MOSTLY CACHED,0.3.249,,healthbench_professional.py,2026-07-24T23-41-11-00-00_healthbench-professional-consult_muhCA3DyeXvwNpDvn7gpN4.eval
+hs-prof-subsets,2026-07-24T23:41:59,healthbench_professional_writing,0,vllm/google/medgemma-27b-text-it,8,1136,success,openai/gpt-5.4,low,,,0.1891,0.0913,5260,4.6,19696,MOSTLY CACHED,0.3.249,,healthbench_professional.py,2026-07-24T23-41-59-00-00_healthbench-professional-writing_CMBT9aSmchzcQFRpwZdCqR.eval
+hs-prof-subsets,2026-07-24T23:42:33,healthbench_professional_research,0,vllm/google/medgemma-27b-text-it,8,1176,success,openai/gpt-5.4,low,,,0.4776,0.344,12406,10.5,45185,MOSTLY CACHED,0.3.249,,healthbench_professional.py,2026-07-24T23-42-33-00-00_healthbench-professional-research_RFx9qF5ZP369Kdi3CNQU6M.eval
+hs-prof-subsets,2026-07-24T23:43:33,healthbench_professional_red_teaming,0,vllm/google/medgemma-27b-text-it,8,1528,success,openai/gpt-5.4,low,,,0.0145,-0.068,0,0.0,0,FULL CACHE REPLAY,0.3.249,,healthbench_professional.py,2026-07-24T23-43-33-00-00_healthbench-professional-red-teaming_ewG7cuk2vXaZpcauRLz7qR.eval
+hs-prof-subsets,2026-07-24T23:44:07,healthbench_professional_physician_baseline,0,vllm/google/medgemma-27b-text-it,8,4200,success,openai/gpt-5.4,low,,,0.439,0.4348,0,0.0,17206029,baseline(no model gen by design),0.3.249,,healthbench_professional.py,2026-07-24T23-44-07-00-00_healthbench-professional-physician-baseline_cY3Z8sUNPRQTpSA3UHa77J.eval
+hs-original,2026-07-25T00:26:29,healthbench_professional,0,vllm/google/medgemma-4b-it,8,4200,success,openai/gpt-5.4,low,2000.0,0.0147,0.1651,0.0904,7090153,1688.1,22550200,fresh,0.3.249,,healthbench_professional.py,2026-07-25T00-26-29-00-00_healthbench-professional_GEViyNR2PM2B2oUV94DzoK.eval
+hs-prof-subsets,2026-07-25T00:59:37,healthbench_professional_consult,0,vllm/google/medgemma-4b-it,8,1888,success,openai/gpt-5.4,low,,,0.152,0.0819,21234,11.2,54391,MOSTLY CACHED,0.3.249,,healthbench_professional.py,2026-07-25T00-59-37-00-00_healthbench-professional-consult_ffSg7GfyxjArrdwu7fSrBa.eval
+hs-original,2026-07-25T18:24:55,healthbench_professional,0,openrouter/deepseek/deepseek-v4-pro,1,525,success,openai/gpt-5.4,low,2000.0,0.0147,0.343,0.2742,5338,10.2,8795,MOSTLY CACHED,0.3.246,,healthbench_professional.py,2026-07-25T18-24-55-00-00_healthbench-professional_HaKmMaK4Q6JCLuX2RLCPnc.eval
+hs-prof-subsets,2026-07-25T18:26:20,healthbench_professional_consult,0,openrouter/deepseek/deepseek-v4-pro,1,236,success,openai/gpt-5.4,low,,,0.3123,0.2559,0,0.0,0,FULL CACHE REPLAY,0.3.246,,healthbench_professional.py,2026-07-25T18-26-20-00-00_healthbench-professional-consult_3DSH4qRFzYfb27rH9Z9RqP.eval
+hs-prof-subsets,2026-07-25T18:26:25,healthbench_professional_writing,0,openrouter/deepseek/deepseek-v4-pro,1,142,success,openai/gpt-5.4,low,,,0.0973,0.0516,0,0.0,0,FULL CACHE REPLAY,0.3.246,,healthbench_professional.py,2026-07-25T18-26-25-00-00_healthbench-professional-writing_2LJ8vJpJXUX2TbtokQzbxp.eval
+hs-prof-subsets,2026-07-25T18:26:28,healthbench_professional_research,0,openrouter/deepseek/deepseek-v4-pro,1,147,success,openai/gpt-5.4,low,,,0.6298,0.5185,0,0.0,0,FULL CACHE REPLAY,0.3.246,,healthbench_professional.py,2026-07-25T18-26-28-00-00_healthbench-professional-research_9Pb5iwuDu8dmuFzs5FnmBw.eval
+hs-prof-subsets,2026-07-25T18:26:32,healthbench_professional_red_teaming,0,openrouter/deepseek/deepseek-v4-pro,1,191,success,openai/gpt-5.4,low,,,-0.0532,-0.0829,0,0.0,0,FULL CACHE REPLAY,0.3.246,,healthbench_professional.py,2026-07-25T18-26-32-00-00_healthbench-professional-red-teaming_e9hCTjjRXrNTnexAxrYM8x.eval
+hs-prof-subsets,2026-07-25T18:26:35,healthbench_professional_physician_baseline,0,openrouter/deepseek/deepseek-v4-pro,1,525,success,openai/gpt-5.4,low,,,0.4331,0.4289,0,0.0,0,baseline(no model gen by design),0.3.246,,healthbench_professional.py,2026-07-25T18-26-35-00-00_healthbench-professional-physician-baseline_Njv8PLSvCJpAvUbPPMQqkx.eval
+medgemma-healthbench-hard,2026-08-05T06:05:04,healthbench_hard,2,openai-api/vllm/google/medgemma-27b-text-it,1,1000,success,openai/gpt-4.1,,2000,0.0392,0.1332,-0.0442,1108556,1108.6,22501745,fresh,0.3.252,,inspect_evals/src/inspect_evals/healthbench/healthbench.py,2026-08-05T06-05-04-00-00_healthbench-hard_U2qKVPncnWPTbmLEfZfa4y.eval
+medgemma-healthbench-hard,2026-08-05T07:09:21,healthbench_hard,2,openai-api/vllm/google/medgemma-4b-it,1,1000,success,openai/gpt-4.1,,2000,0.0392,-0.0312,-0.1641,918963,919.0,20096686,fresh,0.3.252,,inspect_evals/src/inspect_evals/healthbench/healthbench.py,2026-08-05T07-09-21-00-00_healthbench-hard_4bZEuQss4Bbee8KixwTS4o.eval
+healthbench-hard-medgemma-v2,2026-08-05T07:54:55,healthbench_hard,2,vllm/google/medgemma-4b-it,1,1000,success,openai/gpt-4.1,,2000,0.0392,-0.0347,-0.1256,887308,887.3,19808558,fresh,0.3.252,,inspect_evals/src/inspect_evals/healthbench/healthbench.py,2026-08-05T07-54-55-00-00_healthbench-hard_MANxy36rpJhotpAvdwQiFk.eval
+healthbench-hard-medgemma-v2,2026-08-05T08:07:41,healthbench_hard,2,vllm/google/medgemma-27b-text-it,1,1000,success,openai/gpt-4.1,,2000,0.0392,0.1415,-0.023,1130260,1130.3,22994024,fresh,0.3.252,,inspect_evals/src/inspect_evals/healthbench/healthbench.py,2026-08-05T08-07-41-00-00_healthbench-hard_D9KnFzsoQRE57vVsejUQnW.eval
+healthbench-consensus-medgemma,2026-08-05T08:51:22,healthbench_consensus,2,vllm/google/medgemma-27b-text-it,1,3671,success,openai/gpt-4.1,,2000,0.002,0.9105,0.9007,3662191,997.6,14399926,fresh,0.3.252,,inspect_evals/src/inspect_evals/healthbench/healthbench.py,2026-08-05T08-51-22-00-00_healthbench-consensus_6a8D2h492in82NNQHCFgZw.eval
+healthbench-consensus-medgemma,2026-08-05T10:32:09,healthbench_consensus,2,vllm/google/medgemma-4b-it,1,3671,success,openai/gpt-4.1,,2000,0.002,0.7583,0.7522,2959598,806.2,12979748,fresh,0.3.252,,inspect_evals/src/inspect_evals/healthbench/healthbench.py,2026-08-05T10-32-09-00-00_healthbench-consensus_6WXSuGaZvDNS5Zze6Jrm3h.eval
+healthbench-professional-deepseek-v4-pro,2026-08-06T02:35:30,healthbench_professional,0,openrouter/deepseek/deepseek-v4-pro,1,525,success,openai/gpt-5.4,low,2000,0.0147,0.3781,0.3099,1258221,2396.6,2810449,fresh,0.3.252,,healthbench_professional.py,2026-08-06T02-35-30-00-00_healthbench-professional_bbeHCEMNSj6Q362BQHa7mS.eval

data/log_mapping.csv

@@ -0,0 +1,69 @@
+filename,bench,model,date,epochs,samples,judge,provenance,status,raw_score,len_adj_score,source_space,source_file
+hard__gpt-5-nano__2026-07-09-FAILED.eval,hard,gpt-5-nano,2026-07-09,1,,,not-a-run,error,,,healthbench-v1,2026-07-09T08-07-45-00-00_healthbench-hard_VLbsW5zXdzSbfgeNMziFZ2.eval
+hard__gpt-5.5__2026-07-09-FAILED.eval,hard,gpt-5.5,2026-07-09,1,,,not-a-run,error,,,healthbench-v1,2026-07-09T08-10-01-00-00_healthbench-hard_jofngniV8jDhpdvbXSg9S4.eval
+hard__opus-4.7__2026-07-09-FAILED.eval,hard,opus-4.7,2026-07-09,1,,,not-a-run,cancelled,,,healthbench-v1,2026-07-09T08-11-34-00-00_healthbench-hard_CzHRcCD6CoJpUFoxxWadFn.eval
+full__gpt-5.5__2026-07-09-FAILED.eval,full,gpt-5.5,2026-07-09,1,,openai/gpt-4o-mini,not-a-run,error,,,healthbench-v1,2026-07-09T09-03-02-00-00_healthbench_EVLdZCZJQYG5EuWtS2TLsK.eval
+full__gpt-5.5__2026-07-09-FAILED__0905.eval,full,gpt-5.5,2026-07-09,1,,openai/gpt-4o-mini,not-a-run,error,,,healthbench-v1,2026-07-09T09-05-32-00-00_healthbench_Aucfe2K5BgdpYrkukVNJcf.eval
+full__gpt-5.5__2026-07-09.eval,full,gpt-5.5,2026-07-09,1,5000,openai/gpt-4o-mini,fresh,success,0.4873,,healthbench-v1,2026-07-09T09-09-45-00-00_healthbench_gGjNAEJ4UezLEQvJPVi7Zi.eval
+full__opus-4.7__2026-07-09.eval,full,opus-4.7,2026-07-09,1,5000,openai/gpt-4o-mini,fresh,success,0.4762,,healthbench-v1,2026-07-09T10-33-03-00-00_healthbench_QeHrUjumcLfkEWL5T7HaUa.eval
+full__gpt-5.5__2026-07-15-replay.eval,full,gpt-5.5,2026-07-15,1,5000,openai/gpt-4.1,FULL CACHE REPLAY,success,0.5687,0.5583,hs-non-professional,2026-07-15T06-51-58-00-00_healthbench_WJDhL9LgZJKMGgYizSLYpL.eval
+full__opus-4.7__2026-07-15-replay.eval,full,opus-4.7,2026-07-15,1,5000,openai/gpt-4.1,FULL CACHE REPLAY,success,0.5344,0.5431,hs-non-professional,2026-07-15T07-01-24-00-00_healthbench_Hah9CRDdSrTz7K5EFgRWF8.eval
+full__deepseek-v4-pro__2026-07-16.eval,full,deepseek-v4-pro,2026-07-16,1,5000,openai/gpt-4.1,fresh,success,0.5137,0.4173,hs-non-professional,2026-07-16T04-29-43-00-00_healthbench_jBnP8VSE7rSvrMpzRStQFK.eval
+full__plamo-3.0-prime__2026-07-16.eval,full,plamo-3.0-prime,2026-07-16,1,5000,openai/gpt-4.1,fresh,success,0.3942,0.3243,hs-non-professional,2026-07-16T05-23-21-00-00_healthbench_iexipStaaxm8E5ni3Gg96Z.eval
+hard__gpt-5.5__2026-07-16-replay.eval,hard,gpt-5.5,2026-07-16,1,1000,"openai/gpt-4o-mini (default, inferred)",FULL CACHE REPLAY,success,0.2727,0.2603,hs-hard,2026-07-16T07-28-45-00-00_healthbench-hard_dxTyk6Wmosx3kJAp4iULs9.eval
+hard__opus-4.7__2026-07-16-replay.eval,hard,opus-4.7,2026-07-16,1,1000,"openai/gpt-4o-mini (default, inferred)",FULL CACHE REPLAY,success,0.2661,0.278,hs-hard,2026-07-16T18-52-11-00-00_healthbench-hard_Z9QDknMd4SukiAP7Q4FSPz.eval
+hard__deepseek-v4-pro__2026-07-16-replay.eval,hard,deepseek-v4-pro,2026-07-16,1,1000,"openai/gpt-4o-mini (default, inferred)",FULL CACHE REPLAY,success,0.2484,0.138,hs-hard,2026-07-16T19-07-31-00-00_healthbench-hard_SABwaQjc3Xon5iDMpWFQV3.eval
+hard__plamo-3.0-prime__2026-07-16-replay.eval,hard,plamo-3.0-prime,2026-07-16,1,1000,"openai/gpt-4o-mini (default, inferred)",FULL CACHE REPLAY,success,0.1743,0.0961,hs-hard,2026-07-16T19-23-40-00-00_healthbench-hard_mEs7dJHDapQ4JVqHQtfTJc.eval
+consensus__plamo-3.0-prime__2026-07-16-replay.eval,consensus,plamo-3.0-prime,2026-07-16,1,3671,"openai/gpt-4o-mini (default, inferred)",FULL CACHE REPLAY,success,0.7525,0.7478,hs-consensus,2026-07-16T20-09-58-00-00_healthbench-consensus_87a5hkWgWY5rrHyoygzDVx.eval
+consensus__gpt-5.5__2026-07-16-replay.eval,consensus,gpt-5.5,2026-07-16,1,3671,"openai/gpt-4o-mini (default, inferred)",FULL CACHE REPLAY,success,0.8208,0.8202,hs-consensus,2026-07-16T20-20-37-00-00_healthbench-consensus_gmfrK88PFGpuvRWqnk7uc5.eval
+consensus__deepseek-v4-pro__2026-07-16-replay.eval,consensus,deepseek-v4-pro,2026-07-16,1,3671,"openai/gpt-4o-mini (default, inferred)",FULL CACHE REPLAY,success,0.791,0.7846,hs-consensus,2026-07-16T20-44-07-00-00_healthbench-consensus_EbuNesZCwbaRVAqRedBwjP.eval
+consensus__opus-4.7__2026-07-16-replay.eval,consensus,opus-4.7,2026-07-16,1,3671,"openai/gpt-4o-mini (default, inferred)",FULL CACHE REPLAY,success,0.8017,0.8023,hs-consensus,2026-07-16T21-11-43-00-00_healthbench-consensus_k2nEohAktRtjKfZ5BNGJSG.eval
+consensus__medgemma-4b__2026-07-24.eval,consensus,medgemma-4b,2026-07-24,1,3671,"openai/gpt-4o-mini (default, inferred)",fresh,success,0.7137,0.7077,hs-consensus,2026-07-24T09-07-55-00-00_healthbench-consensus_QX6zAfF9oDZDAX4p6k6yb9.eval
+hard__medgemma-4b__2026-07-24.eval,hard,medgemma-4b,2026-07-24,1,1000,"openai/gpt-4o-mini (default, inferred)",fresh,success,0.106,0.0126,hs-hard,2026-07-24T09-18-15-00-00_healthbench-hard_WkGg4UV64qpccys2fbViHY.eval
+full__medgemma-4b__2026-07-24-cached.eval,full,medgemma-4b,2026-07-24,1,5000,openai/gpt-4.1,MOSTLY CACHED,success,0.2697,0.1832,hs-non-professional,2026-07-24T09-32-53-00-00_healthbench_6Co8iMuXsKxGNrMeAkRbdp.eval
+consensus__medgemma-27b__2026-07-24.eval,consensus,medgemma-27b,2026-07-24,1,3671,"openai/gpt-4o-mini (default, inferred)",fresh,success,0.7758,0.7661,hs-consensus,2026-07-24T11-39-48-00-00_healthbench-consensus_7b4ZMKGYfFkgFNSJXJs53z.eval
+full__medgemma-27b__2026-07-24.eval,full,medgemma-27b,2026-07-24,1,5000,openai/gpt-4.1,fresh,success,0.472,0.3321,hs-non-professional,2026-07-24T12-08-49-00-00_healthbench_cuBWJL4bbU32vR5cxHsjEC.eval
+hard__medgemma-27b__2026-07-24-cached.eval,hard,medgemma-27b,2026-07-24,1,1000,"openai/gpt-4o-mini (default, inferred)",MOSTLY CACHED,success,0.2109,0.0483,hs-hard,2026-07-24T14-36-42-00-00_healthbench-hard_BU7wDz79JMGuziKwJrRrpN.eval
+professional__gpt-5.5__2026-07-24.eval,professional,gpt-5.5,2026-07-24,8,4200,openai/gpt-5.4,fresh,success,0.5295,0.4781,hs-original,2026-07-24T15-57-49-00-00_healthbench-professional_7PZFsBJYTZRCdXZkJPfki7.eval
+professional-consult__gpt-5.5__2026-07-24-replay.eval,professional-consult,gpt-5.5,2026-07-24,8,1888,openai/gpt-5.4,FULL CACHE REPLAY,success,0.51,0.4864,hs-prof-subsets,2026-07-24T16-36-59-00-00_healthbench-professional-consult_D8Z93ZWFtkR4CDbDBLZvBT.eval
+professional-writing__gpt-5.5__2026-07-24-replay.eval,professional-writing,gpt-5.5,2026-07-24,8,1136,openai/gpt-5.4,FULL CACHE REPLAY,success,0.4062,0.3596,hs-prof-subsets,2026-07-24T16-37-37-00-00_healthbench-professional-writing_joZQNJ9ZAZidpUaSpEqyvS.eval
+professional-research__gpt-5.5__2026-07-24-replay.eval,professional-research,gpt-5.5,2026-07-24,8,1176,openai/gpt-5.4,FULL CACHE REPLAY,success,0.6799,0.5792,hs-prof-subsets,2026-07-24T16-37-58-00-00_healthbench-professional-research_3Vd4jhz6oiMQTsC2Mhw849.eval
+professional-redteam__gpt-5.5__2026-07-24-replay.eval,professional-redteam,gpt-5.5,2026-07-24,8,1528,openai/gpt-5.4,FULL CACHE REPLAY,success,0.2992,0.2819,hs-prof-subsets,2026-07-24T16-38-22-00-00_healthbench-professional-red-teaming_Xund3mwQvSqTJfxno7icGe.eval
+professional-baseline__gpt-5.5__2026-07-24.eval,professional-baseline,gpt-5.5,2026-07-24,8,4200,openai/gpt-5.4,baseline(no model gen by design),success,0.4429,0.4387,hs-prof-subsets,2026-07-24T16-38-52-00-00_healthbench-professional-physician-baseline_XADueRF5xwkSmDPWc4LtNf.eval
+professional__opus-4.7__2026-07-24.eval,professional,opus-4.7,2026-07-24,8,4200,openai/gpt-5.4,fresh,success,0.5083,0.4803,hs-original,2026-07-24T16-55-39-00-00_healthbench-professional_boiooCVNG7ZkBVsnEpKLNT.eval
+professional-consult__opus-4.7__2026-07-24-replay.eval,professional-consult,opus-4.7,2026-07-24,8,1888,openai/gpt-5.4,FULL CACHE REPLAY,success,0.4906,0.4704,hs-prof-subsets,2026-07-24T17-19-14-00-00_healthbench-professional-consult_hZgDXu6oT2MuZrn4L7msXN.eval
+professional-writing__opus-4.7__2026-07-24-replay.eval,professional-writing,opus-4.7,2026-07-24,8,1136,openai/gpt-5.4,FULL CACHE REPLAY,success,0.3947,0.3615,hs-prof-subsets,2026-07-24T17-19-42-00-00_healthbench-professional-writing_haJJesaYyVNeNRLHnAi8HD.eval
+professional-research__opus-4.7__2026-07-24-replay.eval,professional-research,opus-4.7,2026-07-24,8,1176,openai/gpt-5.4,FULL CACHE REPLAY,success,0.6464,0.6107,hs-prof-subsets,2026-07-24T17-19-59-00-00_healthbench-professional-research_7RUfdNFLnH7EpAC92NeL7e.eval
+professional-redteam__opus-4.7__2026-07-24-replay.eval,professional-redteam,opus-4.7,2026-07-24,8,1528,openai/gpt-5.4,FULL CACHE REPLAY,success,0.2828,0.2674,hs-prof-subsets,2026-07-24T17-20-17-00-00_healthbench-professional-red-teaming_DLPPa6kbs9VeKS6cfPDhdv.eval
+professional-baseline__opus-4.7__2026-07-24.eval,professional-baseline,opus-4.7,2026-07-24,8,4200,openai/gpt-5.4,baseline(no model gen by design),success,0.4429,0.4387,hs-prof-subsets,2026-07-24T17-20-42-00-00_healthbench-professional-physician-baseline_98adhjPPBGUtnnuJVmWXjn.eval
+professional__plamo-3.0-prime__2026-07-24.eval,professional,plamo-3.0-prime,2026-07-24,8,4200,openai/gpt-5.4,fresh,success,0.2078,0.1374,hs-original,2026-07-24T17-30-37-00-00_healthbench-professional_c9RnXrEaoi4T33bvwXEA5h.eval
+professional-consult__plamo-3.0-prime__2026-07-24-replay.eval,professional-consult,plamo-3.0-prime,2026-07-24,8,1888,openai/gpt-5.4,FULL CACHE REPLAY,success,0.2184,0.1536,hs-prof-subsets,2026-07-24T19-33-04-00-00_healthbench-professional-consult_dgWnf9y2uVjcWoDcyTm3Mm.eval
+professional-writing__plamo-3.0-prime__2026-07-24-replay.eval,professional-writing,plamo-3.0-prime,2026-07-24,8,1136,openai/gpt-5.4,FULL CACHE REPLAY,success,-0.0276,-0.0431,hs-prof-subsets,2026-07-24T19-33-35-00-00_healthbench-professional-writing_eAmnviGLop8QfjxGLMp9Dz.eval
+professional-research__plamo-3.0-prime__2026-07-24-replay.eval,professional-research,plamo-3.0-prime,2026-07-24,8,1176,openai/gpt-5.4,FULL CACHE REPLAY,success,0.4183,0.2858,hs-prof-subsets,2026-07-24T19-33-52-00-00_healthbench-professional-research_WF5VpCWo3K285bwUiVdCUm.eval
+professional-redteam__plamo-3.0-prime__2026-07-24-replay.eval,professional-redteam,plamo-3.0-prime,2026-07-24,8,1528,openai/gpt-5.4,FULL CACHE REPLAY,success,-0.0986,-0.1182,hs-prof-subsets,2026-07-24T19-34-11-00-00_healthbench-professional-red-teaming_4db2cTWDv6VCVoFSFqWpmW.eval
+professional-baseline__plamo-3.0-prime__2026-07-24.eval,professional-baseline,plamo-3.0-prime,2026-07-24,8,4200,openai/gpt-5.4,baseline(no model gen by design),success,0.4429,0.4387,hs-prof-subsets,2026-07-24T19-34-35-00-00_healthbench-professional-physician-baseline_S4NuFZgLXeK7RAgHpRKqTy.eval
+professional-writing__deepseek-v4-pro__2026-07-24.eval,professional-writing,deepseek-v4-pro,2026-07-24,8,1136,openai/gpt-5.4,fresh,success,0.0954,0.0495,hs-prof-subsets,2026-07-24T19-59-01-00-00_healthbench-professional-writing_933975rUF6cdWKJECk44bJ.eval
+professional-research__deepseek-v4-pro__2026-07-24.eval,professional-research,deepseek-v4-pro,2026-07-24,8,1176,openai/gpt-5.4,fresh,success,0.6368,0.5291,hs-prof-subsets,2026-07-24T20-07-30-00-00_healthbench-professional-research_7gKGNrErBwsumd7oKKUcuN.eval
+professional-redteam__deepseek-v4-pro__2026-07-24.eval,professional-redteam,deepseek-v4-pro,2026-07-24,8,1528,openai/gpt-5.4,fresh,success,-0.0372,-0.0687,hs-prof-subsets,2026-07-24T20-18-24-00-00_healthbench-professional-red-teaming_HmwUcZnfLNqmUkyZkHDTdJ.eval
+professional-baseline__deepseek-v4-pro__2026-07-24.eval,professional-baseline,deepseek-v4-pro,2026-07-24,8,4200,openai/gpt-5.4,baseline(no model gen by design),success,0.4429,0.4387,hs-prof-subsets,2026-07-24T20-28-47-00-00_healthbench-professional-physician-baseline_kjY8x2CXBcbHZUSHs55G7D.eval
+professional__medgemma-27b__2026-07-24.eval,professional,medgemma-27b,2026-07-24,8,4200,openai/gpt-5.4,fresh,success,0.3125,0.2003,hs-original,2026-07-24T22-44-03-00-00_healthbench-professional_6AYXiEDgGuzNtAcyWPDGrR.eval
+professional-consult__medgemma-27b__2026-07-24-cached.eval,professional-consult,medgemma-27b,2026-07-24,8,1888,openai/gpt-5.4,MOSTLY CACHED,success,0.2851,0.1778,hs-prof-subsets,2026-07-24T23-41-11-00-00_healthbench-professional-consult_muhCA3DyeXvwNpDvn7gpN4.eval
+professional-writing__medgemma-27b__2026-07-24-cached.eval,professional-writing,medgemma-27b,2026-07-24,8,1136,openai/gpt-5.4,MOSTLY CACHED,success,0.1891,0.0913,hs-prof-subsets,2026-07-24T23-41-59-00-00_healthbench-professional-writing_CMBT9aSmchzcQFRpwZdCqR.eval
+professional-research__medgemma-27b__2026-07-24-cached.eval,professional-research,medgemma-27b,2026-07-24,8,1176,openai/gpt-5.4,MOSTLY CACHED,success,0.4776,0.344,hs-prof-subsets,2026-07-24T23-42-33-00-00_healthbench-professional-research_RFx9qF5ZP369Kdi3CNQU6M.eval
+professional-redteam__medgemma-27b__2026-07-24-replay.eval,professional-redteam,medgemma-27b,2026-07-24,8,1528,openai/gpt-5.4,FULL CACHE REPLAY,success,0.0145,-0.068,hs-prof-subsets,2026-07-24T23-43-33-00-00_healthbench-professional-red-teaming_ewG7cuk2vXaZpcauRLz7qR.eval
+professional-baseline__medgemma-27b__2026-07-24.eval,professional-baseline,medgemma-27b,2026-07-24,8,4200,openai/gpt-5.4,baseline(no model gen by design),success,0.439,0.4348,hs-prof-subsets,2026-07-24T23-44-07-00-00_healthbench-professional-physician-baseline_cY3Z8sUNPRQTpSA3UHa77J.eval
+professional__medgemma-4b__2026-07-25.eval,professional,medgemma-4b,2026-07-25,8,4200,openai/gpt-5.4,fresh,success,0.1651,0.0904,hs-original,2026-07-25T00-26-29-00-00_healthbench-professional_GEViyNR2PM2B2oUV94DzoK.eval
+professional-consult__medgemma-4b__2026-07-25-cached.eval,professional-consult,medgemma-4b,2026-07-25,8,1888,openai/gpt-5.4,MOSTLY CACHED,success,0.152,0.0819,hs-prof-subsets,2026-07-25T00-59-37-00-00_healthbench-professional-consult_ffSg7GfyxjArrdwu7fSrBa.eval
+professional__deepseek-v4-pro__2026-07-25-cached.eval,professional,deepseek-v4-pro,2026-07-25,1,525,openai/gpt-5.4,MOSTLY CACHED,success,0.343,0.2742,hs-original,2026-07-25T18-24-55-00-00_healthbench-professional_HaKmMaK4Q6JCLuX2RLCPnc.eval
+professional-consult__deepseek-v4-pro__2026-07-25-replay.eval,professional-consult,deepseek-v4-pro,2026-07-25,1,236,openai/gpt-5.4,FULL CACHE REPLAY,success,0.3123,0.2559,hs-prof-subsets,2026-07-25T18-26-20-00-00_healthbench-professional-consult_3DSH4qRFzYfb27rH9Z9RqP.eval
+professional-writing__deepseek-v4-pro__2026-07-25-replay.eval,professional-writing,deepseek-v4-pro,2026-07-25,1,142,openai/gpt-5.4,FULL CACHE REPLAY,success,0.0973,0.0516,hs-prof-subsets,2026-07-25T18-26-25-00-00_healthbench-professional-writing_2LJ8vJpJXUX2TbtokQzbxp.eval
+professional-research__deepseek-v4-pro__2026-07-25-replay.eval,professional-research,deepseek-v4-pro,2026-07-25,1,147,openai/gpt-5.4,FULL CACHE REPLAY,success,0.6298,0.5185,hs-prof-subsets,2026-07-25T18-26-28-00-00_healthbench-professional-research_9Pb5iwuDu8dmuFzs5FnmBw.eval
+professional-redteam__deepseek-v4-pro__2026-07-25-replay.eval,professional-redteam,deepseek-v4-pro,2026-07-25,1,191,openai/gpt-5.4,FULL CACHE REPLAY,success,-0.0532,-0.0829,hs-prof-subsets,2026-07-25T18-26-32-00-00_healthbench-professional-red-teaming_e9hCTjjRXrNTnexAxrYM8x.eval
+professional-baseline__deepseek-v4-pro__2026-07-25.eval,professional-baseline,deepseek-v4-pro,2026-07-25,1,525,openai/gpt-5.4,baseline(no model gen by design),success,0.4331,0.4289,hs-prof-subsets,2026-07-25T18-26-35-00-00_healthbench-professional-physician-baseline_Njv8PLSvCJpAvUbPPMQqkx.eval
+hard__medgemma-27b__2026-08-05.eval,hard,medgemma-27b,2026-08-05,1,1000,openai/gpt-4.1,fresh,success,0.1332,-0.0442,medgemma-healthbench-hard,2026-08-05T06-05-04-00-00_healthbench-hard_U2qKVPncnWPTbmLEfZfa4y.eval
+hard__medgemma-4b__2026-08-05.eval,hard,medgemma-4b,2026-08-05,1,1000,openai/gpt-4.1,fresh,success,-0.0312,-0.1641,medgemma-healthbench-hard,2026-08-05T07-09-21-00-00_healthbench-hard_4bZEuQss4Bbee8KixwTS4o.eval
+hard__medgemma-4b__2026-08-05__0754.eval,hard,medgemma-4b,2026-08-05,1,1000,openai/gpt-4.1,fresh,success,-0.0347,-0.1256,healthbench-hard-medgemma-v2,2026-08-05T07-54-55-00-00_healthbench-hard_MANxy36rpJhotpAvdwQiFk.eval
+hard__medgemma-27b__2026-08-05__0807.eval,hard,medgemma-27b,2026-08-05,1,1000,openai/gpt-4.1,fresh,success,0.1415,-0.023,healthbench-hard-medgemma-v2,2026-08-05T08-07-41-00-00_healthbench-hard_D9KnFzsoQRE57vVsejUQnW.eval
+consensus__medgemma-27b__2026-08-05.eval,consensus,medgemma-27b,2026-08-05,1,3671,openai/gpt-4.1,fresh,success,0.9105,0.9007,healthbench-consensus-medgemma,2026-08-05T08-51-22-00-00_healthbench-consensus_6a8D2h492in82NNQHCFgZw.eval
+consensus__medgemma-4b__2026-08-05.eval,consensus,medgemma-4b,2026-08-05,1,3671,openai/gpt-4.1,fresh,success,0.7583,0.7522,healthbench-consensus-medgemma,2026-08-05T10-32-09-00-00_healthbench-consensus_6WXSuGaZvDNS5Zze6Jrm3h.eval
+professional__deepseek-v4-pro__2026-08-06.eval,professional,deepseek-v4-pro,2026-08-06,1,525,openai/gpt-5.4,fresh,success,0.3781,0.3099,healthbench-professional-deepseek-v4-pro,2026-08-06T02-35-30-00-00_healthbench-professional_bbeHCEMNSj6Q362BQHa7mS.eval

index.html

@@ -1,19 +1,378 @@
 <!doctype html>
-<html>
-	<head>
-		<meta charset="utf-8" />
-		<meta name="viewport" content="width=device-width" />
-		<title>My static Space</title>
-		<link rel="stylesheet" href="style.css" />
-	</head>
-	<body>
-		<div class="card">
-			<h1>Welcome to your static Space!</h1>
-			<p>You can modify this app directly by editing <i>index.html</i> in the Files and versions tab.</p>
-			<p>
-				Also don't forget to check the
-				<a href="https://huggingface.co/docs/hub/spaces" target="_blank">Spaces documentation</a>.
-			</p>
-		</div>
-	</body>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<link rel="icon" href="favicon.svg">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>HealthBench eval logs: 68 runs, 6 models, 8 benches</title>
+<style>
+:root{--ivory:#FAF9F5;--slate:#141413;--clay:#D97757;--oat:#E3DACC;--olive:#788C5D;
+--gray-150:#F0EEE6;--gray-300:#D1CFC5;--gray-500:#87867F;--gray-700:#3D3D3A;
+--serif:ui-serif,Georgia,"Times New Roman",serif;--sans:system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;
+--mono:ui-monospace,"SF Mono",Menlo,Consolas,monospace}
+*{box-sizing:border-box;margin:0;padding:0}html{scroll-behavior:smooth}
+body{background:var(--ivory);color:var(--gray-700);font-family:var(--sans);font-size:15px;line-height:1.65;
+-webkit-font-smoothing:antialiased;padding:56px 24px 120px}
+.page{max-width:1160px;margin:0 auto;display:grid;grid-template-columns:210px minmax(0,1fr);gap:44px}
+@media(max-width:920px){.page{grid-template-columns:1fr}nav{display:none}}
+nav{position:sticky;top:32px;align-self:start;font-size:13px}
+nav .label{font-family:var(--mono);font-size:10px;letter-spacing:.1em;text-transform:uppercase;color:var(--gray-500);margin-bottom:12px}
+nav a{display:block;padding:5px 0 5px 12px;border-left:2px solid var(--gray-300);color:var(--gray-700);text-decoration:none}
+nav a:hover{color:var(--slate);border-color:var(--slate)}
+nav a.l2{padding-left:20px;font-size:12px;color:var(--gray-500)}
+nav a .ct{font-family:var(--mono);font-size:10px;color:var(--gray-300)}
+nav .files{margin-top:26px;border-top:1px solid var(--gray-300);padding-top:14px}
+header{margin-bottom:12px}
+.eyebrow{font-family:var(--mono);font-size:11px;letter-spacing:.08em;text-transform:uppercase;color:var(--gray-500);margin-bottom:10px}
+h1{font-family:var(--serif);font-weight:500;font-size:32px;color:var(--slate);letter-spacing:-.01em;margin-bottom:14px}
+.dateline{font-family:var(--mono);font-size:11px;letter-spacing:.04em;color:var(--gray-500);margin-top:-4px}
+h2{font-family:var(--serif);font-weight:500;font-size:22px;color:var(--slate);margin:40px 0 14px;scroll-margin-top:24px}
+h3{font-family:var(--serif);font-weight:500;font-size:16px;color:var(--slate);margin:26px 0 2px;scroll-margin-top:24px}
+h3 .cnt{font-family:var(--mono);font-size:11px;color:var(--gray-500);font-weight:400;margin-left:6px}
+p{margin-bottom:12px;max-width:680px}
+p.blurb{font-size:13px;color:var(--gray-500);margin-bottom:8px}
+ul{margin:0 0 12px 22px;max-width:680px}li{margin-bottom:6px}
+code{font-family:var(--mono);font-size:13px}
+a{color:var(--slate);text-decoration:underline;text-decoration-color:var(--gray-300)}
+a:hover{text-decoration-color:var(--clay)}
+.cards{display:grid;grid-template-columns:1fr 1fr;gap:14px;margin:18px 0;max-width:760px}
+@media(max-width:640px){.cards{grid-template-columns:1fr}}
+.card{border:1.5px solid var(--gray-300);border-radius:10px;background:#fff;padding:16px 18px;text-decoration:none;display:block}
+.card:hover{border-color:var(--clay)}
+.card .k{font-family:var(--mono);font-size:10px;letter-spacing:.08em;text-transform:uppercase;color:var(--gray-500);margin-bottom:6px}
+.card .t{font-family:var(--serif);font-size:18px;color:var(--slate);margin-bottom:6px}
+.card .b{font-size:13px;color:var(--gray-700)}
+.callout{display:flex;gap:12px;border:1.5px solid var(--oat);background:rgba(227,218,204,.35);border-radius:10px;
+padding:14px 16px;margin:18px 0;font-size:14px;max-width:760px}
+.callout .ico{color:var(--clay);font-weight:600}
+.callout p{margin-bottom:6px}.callout p:last-child{margin-bottom:0}
+.stats{display:grid;grid-template-columns:repeat(4,1fr);gap:8px;margin:16px 0;max-width:600px}
+@media(max-width:640px){.stats{grid-template-columns:repeat(2,1fr)}}
+.stat{border:1px solid var(--gray-300);border-radius:8px;background:#fff;padding:10px 12px}
+.stat .v{font-family:var(--serif);font-size:22px;color:var(--slate);line-height:1.2}
+.stat .l{font-family:var(--mono);font-size:9.5px;letter-spacing:.06em;text-transform:uppercase;color:var(--gray-500)}
+#filter{font-family:var(--mono);font-size:13px;padding:9px 12px;border:1.5px solid var(--gray-300);border-radius:8px;
+background:#fff;width:100%;max-width:420px;color:var(--slate)}
+#filter:focus{outline:none;border-color:var(--clay)}
+.fcount{font-family:var(--mono);font-size:11px;color:var(--gray-500);margin-top:6px}
+table.logs{border-collapse:collapse;width:100%;max-width:900px;margin:6px 0 4px;font-size:13px;
+border:1.5px solid var(--gray-300);border-radius:8px;background:#fff}
+table.logs th,table.logs td{padding:6px 9px;border-bottom:1px solid var(--gray-300);text-align:left;white-space:nowrap}
+table.logs thead th{font-family:var(--mono);font-size:9.5px;letter-spacing:.05em;text-transform:uppercase;
+color:var(--gray-500);background:var(--gray-150)}
+table.logs th.n,table.logs td.n{text-align:right;font-variant-numeric:tabular-nums}
+table.logs tbody tr:last-child td{border-bottom:none}
+table.logs td.m{font-weight:500;color:var(--slate)}
+table.logs td.d,table.logs td.j,table.logs td.n{font-family:var(--mono);font-size:12px}
+table.logs td.f a{font-family:var(--mono);font-size:11.5px;color:var(--clay);text-decoration:none;
+border:1px solid var(--gray-300);border-radius:5px;padding:2px 7px}
+table.logs td.f a:hover{border-color:var(--clay);background:rgba(217,119,87,.08)}
+.tag{font-family:var(--mono);font-size:10px;padding:2px 6px;border-radius:4px;border:1px solid var(--gray-300);color:var(--gray-500)}
+.t-ok{border-color:var(--olive);color:#5d6e47;background:rgba(120,140,93,.1)}
+.t-warn{border-color:var(--clay);color:#b0533a;background:rgba(217,119,87,.1)}
+.t-bad{border-color:#B0533A;color:#B0533A;background:rgba(176,83,58,.14)}
+.t-neutral{background:var(--gray-150)}
+.inf{color:var(--clay);font-size:10px}
+table.plain{border-collapse:collapse;width:100%;max-width:760px;margin:14px 0;font-size:14px}
+table.plain th,table.plain td{padding:8px 12px;border-bottom:1px solid var(--gray-300);text-align:left;vertical-align:top}
+table.plain th{font-family:var(--mono);font-size:11px;letter-spacing:.06em;text-transform:uppercase;color:var(--gray-500);
+font-weight:600;border-bottom:1.5px solid var(--gray-300)}
+.legend{font-family:var(--mono);font-size:11px;color:var(--gray-500);margin:10px 0 0;max-width:900px;line-height:1.9}
+</style>
+</head>
+<body>
+<div class="page">
+  <nav>
+    <div class="label">On this page</div>
+    <a href="#start">Start here</a>
+    <a href="#logs">All 68 logs</a>
+        <a href="#full" class="l2">HealthBench full <span class="ct">10</span></a>
+    <a href="#consensus" class="l2">HealthBench consensus <span class="ct">8</span></a>
+    <a href="#hard" class="l2">HealthBench hard <span class="ct">13</span></a>
+    <a href="#professional" class="l2">HealthBench Professional <span class="ct">7</span></a>
+    <a href="#professional-consult" class="l2">Professional: consult <span class="ct">6</span></a>
+    <a href="#professional-writing" class="l2">Professional: writing <span class="ct">6</span></a>
+    <a href="#professional-research" class="l2">Professional: research <span class="ct">6</span></a>
+    <a href="#professional-redteam" class="l2">Professional: red-teaming <span class="ct">6</span></a>
+    <a href="#professional-baseline" class="l2">Professional: physician baseline <span class="ct">6</span></a>
+    <a href="#reports">Analysis</a>
+    <a href="#trust">Before you quote a number</a>
+    <a href="#how">How this was assembled</a>
+    <div class="files">
+      <div class="label">Data files</div>
+      <a href="data/log_mapping.csv">log_mapping.csv</a>
+      <a href="data/MANIFEST.csv">MANIFEST.csv</a>
+      <a href="data/INDEX.md">INDEX.md</a>
+    </div>
+  </nav>
+
+  <main>
+    <header>
+      <div class="eyebrow">HealthBench &middot; Inspect eval logs</div>
+      <h1>HealthBench eval logs: 68 runs, 6 models, 8 benches</h1>
+      <div class="dateline">Created 2026-08-07</div>
+    </header>
+
+    <h2 id="start">Start here</h2>
+
+    <p>Every HealthBench run we have, as Inspect <code>.eval</code> logs, in one place. Six models
+    (GPT-5.5, Claude Opus 4.7, DeepSeek-V4-Pro, PLaMo-3.0-Prime, MedGemma-27B, MedGemma-4B) across
+    HealthBench full, consensus, hard, and Professional with its four use-case slices.</p>
+
+    <div class="stats">
+      <div class="stat"><div class="v">68</div><div class="l">eval logs</div></div>
+      <div class="stat"><div class="v">2.3 GB</div><div class="l">total size</div></div>
+      <div class="stat"><div class="v">23</div><div class="l">fresh runs</div></div>
+      <div class="stat"><div class="v">34</div><div class="l">cache replays</div></div>
+    </div>
+
+    <div class="cards">
+      <a class="card" href="viewer/index.html">
+        <div class="k">browse in-browser</div>
+        <div class="t">Open the log viewer &rarr;</div>
+        <div class="b">The Inspect viewer, loaded with the 7 Professional whole-set runs (Spaces cap out at 1 GB, so the full 2.1 GB set lives in the dataset). Click any run to read
+        individual samples, the model's answer, and every rubric verdict the judge made.</div>
+      </a>
+      <a class="card" href="matrix.html">
+        <div class="k">what exists, what does not</div>
+        <div class="t">Coverage matrix &rarr;</div>
+        <div class="b">Model &times; bench grid: which cells are complete, which are off-config, and
+        the 5 runs still missing. Read this before comparing any two numbers.</div>
+      </a>
+    </div>
+
+    <p>To pull a single log straight down, use the <code>.eval</code> button in the tables below, or
+    fetch it directly:</p>
+
+    <table class="plain">
+      <tbody>
+        <tr><td><code>huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/&lt;bench&gt;__&lt;model&gt;__&lt;date&gt;.eval</code></td></tr>
+      </tbody>
+    </table>
+
+    <p>Filenames carry the config, so <code>professional__gpt-5.5__2026-07-24.eval</code> needs no
+    lookup. A <code>-replay</code> or <code>-cached</code> suffix means the model responses came from
+    Inspect's cache rather than a fresh generation; <code>-FAILED</code> means the run errored and is
+    kept only for provenance.</p>
+
+    <h2 id="logs">All 68 logs</h2>
+
+    <p>Scores are &times;100. <code>raw</code> is the HealthBench score, <code>adj</code> is the
+    length-adjusted one. Grouped by bench, sorted by model.</p>
+
+    <input id="filter" type="search" placeholder="filter: try &quot;professional medgemma&quot; or &quot;replay&quot;" autocomplete="off">
+    <div class="fcount" id="fcount">68 logs</div>
+
+<h3 id="full">HealthBench full <span class="cnt">10 logs</span></h3>
+<p class="blurb">5000 samples, the whole open-ended set</p>
+<table class="logs"><thead><tr><th>Model</th><th>Date</th><th class="n">ep</th><th class="n">n</th><th>Judge</th><th class="n">raw</th><th class="n">adj</th><th>Provenance</th><th>File</th></tr></thead><tbody>
+<tr data-s="gpt-5.5 full 2026-07-09 openai/gpt-4o-mini not-a-run"><td class="m">gpt-5.5</td><td class="d">2026-07-09</td><td class="n">1</td><td class="n">&mdash;</td><td class="j">openai/gpt-4o-mini</td><td class="n">&mdash;</td><td class="n">&mdash;</td><td><span class="tag t-bad">failed</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/full__gpt-5.5__2026-07-09-FAILED.eval" download>.eval</a></td></tr>
+<tr data-s="gpt-5.5 full 2026-07-09 openai/gpt-4o-mini not-a-run"><td class="m">gpt-5.5</td><td class="d">2026-07-09</td><td class="n">1</td><td class="n">&mdash;</td><td class="j">openai/gpt-4o-mini</td><td class="n">&mdash;</td><td class="n">&mdash;</td><td><span class="tag t-bad">failed</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/full__gpt-5.5__2026-07-09-FAILED__0905.eval" download>.eval</a></td></tr>
+<tr data-s="gpt-5.5 full 2026-07-09 openai/gpt-4o-mini fresh"><td class="m">gpt-5.5</td><td class="d">2026-07-09</td><td class="n">1</td><td class="n">5000</td><td class="j">openai/gpt-4o-mini</td><td class="n">48.7</td><td class="n">&mdash;</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/full__gpt-5.5__2026-07-09.eval" download>.eval</a></td></tr>
+<tr data-s="gpt-5.5 full 2026-07-15 gpt-4.1 full cache replay"><td class="m">gpt-5.5</td><td class="d">2026-07-15</td><td class="n">1</td><td class="n">5000</td><td class="j">gpt-4.1</td><td class="n">56.9</td><td class="n">55.8</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/full__gpt-5.5__2026-07-15-replay.eval" download>.eval</a></td></tr>
+<tr data-s="opus-4.7 full 2026-07-09 openai/gpt-4o-mini fresh"><td class="m">opus-4.7</td><td class="d">2026-07-09</td><td class="n">1</td><td class="n">5000</td><td class="j">openai/gpt-4o-mini</td><td class="n">47.6</td><td class="n">&mdash;</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/full__opus-4.7__2026-07-09.eval" download>.eval</a></td></tr>
+<tr data-s="opus-4.7 full 2026-07-15 gpt-4.1 full cache replay"><td class="m">opus-4.7</td><td class="d">2026-07-15</td><td class="n">1</td><td class="n">5000</td><td class="j">gpt-4.1</td><td class="n">53.4</td><td class="n">54.3</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/full__opus-4.7__2026-07-15-replay.eval" download>.eval</a></td></tr>
+<tr data-s="deepseek-v4-pro full 2026-07-16 gpt-4.1 fresh"><td class="m">deepseek-v4-pro</td><td class="d">2026-07-16</td><td class="n">1</td><td class="n">5000</td><td class="j">gpt-4.1</td><td class="n">51.4</td><td class="n">41.7</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/full__deepseek-v4-pro__2026-07-16.eval" download>.eval</a></td></tr>
+<tr data-s="plamo-3.0-prime full 2026-07-16 gpt-4.1 fresh"><td class="m">plamo-3.0-prime</td><td class="d">2026-07-16</td><td class="n">1</td><td class="n">5000</td><td class="j">gpt-4.1</td><td class="n">39.4</td><td class="n">32.4</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/full__plamo-3.0-prime__2026-07-16.eval" download>.eval</a></td></tr>
+<tr data-s="medgemma-27b full 2026-07-24 gpt-4.1 fresh"><td class="m">medgemma-27b</td><td class="d">2026-07-24</td><td class="n">1</td><td class="n">5000</td><td class="j">gpt-4.1</td><td class="n">47.2</td><td class="n">33.2</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/full__medgemma-27b__2026-07-24.eval" download>.eval</a></td></tr>
+<tr data-s="medgemma-4b full 2026-07-24 gpt-4.1 mostly cached"><td class="m">medgemma-4b</td><td class="d">2026-07-24</td><td class="n">1</td><td class="n">5000</td><td class="j">gpt-4.1</td><td class="n">27.0</td><td class="n">18.3</td><td><span class="tag t-warn">cached</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/full__medgemma-4b__2026-07-24-cached.eval" download>.eval</a></td></tr>
+</tbody></table>
+<h3 id="consensus">HealthBench consensus <span class="cnt">8 logs</span></h3>
+<p class="blurb">3671 samples, criteria physicians agreed on</p>
+<table class="logs"><thead><tr><th>Model</th><th>Date</th><th class="n">ep</th><th class="n">n</th><th>Judge</th><th class="n">raw</th><th class="n">adj</th><th>Provenance</th><th>File</th></tr></thead><tbody>
+<tr data-s="gpt-5.5 consensus 2026-07-16 gpt-4o-mini full cache replay"><td class="m">gpt-5.5</td><td class="d">2026-07-16</td><td class="n">1</td><td class="n">3671</td><td class="j">gpt-4o-mini <span class="inf">?</span></td><td class="n">82.1</td><td class="n">82.0</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/consensus__gpt-5.5__2026-07-16-replay.eval" download>.eval</a></td></tr>
+<tr data-s="opus-4.7 consensus 2026-07-16 gpt-4o-mini full cache replay"><td class="m">opus-4.7</td><td class="d">2026-07-16</td><td class="n">1</td><td class="n">3671</td><td class="j">gpt-4o-mini <span class="inf">?</span></td><td class="n">80.2</td><td class="n">80.2</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/consensus__opus-4.7__2026-07-16-replay.eval" download>.eval</a></td></tr>
+<tr data-s="deepseek-v4-pro consensus 2026-07-16 gpt-4o-mini full cache replay"><td class="m">deepseek-v4-pro</td><td class="d">2026-07-16</td><td class="n">1</td><td class="n">3671</td><td class="j">gpt-4o-mini <span class="inf">?</span></td><td class="n">79.1</td><td class="n">78.5</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/consensus__deepseek-v4-pro__2026-07-16-replay.eval" download>.eval</a></td></tr>
+<tr data-s="plamo-3.0-prime consensus 2026-07-16 gpt-4o-mini full cache replay"><td class="m">plamo-3.0-prime</td><td class="d">2026-07-16</td><td class="n">1</td><td class="n">3671</td><td class="j">gpt-4o-mini <span class="inf">?</span></td><td class="n">75.2</td><td class="n">74.8</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/consensus__plamo-3.0-prime__2026-07-16-replay.eval" download>.eval</a></td></tr>
+<tr data-s="medgemma-27b consensus 2026-07-24 gpt-4o-mini fresh"><td class="m">medgemma-27b</td><td class="d">2026-07-24</td><td class="n">1</td><td class="n">3671</td><td class="j">gpt-4o-mini <span class="inf">?</span></td><td class="n">77.6</td><td class="n">76.6</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/consensus__medgemma-27b__2026-07-24.eval" download>.eval</a></td></tr>
+<tr data-s="medgemma-27b consensus 2026-08-05 gpt-4.1 fresh"><td class="m">medgemma-27b</td><td class="d">2026-08-05</td><td class="n">1</td><td class="n">3671</td><td class="j">gpt-4.1</td><td class="n">91.0</td><td class="n">90.1</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/consensus__medgemma-27b__2026-08-05.eval" download>.eval</a></td></tr>
+<tr data-s="medgemma-4b consensus 2026-07-24 gpt-4o-mini fresh"><td class="m">medgemma-4b</td><td class="d">2026-07-24</td><td class="n">1</td><td class="n">3671</td><td class="j">gpt-4o-mini <span class="inf">?</span></td><td class="n">71.4</td><td class="n">70.8</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/consensus__medgemma-4b__2026-07-24.eval" download>.eval</a></td></tr>
+<tr data-s="medgemma-4b consensus 2026-08-05 gpt-4.1 fresh"><td class="m">medgemma-4b</td><td class="d">2026-08-05</td><td class="n">1</td><td class="n">3671</td><td class="j">gpt-4.1</td><td class="n">75.8</td><td class="n">75.2</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/consensus__medgemma-4b__2026-08-05.eval" download>.eval</a></td></tr>
+</tbody></table>
+<h3 id="hard">HealthBench hard <span class="cnt">13 logs</span></h3>
+<p class="blurb">1000 samples, the hardest slice</p>
+<table class="logs"><thead><tr><th>Model</th><th>Date</th><th class="n">ep</th><th class="n">n</th><th>Judge</th><th class="n">raw</th><th class="n">adj</th><th>Provenance</th><th>File</th></tr></thead><tbody>
+<tr data-s="gpt-5.5 hard 2026-07-09 &amp;mdash; not-a-run"><td class="m">gpt-5.5</td><td class="d">2026-07-09</td><td class="n">1</td><td class="n">&mdash;</td><td class="j">&mdash;</td><td class="n">&mdash;</td><td class="n">&mdash;</td><td><span class="tag t-bad">failed</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/hard__gpt-5.5__2026-07-09-FAILED.eval" download>.eval</a></td></tr>
+<tr data-s="gpt-5.5 hard 2026-07-16 gpt-4o-mini full cache replay"><td class="m">gpt-5.5</td><td class="d">2026-07-16</td><td class="n">1</td><td class="n">1000</td><td class="j">gpt-4o-mini <span class="inf">?</span></td><td class="n">27.3</td><td class="n">26.0</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/hard__gpt-5.5__2026-07-16-replay.eval" download>.eval</a></td></tr>
+<tr data-s="opus-4.7 hard 2026-07-09 &amp;mdash; not-a-run"><td class="m">opus-4.7</td><td class="d">2026-07-09</td><td class="n">1</td><td class="n">&mdash;</td><td class="j">&mdash;</td><td class="n">&mdash;</td><td class="n">&mdash;</td><td><span class="tag t-bad">failed</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/hard__opus-4.7__2026-07-09-FAILED.eval" download>.eval</a></td></tr>
+<tr data-s="opus-4.7 hard 2026-07-16 gpt-4o-mini full cache replay"><td class="m">opus-4.7</td><td class="d">2026-07-16</td><td class="n">1</td><td class="n">1000</td><td class="j">gpt-4o-mini <span class="inf">?</span></td><td class="n">26.6</td><td class="n">27.8</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/hard__opus-4.7__2026-07-16-replay.eval" download>.eval</a></td></tr>
+<tr data-s="deepseek-v4-pro hard 2026-07-16 gpt-4o-mini full cache replay"><td class="m">deepseek-v4-pro</td><td class="d">2026-07-16</td><td class="n">1</td><td class="n">1000</td><td class="j">gpt-4o-mini <span class="inf">?</span></td><td class="n">24.8</td><td class="n">13.8</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/hard__deepseek-v4-pro__2026-07-16-replay.eval" download>.eval</a></td></tr>
+<tr data-s="plamo-3.0-prime hard 2026-07-16 gpt-4o-mini full cache replay"><td class="m">plamo-3.0-prime</td><td class="d">2026-07-16</td><td class="n">1</td><td class="n">1000</td><td class="j">gpt-4o-mini <span class="inf">?</span></td><td class="n">17.4</td><td class="n">9.6</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/hard__plamo-3.0-prime__2026-07-16-replay.eval" download>.eval</a></td></tr>
+<tr data-s="medgemma-27b hard 2026-07-24 gpt-4o-mini mostly cached"><td class="m">medgemma-27b</td><td class="d">2026-07-24</td><td class="n">1</td><td class="n">1000</td><td class="j">gpt-4o-mini <span class="inf">?</span></td><td class="n">21.1</td><td class="n">4.8</td><td><span class="tag t-warn">cached</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/hard__medgemma-27b__2026-07-24-cached.eval" download>.eval</a></td></tr>
+<tr data-s="medgemma-27b hard 2026-08-05 gpt-4.1 fresh"><td class="m">medgemma-27b</td><td class="d">2026-08-05</td><td class="n">1</td><td class="n">1000</td><td class="j">gpt-4.1</td><td class="n">13.3</td><td class="n">-4.4</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/hard__medgemma-27b__2026-08-05.eval" download>.eval</a></td></tr>
+<tr data-s="medgemma-27b hard 2026-08-05 gpt-4.1 fresh"><td class="m">medgemma-27b</td><td class="d">2026-08-05</td><td class="n">1</td><td class="n">1000</td><td class="j">gpt-4.1</td><td class="n">14.1</td><td class="n">-2.3</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/hard__medgemma-27b__2026-08-05__0807.eval" download>.eval</a></td></tr>
+<tr data-s="medgemma-4b hard 2026-07-24 gpt-4o-mini fresh"><td class="m">medgemma-4b</td><td class="d">2026-07-24</td><td class="n">1</td><td class="n">1000</td><td class="j">gpt-4o-mini <span class="inf">?</span></td><td class="n">10.6</td><td class="n">1.3</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/hard__medgemma-4b__2026-07-24.eval" download>.eval</a></td></tr>
+<tr data-s="medgemma-4b hard 2026-08-05 gpt-4.1 fresh"><td class="m">medgemma-4b</td><td class="d">2026-08-05</td><td class="n">1</td><td class="n">1000</td><td class="j">gpt-4.1</td><td class="n">-3.1</td><td class="n">-16.4</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/hard__medgemma-4b__2026-08-05.eval" download>.eval</a></td></tr>
+<tr data-s="medgemma-4b hard 2026-08-05 gpt-4.1 fresh"><td class="m">medgemma-4b</td><td class="d">2026-08-05</td><td class="n">1</td><td class="n">1000</td><td class="j">gpt-4.1</td><td class="n">-3.5</td><td class="n">-12.6</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/hard__medgemma-4b__2026-08-05__0754.eval" download>.eval</a></td></tr>
+<tr data-s="gpt-5-nano hard 2026-07-09 &amp;mdash; not-a-run"><td class="m">gpt-5-nano</td><td class="d">2026-07-09</td><td class="n">1</td><td class="n">&mdash;</td><td class="j">&mdash;</td><td class="n">&mdash;</td><td class="n">&mdash;</td><td><span class="tag t-bad">failed</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/hard__gpt-5-nano__2026-07-09-FAILED.eval" download>.eval</a></td></tr>
+</tbody></table>
+<h3 id="professional">HealthBench Professional <span class="cnt">7 logs</span></h3>
+<p class="blurb">525 samples, physician-written, has a human baseline</p>
+<table class="logs"><thead><tr><th>Model</th><th>Date</th><th class="n">ep</th><th class="n">n</th><th>Judge</th><th class="n">raw</th><th class="n">adj</th><th>Provenance</th><th>File</th></tr></thead><tbody>
+<tr data-s="gpt-5.5 professional 2026-07-24 gpt-5.4 fresh"><td class="m">gpt-5.5</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">4200</td><td class="j">gpt-5.4</td><td class="n">52.9</td><td class="n">47.8</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional__gpt-5.5__2026-07-24.eval" download>.eval</a></td></tr>
+<tr data-s="opus-4.7 professional 2026-07-24 gpt-5.4 fresh"><td class="m">opus-4.7</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">4200</td><td class="j">gpt-5.4</td><td class="n">50.8</td><td class="n">48.0</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional__opus-4.7__2026-07-24.eval" download>.eval</a></td></tr>
+<tr data-s="deepseek-v4-pro professional 2026-07-25 gpt-5.4 mostly cached"><td class="m">deepseek-v4-pro</td><td class="d">2026-07-25</td><td class="n">1</td><td class="n">525</td><td class="j">gpt-5.4</td><td class="n">34.3</td><td class="n">27.4</td><td><span class="tag t-warn">cached</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional__deepseek-v4-pro__2026-07-25-cached.eval" download>.eval</a></td></tr>
+<tr data-s="deepseek-v4-pro professional 2026-08-06 gpt-5.4 fresh"><td class="m">deepseek-v4-pro</td><td class="d">2026-08-06</td><td class="n">1</td><td class="n">525</td><td class="j">gpt-5.4</td><td class="n">37.8</td><td class="n">31.0</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional__deepseek-v4-pro__2026-08-06.eval" download>.eval</a></td></tr>
+<tr data-s="plamo-3.0-prime professional 2026-07-24 gpt-5.4 fresh"><td class="m">plamo-3.0-prime</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">4200</td><td class="j">gpt-5.4</td><td class="n">20.8</td><td class="n">13.7</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional__plamo-3.0-prime__2026-07-24.eval" download>.eval</a></td></tr>
+<tr data-s="medgemma-27b professional 2026-07-24 gpt-5.4 fresh"><td class="m">medgemma-27b</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">4200</td><td class="j">gpt-5.4</td><td class="n">31.2</td><td class="n">20.0</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional__medgemma-27b__2026-07-24.eval" download>.eval</a></td></tr>
+<tr data-s="medgemma-4b professional 2026-07-25 gpt-5.4 fresh"><td class="m">medgemma-4b</td><td class="d">2026-07-25</td><td class="n">8</td><td class="n">4200</td><td class="j">gpt-5.4</td><td class="n">16.5</td><td class="n">9.0</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional__medgemma-4b__2026-07-25.eval" download>.eval</a></td></tr>
+</tbody></table>
+<h3 id="professional-consult">Professional: consult <span class="cnt">6 logs</span></h3>
+<p class="blurb">236 samples</p>
+<table class="logs"><thead><tr><th>Model</th><th>Date</th><th class="n">ep</th><th class="n">n</th><th>Judge</th><th class="n">raw</th><th class="n">adj</th><th>Provenance</th><th>File</th></tr></thead><tbody>
+<tr data-s="gpt-5.5 professional-consult 2026-07-24 gpt-5.4 full cache replay"><td class="m">gpt-5.5</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">1888</td><td class="j">gpt-5.4</td><td class="n">51.0</td><td class="n">48.6</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-consult__gpt-5.5__2026-07-24-replay.eval" download>.eval</a></td></tr>
+<tr data-s="opus-4.7 professional-consult 2026-07-24 gpt-5.4 full cache replay"><td class="m">opus-4.7</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">1888</td><td class="j">gpt-5.4</td><td class="n">49.1</td><td class="n">47.0</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-consult__opus-4.7__2026-07-24-replay.eval" download>.eval</a></td></tr>
+<tr data-s="deepseek-v4-pro professional-consult 2026-07-25 gpt-5.4 full cache replay"><td class="m">deepseek-v4-pro</td><td class="d">2026-07-25</td><td class="n">1</td><td class="n">236</td><td class="j">gpt-5.4</td><td class="n">31.2</td><td class="n">25.6</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-consult__deepseek-v4-pro__2026-07-25-replay.eval" download>.eval</a></td></tr>
+<tr data-s="plamo-3.0-prime professional-consult 2026-07-24 gpt-5.4 full cache replay"><td class="m">plamo-3.0-prime</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">1888</td><td class="j">gpt-5.4</td><td class="n">21.8</td><td class="n">15.4</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-consult__plamo-3.0-prime__2026-07-24-replay.eval" download>.eval</a></td></tr>
+<tr data-s="medgemma-27b professional-consult 2026-07-24 gpt-5.4 mostly cached"><td class="m">medgemma-27b</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">1888</td><td class="j">gpt-5.4</td><td class="n">28.5</td><td class="n">17.8</td><td><span class="tag t-warn">cached</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-consult__medgemma-27b__2026-07-24-cached.eval" download>.eval</a></td></tr>
+<tr data-s="medgemma-4b professional-consult 2026-07-25 gpt-5.4 mostly cached"><td class="m">medgemma-4b</td><td class="d">2026-07-25</td><td class="n">8</td><td class="n">1888</td><td class="j">gpt-5.4</td><td class="n">15.2</td><td class="n">8.2</td><td><span class="tag t-warn">cached</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-consult__medgemma-4b__2026-07-25-cached.eval" download>.eval</a></td></tr>
+</tbody></table>
+<h3 id="professional-writing">Professional: writing <span class="cnt">6 logs</span></h3>
+<p class="blurb">142 samples</p>
+<table class="logs"><thead><tr><th>Model</th><th>Date</th><th class="n">ep</th><th class="n">n</th><th>Judge</th><th class="n">raw</th><th class="n">adj</th><th>Provenance</th><th>File</th></tr></thead><tbody>
+<tr data-s="gpt-5.5 professional-writing 2026-07-24 gpt-5.4 full cache replay"><td class="m">gpt-5.5</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">1136</td><td class="j">gpt-5.4</td><td class="n">40.6</td><td class="n">36.0</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-writing__gpt-5.5__2026-07-24-replay.eval" download>.eval</a></td></tr>
+<tr data-s="opus-4.7 professional-writing 2026-07-24 gpt-5.4 full cache replay"><td class="m">opus-4.7</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">1136</td><td class="j">gpt-5.4</td><td class="n">39.5</td><td class="n">36.1</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-writing__opus-4.7__2026-07-24-replay.eval" download>.eval</a></td></tr>
+<tr data-s="deepseek-v4-pro professional-writing 2026-07-24 gpt-5.4 fresh"><td class="m">deepseek-v4-pro</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">1136</td><td class="j">gpt-5.4</td><td class="n">9.5</td><td class="n">5.0</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-writing__deepseek-v4-pro__2026-07-24.eval" download>.eval</a></td></tr>
+<tr data-s="deepseek-v4-pro professional-writing 2026-07-25 gpt-5.4 full cache replay"><td class="m">deepseek-v4-pro</td><td class="d">2026-07-25</td><td class="n">1</td><td class="n">142</td><td class="j">gpt-5.4</td><td class="n">9.7</td><td class="n">5.2</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-writing__deepseek-v4-pro__2026-07-25-replay.eval" download>.eval</a></td></tr>
+<tr data-s="plamo-3.0-prime professional-writing 2026-07-24 gpt-5.4 full cache replay"><td class="m">plamo-3.0-prime</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">1136</td><td class="j">gpt-5.4</td><td class="n">-2.8</td><td class="n">-4.3</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-writing__plamo-3.0-prime__2026-07-24-replay.eval" download>.eval</a></td></tr>
+<tr data-s="medgemma-27b professional-writing 2026-07-24 gpt-5.4 mostly cached"><td class="m">medgemma-27b</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">1136</td><td class="j">gpt-5.4</td><td class="n">18.9</td><td class="n">9.1</td><td><span class="tag t-warn">cached</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-writing__medgemma-27b__2026-07-24-cached.eval" download>.eval</a></td></tr>
+</tbody></table>
+<h3 id="professional-research">Professional: research <span class="cnt">6 logs</span></h3>
+<p class="blurb">147 samples</p>
+<table class="logs"><thead><tr><th>Model</th><th>Date</th><th class="n">ep</th><th class="n">n</th><th>Judge</th><th class="n">raw</th><th class="n">adj</th><th>Provenance</th><th>File</th></tr></thead><tbody>
+<tr data-s="gpt-5.5 professional-research 2026-07-24 gpt-5.4 full cache replay"><td class="m">gpt-5.5</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">1176</td><td class="j">gpt-5.4</td><td class="n">68.0</td><td class="n">57.9</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-research__gpt-5.5__2026-07-24-replay.eval" download>.eval</a></td></tr>
+<tr data-s="opus-4.7 professional-research 2026-07-24 gpt-5.4 full cache replay"><td class="m">opus-4.7</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">1176</td><td class="j">gpt-5.4</td><td class="n">64.6</td><td class="n">61.1</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-research__opus-4.7__2026-07-24-replay.eval" download>.eval</a></td></tr>
+<tr data-s="deepseek-v4-pro professional-research 2026-07-24 gpt-5.4 fresh"><td class="m">deepseek-v4-pro</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">1176</td><td class="j">gpt-5.4</td><td class="n">63.7</td><td class="n">52.9</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-research__deepseek-v4-pro__2026-07-24.eval" download>.eval</a></td></tr>
+<tr data-s="deepseek-v4-pro professional-research 2026-07-25 gpt-5.4 full cache replay"><td class="m">deepseek-v4-pro</td><td class="d">2026-07-25</td><td class="n">1</td><td class="n">147</td><td class="j">gpt-5.4</td><td class="n">63.0</td><td class="n">51.8</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-research__deepseek-v4-pro__2026-07-25-replay.eval" download>.eval</a></td></tr>
+<tr data-s="plamo-3.0-prime professional-research 2026-07-24 gpt-5.4 full cache replay"><td class="m">plamo-3.0-prime</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">1176</td><td class="j">gpt-5.4</td><td class="n">41.8</td><td class="n">28.6</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-research__plamo-3.0-prime__2026-07-24-replay.eval" download>.eval</a></td></tr>
+<tr data-s="medgemma-27b professional-research 2026-07-24 gpt-5.4 mostly cached"><td class="m">medgemma-27b</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">1176</td><td class="j">gpt-5.4</td><td class="n">47.8</td><td class="n">34.4</td><td><span class="tag t-warn">cached</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-research__medgemma-27b__2026-07-24-cached.eval" download>.eval</a></td></tr>
+</tbody></table>
+<h3 id="professional-redteam">Professional: red-teaming <span class="cnt">6 logs</span></h3>
+<p class="blurb">191 samples</p>
+<table class="logs"><thead><tr><th>Model</th><th>Date</th><th class="n">ep</th><th class="n">n</th><th>Judge</th><th class="n">raw</th><th class="n">adj</th><th>Provenance</th><th>File</th></tr></thead><tbody>
+<tr data-s="gpt-5.5 professional-redteam 2026-07-24 gpt-5.4 full cache replay"><td class="m">gpt-5.5</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">1528</td><td class="j">gpt-5.4</td><td class="n">29.9</td><td class="n">28.2</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-redteam__gpt-5.5__2026-07-24-replay.eval" download>.eval</a></td></tr>
+<tr data-s="opus-4.7 professional-redteam 2026-07-24 gpt-5.4 full cache replay"><td class="m">opus-4.7</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">1528</td><td class="j">gpt-5.4</td><td class="n">28.3</td><td class="n">26.7</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-redteam__opus-4.7__2026-07-24-replay.eval" download>.eval</a></td></tr>
+<tr data-s="deepseek-v4-pro professional-redteam 2026-07-24 gpt-5.4 fresh"><td class="m">deepseek-v4-pro</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">1528</td><td class="j">gpt-5.4</td><td class="n">-3.7</td><td class="n">-6.9</td><td><span class="tag t-ok">fresh</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-redteam__deepseek-v4-pro__2026-07-24.eval" download>.eval</a></td></tr>
+<tr data-s="deepseek-v4-pro professional-redteam 2026-07-25 gpt-5.4 full cache replay"><td class="m">deepseek-v4-pro</td><td class="d">2026-07-25</td><td class="n">1</td><td class="n">191</td><td class="j">gpt-5.4</td><td class="n">-5.3</td><td class="n">-8.3</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-redteam__deepseek-v4-pro__2026-07-25-replay.eval" download>.eval</a></td></tr>
+<tr data-s="plamo-3.0-prime professional-redteam 2026-07-24 gpt-5.4 full cache replay"><td class="m">plamo-3.0-prime</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">1528</td><td class="j">gpt-5.4</td><td class="n">-9.9</td><td class="n">-11.8</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-redteam__plamo-3.0-prime__2026-07-24-replay.eval" download>.eval</a></td></tr>
+<tr data-s="medgemma-27b professional-redteam 2026-07-24 gpt-5.4 full cache replay"><td class="m">medgemma-27b</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">1528</td><td class="j">gpt-5.4</td><td class="n">1.5</td><td class="n">-6.8</td><td><span class="tag t-warn">replay</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-redteam__medgemma-27b__2026-07-24-replay.eval" download>.eval</a></td></tr>
+</tbody></table>
+<h3 id="professional-baseline">Professional: physician baseline <span class="cnt">6 logs</span></h3>
+<p class="blurb">the human reference, model-independent</p>
+<table class="logs"><thead><tr><th>Model</th><th>Date</th><th class="n">ep</th><th class="n">n</th><th>Judge</th><th class="n">raw</th><th class="n">adj</th><th>Provenance</th><th>File</th></tr></thead><tbody>
+<tr data-s="gpt-5.5 professional-baseline 2026-07-24 gpt-5.4 baseline(no model gen by design)"><td class="m">gpt-5.5</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">4200</td><td class="j">gpt-5.4</td><td class="n">44.3</td><td class="n">43.9</td><td><span class="tag t-neutral">baseline</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-baseline__gpt-5.5__2026-07-24.eval" download>.eval</a></td></tr>
+<tr data-s="opus-4.7 professional-baseline 2026-07-24 gpt-5.4 baseline(no model gen by design)"><td class="m">opus-4.7</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">4200</td><td class="j">gpt-5.4</td><td class="n">44.3</td><td class="n">43.9</td><td><span class="tag t-neutral">baseline</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-baseline__opus-4.7__2026-07-24.eval" download>.eval</a></td></tr>
+<tr data-s="deepseek-v4-pro professional-baseline 2026-07-24 gpt-5.4 baseline(no model gen by design)"><td class="m">deepseek-v4-pro</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">4200</td><td class="j">gpt-5.4</td><td class="n">44.3</td><td class="n">43.9</td><td><span class="tag t-neutral">baseline</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-baseline__deepseek-v4-pro__2026-07-24.eval" download>.eval</a></td></tr>
+<tr data-s="deepseek-v4-pro professional-baseline 2026-07-25 gpt-5.4 baseline(no model gen by design)"><td class="m">deepseek-v4-pro</td><td class="d">2026-07-25</td><td class="n">1</td><td class="n">525</td><td class="j">gpt-5.4</td><td class="n">43.3</td><td class="n">42.9</td><td><span class="tag t-neutral">baseline</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-baseline__deepseek-v4-pro__2026-07-25.eval" download>.eval</a></td></tr>
+<tr data-s="plamo-3.0-prime professional-baseline 2026-07-24 gpt-5.4 baseline(no model gen by design)"><td class="m">plamo-3.0-prime</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">4200</td><td class="j">gpt-5.4</td><td class="n">44.3</td><td class="n">43.9</td><td><span class="tag t-neutral">baseline</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-baseline__plamo-3.0-prime__2026-07-24.eval" download>.eval</a></td></tr>
+<tr data-s="medgemma-27b professional-baseline 2026-07-24 gpt-5.4 baseline(no model gen by design)"><td class="m">medgemma-27b</td><td class="d">2026-07-24</td><td class="n">8</td><td class="n">4200</td><td class="j">gpt-5.4</td><td class="n">43.9</td><td class="n">43.5</td><td><span class="tag t-neutral">baseline</span></td><td class="f"><a href="https://huggingface.co/datasets/kirby44/healthbench-eval-logs/resolve/main/logs/professional-baseline__medgemma-27b__2026-07-24.eval" download>.eval</a></td></tr>
+</tbody></table>
+
+    <p class="legend">
+      <span class="tag t-ok">fresh</span> model responses generated in this run &nbsp;
+      <span class="tag t-warn">replay</span> every call served from cache &nbsp;
+      <span class="tag t-warn">cached</span> under 200 candidate tokens per sample &nbsp;
+      <span class="tag t-neutral">baseline</span> human responses, no generation by design &nbsp;
+      <span class="tag t-bad">failed</span> errored or cancelled<br>
+      <span class="inf">?</span> judge not recorded in <code>task_args</code>, inferred from the
+      scorer default and confirmed against <code>stats.model_usage</code>
+    </p>
+
+    <h2 id="reports">Analysis</h2>
+
+    <table class="plain">
+      <thead><tr><th>Document</th><th>What it answers</th></tr></thead>
+      <tbody>
+        <tr>
+          <td><a href="matrix.html">Coverage matrix</a></td>
+          <td>Which model ran which bench, which cells are comparable, and what is still missing.</td>
+        </tr>
+        <tr>
+          <td><a href="config-check-v2.html">Config check v2</a></td>
+          <td>Do our numbers reproduce OpenAI's published HealthBench results? Anchored on the
+          physician baseline (ours 43.9 against their 43.7).</td>
+        </tr>
+        <tr>
+          <td><a href="config-check-v1.html">Config check v1</a></td>
+          <td>The earlier pass over the first four spaces. Superseded by v2, kept for history.</td>
+        </tr>
+      </tbody>
+    </table>
+
+    <h2 id="trust">Before you quote a number</h2>
+
+    <p>Four things will bite you if you take a score straight out of a log.</p>
+
+    <table class="plain">
+      <thead><tr><th>Issue</th><th>What to do</th></tr></thead>
+      <tbody>
+        <tr>
+          <td><b>The judge is not constant.</b> Three graders are in play: <code>gpt-4o-mini</code>
+          (hard, consensus), <code>gpt-4.1</code> (full, and the Aug-05 MedGemma re-runs),
+          <code>gpt-5.4</code> (all Professional). Swapping the judge moves a score by up to 14
+          points, and not always in the same direction.</td>
+          <td>Only compare runs sharing a judge. The <code>Judge</code> column above is the check.</td>
+        </tr>
+        <tr>
+          <td><b>Professional epochs are inconsistent.</b> 8 epochs for most models, 1 for DeepSeek.</td>
+          <td>Check the <code>ep</code> column before putting two Professional rows side by side.</td>
+        </tr>
+        <tr>
+          <td><b>In-log subset metrics are wrong.</b> <code>use_case_*_score</code>,
+          <code>specialty_*_score</code>, <code>difficulty_*_score</code> and
+          <code>source_slice_*_score</code> drop the length adjustment and clip each sample to
+          [0,1] first. Errors run up to +32 points, always upward.</td>
+          <td>Use the standalone <code>professional-*</code> logs above for the four use-case
+          slices. For specialty and difficulty, re-aggregate from per-sample scores yourself.</td>
+        </tr>
+        <tr>
+          <td><b><code>cache=true</code> on every run.</b> 34 of 68 logs served some
+          or all calls from cache, so an empty <code>stats.model_usage</code> is a replay, not a run.</td>
+          <td>The <code>Provenance</code> column above already classifies this.</td>
+        </tr>
+      </tbody>
+    </table>
+
+    <div class="callout">
+      <span class="ico">&#9733;</span>
+      <div>
+        <p>The pipeline itself is validated: the physician baseline on Professional lands at
+        <b>43.9</b> against OpenAI's published <b>43.7</b>. Discrepancies in the model numbers are
+        config drift, not a broken harness.</p>
+      </div>
+    </div>
+
+    <h2 id="how">How this was assembled</h2>
+
+    <p>The runs were executed by Ajay between 2026-07-09 and 2026-08-06 and originally published as
+    ten separate HuggingFace Spaces under <code>ajay-citadel</code>. This Space consolidates all of
+    them into one viewer, renames the logs so the config is legible from the filename, and adds the
+    provenance classification that the raw logs do not carry.</p>
+
+    <p><code>data/log_mapping.csv</code> maps every renamed file back to its original space and
+    filename, so nothing here is a dead end. <code>data/MANIFEST.csv</code> is the full per-run
+    header dump: judge, epochs, token counts, package versions.
+    <code>data/INDEX.md</code> is the short version of the traps list.</p>
+
+    <p>The original spaces remain the upstream source. If a number here disagrees with one there,
+    the logs are byte-identical, so the difference is in which run you are reading, not in the data.</p>
+  </main>
+</div>
+<script>
+const f=document.getElementById('filter'),rows=[...document.querySelectorAll('table.logs tbody tr')],
+      cnt=document.getElementById('fcount');
+f.addEventListener('input',()=>{
+  const q=f.value.trim().toLowerCase().split(/\s+/).filter(Boolean);
+  let shown=0;
+  rows.forEach(r=>{const s=r.dataset.s,hit=q.every(t=>s.includes(t));r.style.display=hit?'':'none';if(hit)shown++;});
+  document.querySelectorAll('table.logs').forEach(t=>{
+    const any=[...t.querySelectorAll('tbody tr')].some(r=>r.style.display!=='none');
+    const h3=t.previousElementSibling.previousElementSibling;
+    t.style.display=any?'':'none';
+    if(h3&&h3.tagName==='H3'){h3.style.display=any?'':'none';t.previousElementSibling.style.display=any?'':'none';}
+  });
+  cnt.textContent=q.length?shown+' of '+rows.length+' logs match':rows.length+' logs';
+});
+</script>
+</body>
 </html>

matrix.html

@@ -0,0 +1,583 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<link rel="icon" href="favicon.svg">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>HealthBench coverage: which model ran which bench</title>
+<style>
+  :root {
+    --ivory:   #FAF9F5;
+    --slate:   #141413;
+    --clay:    #D97757;
+    --oat:     #E3DACC;
+    --olive:   #788C5D;
+    --gray-150:#F0EEE6;
+    --gray-300:#D1CFC5;
+    --gray-500:#87867F;
+    --gray-700:#3D3D3A;
+    --serif: ui-serif, Georgia, "Times New Roman", serif;
+    --sans:  system-ui, -apple-system, "Segoe UI", Roboto, sans-serif;
+    --mono:  ui-monospace, "SF Mono", Menlo, Consolas, monospace;
+  }
+  * { box-sizing: border-box; margin: 0; padding: 0; }
+  html { scroll-behavior: smooth; }
+  body {
+    background: var(--ivory);
+    color: var(--gray-700);
+    font-family: var(--sans);
+    font-size: 15px;
+    line-height: 1.65;
+    -webkit-font-smoothing: antialiased;
+    padding: 56px 24px 120px;
+  }
+  .page { max-width: 1160px; margin: 0 auto; display: grid; grid-template-columns: 200px minmax(0, 1fr); gap: 48px; }
+  @media (max-width: 920px) { .page { grid-template-columns: 1fr; } nav { display: none; } }
+
+  nav { position: sticky; top: 32px; align-self: start; font-size: 13px; }
+  nav .label { font-family: var(--mono); font-size: 10px; letter-spacing: 0.1em; text-transform: uppercase; color: var(--gray-500); margin-bottom: 12px; }
+  nav a { display: block; padding: 5px 0 5px 12px; border-left: 2px solid var(--gray-300); color: var(--gray-700); text-decoration: none; }
+  nav a:hover { color: var(--slate); border-color: var(--slate); }
+  nav a.l2 { padding-left: 24px; font-size: 12.5px; color: var(--gray-500); }
+  nav .files { margin-top: 28px; border-top: 1px solid var(--gray-300); padding-top: 16px; }
+  nav .files a { text-decoration: none; }
+  nav .files code { display: block; font-family: var(--mono); font-size: 11px; color: var(--gray-500); padding: 3px 0; }
+
+  header { margin-bottom: 12px; }
+  .eyebrow { font-family: var(--mono); font-size: 11px; letter-spacing: 0.08em; text-transform: uppercase; color: var(--gray-500); margin-bottom: 10px; }
+  h1 { font-family: var(--serif); font-weight: 500; font-size: 32px; color: var(--slate); letter-spacing: -0.01em; margin-bottom: 14px; }
+  .dateline { font-family: var(--mono); font-size: 11px; letter-spacing: 0.04em; color: var(--gray-500); margin-top: -4px; }
+
+  h2 { font-family: var(--serif); font-weight: 500; font-size: 22px; color: var(--slate); margin: 40px 0 14px; scroll-margin-top: 24px; }
+  h3 { font-family: var(--serif); font-weight: 500; font-size: 17px; color: var(--slate); margin: 22px 0 8px; }
+  p { margin-bottom: 12px; max-width: 680px; }
+  ul, ol { margin: 0 0 12px 22px; max-width: 680px; }
+  li { margin-bottom: 6px; }
+  code { font-family: var(--mono); font-size: 13px; }
+  a { color: var(--slate); text-decoration: underline; text-decoration-color: var(--gray-300); }
+  a:hover { text-decoration-color: var(--clay); }
+
+  details { border: 1.5px solid var(--gray-300); border-radius: 10px; background: #fff; margin: 14px 0; overflow: hidden; }
+  summary { list-style: none; cursor: pointer; padding: 14px 16px; font-family: var(--serif); font-size: 16px; color: var(--slate); display: flex; align-items: baseline; gap: 10px; }
+  summary::-webkit-details-marker { display: none; }
+  summary::before { content: "\25B8"; color: var(--clay); font-family: var(--sans); font-size: 12px; transition: transform 120ms; }
+  details[open] summary::before { transform: rotate(90deg); }
+  summary .where { font-family: var(--mono); font-size: 11px; color: var(--gray-500); margin-left: auto; }
+  details .body { padding: 0 16px 16px; }
+  details .body p { font-size: 14px; }
+  details .body pre { background: var(--gray-150); padding: 12px 14px; border-radius: 6px; font-family: var(--mono); font-size: 12px; line-height: 1.55; overflow-x: auto; margin-top: 8px; }
+
+  .callout { display: flex; gap: 12px; border: 1.5px solid var(--oat); background: rgba(227,218,204,0.35); border-radius: 10px; padding: 14px 16px; margin: 18px 0; font-size: 14px; max-width: 760px; }
+  .callout .ico { color: var(--clay); font-weight: 600; }
+  .callout p { margin-bottom: 6px; }
+  .callout p:last-child { margin-bottom: 0; }
+
+  table { border-collapse: collapse; width: 100%; max-width: 760px; margin: 14px 0; font-size: 14px; }
+  th, td { padding: 8px 12px; border-bottom: 1px solid var(--gray-300); text-align: left; vertical-align: top; }
+  th { font-family: var(--mono); font-size: 11px; letter-spacing: 0.06em; text-transform: uppercase; color: var(--gray-500); font-weight: 600; border-bottom: 1.5px solid var(--gray-300); }
+  td.num { text-align: right; font-variant-numeric: tabular-nums; }
+
+  /* the coverage matrix */
+  .mwrap { overflow-x: auto; margin: 16px 0 6px; border: 1.5px solid var(--gray-300); border-radius: 10px; background: #fff; }
+  table.matrix { max-width: none; width: 100%; margin: 0; font-size: 13px; }
+  table.matrix th, table.matrix td { border-bottom: 1px solid var(--gray-300); padding: 6px 7px; }
+  table.matrix thead th { background: var(--gray-150); text-align: right; font-size: 10px; letter-spacing: 0.02em; white-space: nowrap; }
+  table.matrix thead th.rowhead { text-align: left; }
+  table.matrix thead th .jd { display: block; font-family: var(--mono); font-size: 9px; letter-spacing: 0; text-transform: none; color: var(--gray-500); font-weight: 400; }
+  table.matrix tbody th { font-family: var(--sans); font-size: 12.5px; font-weight: 500; color: var(--slate); text-transform: none; letter-spacing: 0; white-space: nowrap; }
+  table.matrix td { text-align: right; font-family: var(--mono); font-variant-numeric: tabular-nums; font-size: 12px; white-space: nowrap; }
+  table.matrix tbody tr:last-child th, table.matrix tbody tr:last-child td { border-bottom: none; }
+  td.st-ok  { background: rgba(120,140,93,0.11); color: var(--slate); }
+  td.st-off { background: rgba(217,119,87,0.13); color: var(--slate); }
+  td.st-gap { background: repeating-linear-gradient(135deg, transparent, transparent 4px, rgba(135,134,127,0.14) 4px, rgba(135,134,127,0.14) 5px); color: var(--gray-500); text-align: center; }
+  td .mk { color: var(--clay); font-size: 10px; vertical-align: super; }
+  td .mkc { color: var(--gray-500); font-size: 10px; vertical-align: super; }
+  tr.baseline th, tr.baseline td { background: var(--gray-150); color: var(--gray-500); font-style: normal; }
+  .mlegend { font-family: var(--mono); font-size: 11px; color: var(--gray-500); margin: 0 0 4px; max-width: none; line-height: 1.9; }
+  .sw { display: inline-block; width: 10px; height: 10px; border-radius: 2px; vertical-align: -1px; margin-right: 4px; }
+  .sw-ok { background: rgba(120,140,93,0.5); }
+  .sw-off { background: rgba(217,119,87,0.55); }
+  .sw-gap { background: repeating-linear-gradient(135deg, transparent, transparent 3px, rgba(135,134,127,0.5) 3px, rgba(135,134,127,0.5) 4px); border: 1px solid var(--gray-300); }
+
+  .ok   { color: var(--olive); font-weight: 600; }
+  .warn { color: var(--clay);  font-weight: 600; }
+  .bad  { color: #B0533A;      font-weight: 600; }
+
+  .ladder { display: grid; grid-template-columns: repeat(8, 1fr); gap: 6px; margin: 14px 0 4px; max-width: 760px; }
+  .ladder .chip { border: 1px solid var(--gray-300); border-radius: 6px; background: #fff; padding: 8px 0; text-align: center; font-family: var(--mono); font-size: 10.5px; color: var(--gray-500); }
+  .ladder .chip.hit { border-color: var(--olive); color: var(--slate); background: rgba(120,140,93,0.10); font-weight: 600; }
+  .ladder-legend { font-family: var(--mono); font-size: 11px; color: var(--gray-500); margin-top: 2px; }
+  @media (max-width: 720px) { .ladder { grid-template-columns: repeat(4, 1fr); } }
+
+  dl.faq { margin-top: 8px; }
+  dl.faq dt { font-family: var(--serif); font-size: 16px; color: var(--slate); margin-top: 18px; }
+  dl.faq dd { font-size: 14px; margin: 4px 0 0; max-width: 640px; }
+</style>
+
+<style id="hbnav-css">
+  .hbnav{position:sticky;top:0;z-index:50;display:flex;flex-wrap:wrap;align-items:center;gap:4px;
+    padding:8px 16px;margin:-56px -24px 40px;background:rgba(250,249,245,.94);
+    backdrop-filter:saturate(180%) blur(8px);border-bottom:1px solid #D1CFC5;
+    font-family:ui-monospace,"SF Mono",Menlo,Consolas,monospace;font-size:11.5px}
+  .hbnav .home{color:#87867F;margin-right:6px;text-decoration:none}
+  .hbnav .home:hover{color:#141413}
+  .hbnav a.pill{color:#3D3D3A;text-decoration:none;padding:3px 9px;border:1px solid #D1CFC5;border-radius:6px}
+  .hbnav a.pill:hover{border-color:#D97757;color:#141413}
+  .hbnav a.pill.here{background:#141413;border-color:#141413;color:#FAF9F5}
+  .hbnav .sp{flex:1}
+  @media(max-width:640px){.hbnav{margin:-56px -24px 28px}}
+</style>
+</head>
+<body>
+<div class="hbnav"><a class="home" href="index.html">&#8592; HealthBench logs</a><a class="pill here" href="matrix.html">Coverage matrix</a><a class="pill" href="config-check-v2.html">Config check v2</a><a class="pill" href="config-check-v1.html">Config check v1</a><span class="sp"></span><a class="pill" href="viewer/index.html">Log viewer &rarr;</a></div>
+<div class="page">
+
+  <nav>
+    <div class="label">On this page</div>
+    <a href="#picture">The coverage picture</a>
+    <a href="#matrix">Model &times; bench matrix</a>
+    <a href="#matrix" class="l2">Reading the cells</a>
+    <a href="#complete">What is already complete</a>
+    <a href="#todo">What to run next</a>
+    <a href="#todo" class="l2">5 required runs</a>
+    <a href="#todo" class="l2">The gpt-4.1 decision</a>
+    <a href="#axes">Slices with no clean data</a>
+    <a href="#why">Why cells disagree</a>
+    <a href="#noise">How big is a real gap</a>
+    <a href="#files">Where everything lives</a>
+    <a href="#faq">Questions</a>
+    <div class="files">
+      <div class="label">Run artifacts</div>
+      <a href="data/MANIFEST.csv"><code>data/MANIFEST.csv</code></a>
+      <a href="data/log_mapping.csv"><code>data/log_mapping.csv</code></a>
+      <a href="data/INDEX.md"><code>data/INDEX.md</code></a>
+      <a href="data/headers_raw.json"><code>data/headers_raw.json</code></a>
+      <a href="viewer/index.html"><code>viewer/logs/*.eval</code></a>
+    </div>
+  </nav>
+
+  <main>
+    <header>
+      <div class="eyebrow">HealthBench &middot; model coverage audit</div>
+      <h1>Which model ran which bench, and what is still missing</h1>
+      <div class="dateline">Created 2026-08-07</div>
+    </header>
+
+    <h2 id="picture">The coverage picture</h2>
+
+    <p><b>Nominally the matrix is nearly full: 45 of 48 model &times; bench cells have a run.
+    Usably, only 3 of the 8 benches have a complete, apples-to-apples set of all 6 models.</b>
+    The blockers are not missing runs, they are config drift: the grader model changes between
+    benches, and <code>epochs</code> changes between models on the same bench.</p>
+
+    <p>Six models were evaluated: GPT-5.5, Claude Opus 4.7, DeepSeek-V4-Pro, PLaMo-3.0-Prime,
+    MedGemma-27B-text-it and MedGemma-4B-it. Eight benches: HealthBench full, consensus and hard,
+    plus HealthBench Professional whole-set and its four use-case slices (consult, writing,
+    research, red-teaming). 68 logs across 10 HuggingFace spaces.</p>
+
+    <p>The good news for a comprehensive story: only <b>5 runs</b> are strictly required to close
+    every real gap, and 3 of those 5 are grader-only passes that reuse cached model responses, so
+    they are cheap. One further decision (what to do about a half-finished
+    <code>gpt-4.1</code> re-grade) is worth 8 more runs or a deletion, your call.</p>
+
+    <h2 id="matrix">Model &times; bench matrix</h2>
+
+    <p>Scores are the <b>length-adjusted</b> HealthBench score &times; 100, which is the headline
+    metric OpenAI reports. The last four columns are the Professional use-case slices. Each column
+    header carries its judge model (<code>4.1</code> = <code>openai/gpt-4.1</code>,
+    <code>4o-mini</code> = <code>openai/gpt-4o-mini</code>, <code>5.4</code> =
+    <code>openai/gpt-5.4</code>) and sample count: columns are internally consistent but
+    <b>not comparable to each other</b>.</p>
+
+    <div class="mwrap">
+    <table class="matrix">
+      <thead>
+        <tr>
+          <th class="rowhead">Model</th>
+          <th>full<span class="jd">4.1 &middot; 5000</span></th>
+          <th>consens.<span class="jd">4o-mini &middot; 3671</span></th>
+          <th>hard<span class="jd">4o-mini &middot; 1000</span></th>
+          <th>prof<span class="jd">5.4 &middot; 525</span></th>
+          <th>consult<span class="jd">5.4 &middot; 236</span></th>
+          <th>writing<span class="jd">5.4 &middot; 142</span></th>
+          <th>research<span class="jd">5.4 &middot; 147</span></th>
+          <th>red&#8209;team<span class="jd">5.4 &middot; 191</span></th>
+        </tr>
+      </thead>
+      <tbody>
+        <tr>
+          <th>GPT-5.5</th>
+          <td class="st-ok">55.8<span class="mkc">c</span></td>
+          <td class="st-ok">82.0<span class="mkc">c</span></td>
+          <td class="st-ok">26.0<span class="mkc">c</span></td>
+          <td class="st-ok">47.8</td>
+          <td class="st-ok">48.6<span class="mkc">c</span></td>
+          <td class="st-ok">36.0<span class="mkc">c</span></td>
+          <td class="st-ok">57.9<span class="mkc">c</span></td>
+          <td class="st-ok">28.2<span class="mkc">c</span></td>
+        </tr>
+        <tr>
+          <th>Opus 4.7</th>
+          <td class="st-ok">54.3<span class="mkc">c</span></td>
+          <td class="st-ok">80.2<span class="mkc">c</span></td>
+          <td class="st-ok">27.8<span class="mkc">c</span></td>
+          <td class="st-ok">48.0</td>
+          <td class="st-ok">47.0<span class="mkc">c</span></td>
+          <td class="st-ok">36.2<span class="mkc">c</span></td>
+          <td class="st-ok">61.1<span class="mkc">c</span></td>
+          <td class="st-ok">26.7<span class="mkc">c</span></td>
+        </tr>
+        <tr>
+          <th>DeepSeek-V4-Pro</th>
+          <td class="st-ok">41.7</td>
+          <td class="st-ok">78.5<span class="mkc">c</span></td>
+          <td class="st-ok">13.8<span class="mkc">c</span></td>
+          <td class="st-off">31.0<span class="mk">&#42;</span></td>
+          <td class="st-off">25.6<span class="mk">&#42;</span><span class="mkc">c</span></td>
+          <td class="st-ok">5.0</td>
+          <td class="st-ok">52.9</td>
+          <td class="st-ok">&minus;6.9</td>
+        </tr>
+        <tr>
+          <th>PLaMo 3.0 Prime</th>
+          <td class="st-ok">32.4</td>
+          <td class="st-ok">74.8<span class="mkc">c</span></td>
+          <td class="st-ok">9.6<span class="mkc">c</span></td>
+          <td class="st-ok">13.7</td>
+          <td class="st-ok">15.4<span class="mkc">c</span></td>
+          <td class="st-ok">&minus;4.3<span class="mkc">c</span></td>
+          <td class="st-ok">28.6<span class="mkc">c</span></td>
+          <td class="st-ok">&minus;11.8<span class="mkc">c</span></td>
+        </tr>
+        <tr>
+          <th>MedGemma 27B</th>
+          <td class="st-ok">33.2</td>
+          <td class="st-ok">76.6</td>
+          <td class="st-ok">4.8<span class="mkc">p</span></td>
+          <td class="st-ok">20.0</td>
+          <td class="st-ok">17.8<span class="mkc">p</span></td>
+          <td class="st-ok">9.1<span class="mkc">p</span></td>
+          <td class="st-ok">34.4<span class="mkc">p</span></td>
+          <td class="st-ok">&minus;6.8<span class="mkc">c</span></td>
+        </tr>
+        <tr>
+          <th>MedGemma 4B</th>
+          <td class="st-ok">18.3<span class="mkc">p</span></td>
+          <td class="st-ok">70.8</td>
+          <td class="st-ok">1.3</td>
+          <td class="st-ok">9.0</td>
+          <td class="st-ok">8.2<span class="mkc">p</span></td>
+          <td class="st-gap">GAP</td>
+          <td class="st-gap">GAP</td>
+          <td class="st-gap">GAP</td>
+        </tr>
+        <tr class="baseline">
+          <th>physician baseline</th>
+          <td class="st-gap">&mdash;</td>
+          <td class="st-gap">&mdash;</td>
+          <td class="st-gap">&mdash;</td>
+          <td>43.9</td>
+          <td class="st-gap">&mdash;</td>
+          <td class="st-gap">&mdash;</td>
+          <td class="st-gap">&mdash;</td>
+          <td class="st-gap">&mdash;</td>
+        </tr>
+      </tbody>
+    </table>
+    </div>
+    <p class="mlegend">
+      <span class="sw sw-ok"></span>in the comparable set for that column &nbsp;
+      <span class="sw sw-off"></span>present but off-config &nbsp;
+      <span class="sw sw-gap"></span>no run exists<br>
+      <span class="mk">&#42;</span> ran at <code>epochs=1</code> where the rest of the column ran <code>epochs=8</code>
+      &nbsp;&middot;&nbsp; <span class="mkc">c</span> score comes from a full cache replay (no model generation)
+      &nbsp;&middot;&nbsp; <span class="mkc">p</span> mostly cached (&lt;200 candidate tokens per sample)
+    </p>
+
+    <h3>Reading the cells</h3>
+
+    <p>The <code>c</code> and <code>p</code> markers matter less than they look. Caching a model's
+    <em>response</em> and re-grading it is legitimate reuse: the response does not depend on the
+    judge. What the markers tell you is that the number cannot be independently reproduced from
+    the published log alone, because the generation happened in some earlier run. 27 of 68 logs
+    are full replays; only 23 generated fresh candidate responses.</p>
+
+    <p>The <span class="warn">&#42;</span> marker is the one that breaks a table. DeepSeek's
+    Professional runs used <code>epochs=1</code> (525 samples) while every other model used
+    <code>epochs=8</code> (4200 samples). More epochs shrinks per-sample judge noise, so DeepSeek's
+    31.0 carries visibly more error than GPT-5.5's 47.8 next to it.</p>
+
+    <h2 id="complete">What is already complete</h2>
+
+    <p>Three benches have all six models under one judge and one epoch setting. These you can
+    publish today without caveats beyond the usual judge disclosure.</p>
+
+    <div class="ladder">
+      <div class="chip hit">full 6/6</div>
+      <div class="chip hit">consensus 6/6</div>
+      <div class="chip hit">hard 6/6</div>
+      <div class="chip">prof 5/6</div>
+      <div class="chip">consult 5/6</div>
+      <div class="chip">writing 5/6</div>
+      <div class="chip">research 5/6</div>
+      <div class="chip">red-team 5/6</div>
+    </div>
+    <div class="ladder-legend">olive = complete 6-model comparable set &middot; outline = incomplete</div>
+
+    <p>Model rankings are stable across every bench, which is the reassuring part: GPT-5.5 and
+    Opus 4.7 lead and are within noise of each other, DeepSeek is third, then MedGemma-27B and
+    PLaMo trade places depending on the bench, and MedGemma-4B is last everywhere. Whatever you
+    fix in the gaps below is very unlikely to reorder the leaderboard.</p>
+
+    <div class="callout">
+      <span class="ico">&#9733;</span>
+      <div>
+        <p>The harness itself is validated. The physician baseline on Professional comes out at
+        <b>43.9</b> against OpenAI's published <b>43.7</b>. That is the anchor that says the
+        scoring pipeline is correct, so the gaps below are coverage problems, not correctness
+        problems.</p>
+      </div>
+    </div>
+
+    <h2 id="todo">What to run next</h2>
+
+    <h3>5 required runs</h3>
+
+    <p>These five close every genuine hole in the matrix. Ordered by how much they buy you.</p>
+
+    <details open>
+      <summary>1 &middot; DeepSeek-V4-Pro, Professional whole-set at <code>epochs=8</code> <span class="where">fixes the &#42; on prof</span></summary>
+      <div class="body">
+        <p>The only run here that needs fresh generation, so it is the expensive one:
+        roughly 1.3M candidate tokens &times; 8 epochs plus about 22M judge tokens, by analogy with
+        the other <code>epochs=8</code> Professional runs. Everything else in the column already
+        ran at 8 epochs, so this single run makes the Professional table uniform.</p>
+        <pre>inspect eval healthbench_professional.py \
+  --model openrouter/deepseek/deepseek-v4-pro \
+  --epochs 8 \
+  -T judge_model=openai/gpt-5.4 -T judge_reasoning_effort=low</pre>
+      </div>
+    </details>
+
+    <details>
+      <summary>2 &middot; DeepSeek-V4-Pro, <code>consult</code> slice at <code>epochs=8</code> <span class="where">fixes the &#42; on consult</span></summary>
+      <div class="body">
+        <p>Once run 1 is in the cache, this is a grader-only pass over the same responses. It is the
+        one slice DeepSeek never got at 8 epochs (writing, research and red-teaming already exist).</p>
+      </div>
+    </details>
+
+    <details>
+      <summary>3&ndash;5 &middot; MedGemma-4B, the three missing Professional slices <span class="where">writing, research, red-teaming</span></summary>
+      <div class="body">
+        <p>MedGemma-4B has <code>consult</code> but not the other three. Its <code>epochs=8</code>
+        whole-set Professional responses are already cached from the <code>hs-original</code> run on
+        2026-07-25, so all three are grader-only passes. Cheapest wins in the list.</p>
+        <pre>for uc in writing research red_teaming; do
+  inspect eval healthbench_professional.py@healthbench_professional_$uc \
+    --model vllm/google/medgemma-4b-it --epochs 8 \
+    -T judge_model=openai/gpt-5.4 -T judge_reasoning_effort=low
+done</pre>
+      </div>
+    </details>
+
+    <h3>The gpt-4.1 decision</h3>
+
+    <p>On 2026-08-05 the two MedGemma models were re-run on <code>hard</code> and
+    <code>consensus</code> with the judge switched from <code>gpt-4o-mini</code> to
+    <code>gpt-4.1</code>. Nothing else was re-run. That leaves a half-populated second column on
+    both benches: 2 of 6 models have a gpt-4.1 number, 6 of 6 have a gpt-4o-mini number.</p>
+
+    <p>Two clean options, and you should pick one explicitly rather than let both sit in the folder:</p>
+
+    <table>
+      <thead><tr><th>Option</th><th>Cost</th><th>Result</th></tr></thead>
+      <tbody>
+        <tr>
+          <td><b>Drop the Aug-05 runs</b> from the reported set, keep gpt-4o-mini for hard and consensus.</td>
+          <td>free</td>
+          <td>Consistent today. But <code>full</code> stays on gpt-4.1 while hard and consensus stay on gpt-4o-mini, so the three non-professional benches still use two judges.</td>
+        </tr>
+        <tr>
+          <td><b>Finish the gpt-4.1 column</b>: re-grade GPT-5.5, Opus 4.7, DeepSeek and PLaMo on hard and consensus.</td>
+          <td>8 grader-only runs, roughly 140M judge tokens</td>
+          <td>All three non-professional benches on gpt-4.1, matching <code>full</code>. This is the version worth publishing if the audience compares across benches.</td>
+        </tr>
+      </tbody>
+    </table>
+
+    <p>The candidate responses for all 8 of those are already cached from July, so option 2 needs no
+    model generation at all, only judge spend.</p>
+
+    <h2 id="axes">Slices with no clean data at all</h2>
+
+    <p>Beyond the four use-case slices, HealthBench Professional carries three more breakdown axes:
+    <b>difficulty</b> (2 levels), <b>source slice</b> (3), and <b>specialty</b> (36). None of these
+    has a usable number for any model, because they exist only as in-log metrics and those metrics
+    are computed wrongly.</p>
+
+    <p>The in-log subset metrics (<code>use_case_*_score</code>, <code>type_*_score</code>,
+    <code>difficulty_*_score</code>, <code>source_slice_*_score</code>, <code>specialty_*_score</code>)
+    discard the length adjustment and clip each sample to [0, 1] before averaging. Comparing them
+    against the standalone slice runs in <code>hs-prof-subsets</code> shows how far off that is:</p>
+
+    <table>
+      <thead>
+        <tr><th>Model &middot; slice</th><th class="num">in-log metric</th><th class="num">standalone run</th><th class="num">error</th></tr>
+      </thead>
+      <tbody>
+        <tr><td>PLaMo &middot; red-teaming</td><td class="num">16.88</td><td class="num">&minus;11.82</td><td class="num bad">+28.7</td></tr>
+        <tr><td>PLaMo &middot; writing</td><td class="num">19.77</td><td class="num">&minus;4.31</td><td class="num bad">+24.1</td></tr>
+        <tr><td>MedGemma-27B &middot; red-teaming</td><td class="num">25.29</td><td class="num">&minus;6.80</td><td class="num bad">+32.1</td></tr>
+        <tr><td>MedGemma-27B &middot; writing</td><td class="num">38.79</td><td class="num">9.13</td><td class="num bad">+29.7</td></tr>
+        <tr><td>GPT-5.5 &middot; red-teaming</td><td class="num">45.80</td><td class="num">28.19</td><td class="num bad">+17.6</td></tr>
+        <tr><td>GPT-5.5 &middot; consult</td><td class="num">60.58</td><td class="num">48.64</td><td class="num bad">+11.9</td></tr>
+        <tr><td>Opus 4.7 &middot; research</td><td class="num">67.39</td><td class="num">61.07</td><td class="num warn">+6.3</td></tr>
+      </tbody>
+    </table>
+
+    <p>Every error is positive, and the worst cases are the slices where models score near zero,
+    which is exactly where per-sample clipping does the most damage. The four use-case slices are
+    safe only because Ajay built <code>hs-prof-subsets</code> to recompute them as standalone task
+    runs. <b>Difficulty, source slice and specialty have no equivalent</b>, so if you need those
+    breakdowns they must be re-aggregated from per-sample scores, or run as standalone filtered
+    tasks the way the use-case slices were.</p>
+
+    <div class="callout">
+      <span class="ico">&#9733;</span>
+      <div>
+        <p>Practical rule: never quote a <code>_scorer/&lt;axis&gt;_score</code> value out of a
+        Professional log. Use <code>hs-prof-subsets</code> for the four use-case slices, and
+        recompute anything else yourself.</p>
+      </div>
+    </div>
+
+    <h2 id="why">Why cells disagree</h2>
+
+    <p>Three sources of drift, in descending order of how much they move a number.</p>
+
+    <details>
+      <summary>Judge model, worth up to 14 points <span class="where">3 judges across the set</span></summary>
+      <div class="body">
+        <p>Raw <code>healthbench_score</code> is just achieved points over possible points
+        (<code>scorer.py:524</code>), so it is fully determined by the judge's rubric verdicts and is
+        untouched by the length-adjustment arguments. That makes these clean single-variable
+        comparisons: same model, same task code, judge swapped.</p>
+        <pre>                          gpt-4o-mini   gpt-4.1     delta
+consensus, MedGemma-27B      0.7758     0.9105     +13.5
+full, GPT-5.5                0.4873     0.5687      +8.1
+full, Opus-4.7               0.4762     0.5344      +5.8
+consensus, MedGemma-4B       0.7137     0.7583      +4.5
+hard, MedGemma-27B           0.2109     0.1415      -6.9
+hard, MedGemma-4B            0.1060    -0.0347     -14.1</pre>
+        <p>gpt-4.1 is <em>more</em> generous than gpt-4o-mini on consensus and <em>harsher</em> on
+        hard, so you cannot even predict the direction of the shift, let alone correct for it. Three
+        judges are in play across the set: gpt-4o-mini (healthbench-v1, all of hs-hard, all of
+        hs-consensus), gpt-4.1 (all of hs-non-professional, plus the Aug-05 re-runs), and gpt-5.4 at
+        <code>reasoning_effort=low</code> for every Professional run.</p>
+      </div>
+    </details>
+
+    <details>
+      <summary>Epochs, worth a couple of points of extra noise <span class="where">8 vs 1 on Professional</span></summary>
+      <div class="body">
+        <p>Professional ran at <code>epochs=8</code> for five models and <code>epochs=1</code> for
+        DeepSeek. Every other bench ran at <code>epochs=1</code> for everyone, so no repeat-epoch
+        variance estimate exists for full, consensus or hard at all.</p>
+      </div>
+    </details>
+
+    <details>
+      <summary>Cache reuse, worth about 3.5 points on one measured pair <span class="where">cache=true everywhere</span></summary>
+      <div class="body">
+        <p>DeepSeek has two Professional runs with identical nominal config. The 2026-07-25 one in
+        <code>hs-original</code> spent 10 candidate tokens per sample (a replay) and scored 34.30 raw;
+        the 2026-08-06 one in <code>healthbench-professional-deepseek-v4-pro</code> generated 1.26M
+        tokens fresh and scored 37.81 raw. Use the fresh one. The 3.5-point gap is about 1.5 standard
+        errors, so it is plausibly just resampling, but it is the only direct measurement of
+        cached-versus-fresh drift in the whole set.</p>
+      </div>
+    </details>
+
+    <h2 id="noise">How big does a gap have to be to matter</h2>
+
+    <p>Bootstrap standard errors from the logs, so you can tell a real difference from resampling
+    noise before commissioning a re-run:</p>
+
+    <table>
+      <thead><tr><th>Bench</th><th class="num">n</th><th class="num">std error</th><th>reads as</th></tr></thead>
+      <tbody>
+        <tr><td>full</td><td class="num">5000</td><td class="num">0.005</td><td>differences under 1 point are noise</td></tr>
+        <tr><td>consensus</td><td class="num">3671</td><td class="num">0.005</td><td>same</td></tr>
+        <tr><td>hard</td><td class="num">1000</td><td class="num">0.010</td><td>differences under 2 points are noise</td></tr>
+        <tr><td>professional</td><td class="num">525</td><td class="num">0.023</td><td>differences under 4.5 points are noise</td></tr>
+      </tbody>
+    </table>
+
+    <p>This is why GPT-5.5 at 47.8 and Opus 4.7 at 48.0 on Professional should be reported as tied,
+    and why the judge-swap effects above (up to 14 points) are unambiguously real rather than noise.</p>
+
+    <p>One footnote: the Aug-05 MedGemma-4B <code>hard</code> runs report a standard error of exactly
+    <code>0.0</code>. That is a clipping artifact, not precision. The bootstrap clamps the mean to
+    [0, 1], and since that run's mean is negative every resample clamps to 0, collapsing the spread.
+    Treat the std error as unknown for any run with a negative mean.</p>
+
+    <h2 id="files">Where everything lives</h2>
+
+    <p>All ten HealthBench spaces are mirrored locally, 68 logs. <code>healthbench-v1</code> is a
+    symlink into the inspect_ai sandbox where it already lived.</p>
+
+    <table>
+      <thead><tr><th>Directory</th><th class="num">runs</th><th>Contents</th></tr></thead>
+      <tbody>
+        <tr><td><code>healthbench-v1</code></td><td class="num">7</td><td>first attempt, 2 usable, gpt-4o-mini judge, no length adjustment</td></tr>
+        <tr><td><code>hs-non-professional</code></td><td class="num">6</td><td>the <code>full</code> column</td></tr>
+        <tr><td><code>hs-consensus</code></td><td class="num">6</td><td>the <code>consensus</code> column</td></tr>
+        <tr><td><code>hs-hard</code></td><td class="num">6</td><td>the <code>hard</code> column</td></tr>
+        <tr><td><code>hs-original</code></td><td class="num">6</td><td>the <code>professional</code> column</td></tr>
+        <tr><td><code>hs-prof-subsets</code></td><td class="num">26</td><td>the four use-case slices plus physician baseline</td></tr>
+        <tr><td><code>medgemma-healthbench-hard</code></td><td class="num">2</td><td>Aug-05 gpt-4.1 re-grade, hard</td></tr>
+        <tr><td><code>healthbench-hard-medgemma-v2</code></td><td class="num">2</td><td>same experiment, different provider prefix</td></tr>
+        <tr><td><code>healthbench-consensus-medgemma</code></td><td class="num">2</td><td>Aug-05 gpt-4.1 re-grade, consensus</td></tr>
+        <tr><td><code>healthbench-professional-deepseek-v4-pro</code></td><td class="num">1</td><td>the fresh DeepSeek Professional run</td></tr>
+      </tbody>
+    </table>
+
+    <p>Per-run detail with judge, epochs, token counts and a provenance classification is in
+    <code>hb-spaces/MANIFEST.csv</code>, one row per log. <code>hb-spaces/INDEX.md</code> carries the
+    trap list. On HuggingFace, <code>healthbench-profes</code> and
+    <code>healthbench-prof-part1</code> exist but hold no logs, so there is nothing to mirror.</p>
+
+    <h2 id="faq">Questions this raises</h2>
+
+    <dl class="faq">
+      <dt>If I only have budget for one thing, what is it?</dt>
+      <dd>The DeepSeek Professional <code>epochs=8</code> run. It is the single cell that breaks the
+      Professional table, and Professional is the bench with the physician baseline, so it is the one
+      you would actually put in front of someone.</dd>
+
+      <dt>Can I merge the four use-case slices back into a whole-set Professional score?</dt>
+      <dd>No. 236 + 142 + 147 + 191 = 716 against 525 total samples, so samples carry multiple
+      use-case tags and the slices overlap. Use the whole-set run for the headline and the slices
+      only as breakdowns.</dd>
+
+      <dt>Are the two MedGemma hard spaces duplicates?</dt>
+      <dd>Effectively yes. <code>medgemma-healthbench-hard</code> and
+      <code>healthbench-hard-medgemma-v2</code> are the same experiment run two hours apart, differing
+      only in whether the model was addressed as <code>openai-api/vllm/google/&hellip;</code> or
+      <code>vllm/google/&hellip;</code>. The 0.8-point spread between them is vLLM sampling noise, which
+      incidentally gives you a free repeatability estimate for the local models.</dd>
+
+      <dt>Was gpt-5-nano ever evaluated?</dt>
+      <dd>One attempt on 2026-07-09, which died on a 401. It is not part of the model set.</dd>
+
+      <dt>Why do so many runs show zero token usage?</dt>
+      <dd><code>cache=true</code> was set on every run, and Inspect does not count cache hits toward
+      <code>stats.model_usage</code>. An empty <code>model_usage</code> means every call was served
+      from cache, which is the fastest way to spot a replay masquerading as a re-run.</dd>
+    </dl>
+
+  </main>
+</div>
+</body>
+</html>

robots.txt

@@ -0,0 +1,2 @@
+User-agent: *
+Disallow: /

viewer/assets/chunk-DfAF0w94.js

@@ -0,0 +1,15 @@
+//#region ../../node_modules/.pnpm/mathxyjax3@0.8.3/node_modules/mathxyjax3/dist/chunk-DfAF0w94.js
+var e = Object.create, t = Object.defineProperty, n = Object.getOwnPropertyDescriptor, r = Object.getOwnPropertyNames, i = Object.getPrototypeOf, a = Object.prototype.hasOwnProperty, o = (e, t) => () => (t || e((t = { exports: {} }).exports, t), t.exports), s = (e, i, o, s) => {
+	if (i && typeof i == `object` || typeof i == `function`) for (var c = r(i), l = 0, u = c.length, d; l < u; l++) d = c[l], !a.call(e, d) && d !== o && t(e, d, {
+		get: ((e) => i[e]).bind(null, d),
+		enumerable: !(s = n(i, d)) || s.enumerable
+	});
+	return e;
+}, c = (n, r, a) => (a = n == null ? {} : e(i(n)), s(r || !n || !n.__esModule ? t(a, `default`, {
+	value: n,
+	enumerable: !0
+}) : a, n)), l = (e) => (t) => c(t.default, e);
+//#endregion
+export { o as n, l as t };
+
+//# sourceMappingURL=chunk-DfAF0w94.js.map

viewer/assets/lib-CBtriEt5.js

@@ -0,0 +1,3317 @@
+import { n as o$1 } from "./chunk-DfAF0w94.js";
+//#region ../../node_modules/.pnpm/mathxyjax3@0.8.3/node_modules/mathxyjax3/dist/lib-CBtriEt5.js
+var t = o$1(((exports) => {
+	function t(e, t, n) {
+		if (n === void 0 && (n = Array.prototype), e && typeof n.find == `function`) return n.find.call(e, t);
+		for (var r = 0; r < e.length; r++) if (Object.prototype.hasOwnProperty.call(e, r)) {
+			var i = e[r];
+			if (t.call(void 0, i, r, e)) return i;
+		}
+	}
+	function n(e, t) {
+		return t === void 0 && (t = Object), t && typeof t.freeze == `function` ? t.freeze(e) : e;
+	}
+	function r(e, t) {
+		if (typeof e != `object` || !e) throw TypeError(`target is not an object`);
+		for (var n in t) Object.prototype.hasOwnProperty.call(t, n) && (e[n] = t[n]);
+		return e;
+	}
+	var i = n({
+		HTML: `text/html`,
+		isHTML: function(e) {
+			return e === i.HTML;
+		},
+		XML_APPLICATION: `application/xml`,
+		XML_TEXT: `text/xml`,
+		XML_XHTML_APPLICATION: `application/xhtml+xml`,
+		XML_SVG_IMAGE: `image/svg+xml`
+	}), a = n({
+		HTML: `http://www.w3.org/1999/xhtml`,
+		isHTML: function(e) {
+			return e === a.HTML;
+		},
+		SVG: `http://www.w3.org/2000/svg`,
+		XML: `http://www.w3.org/XML/1998/namespace`,
+		XMLNS: `http://www.w3.org/2000/xmlns/`
+	});
+	exports.assign = r, exports.find = t, exports.freeze = n, exports.MIME_TYPE = i, exports.NAMESPACE = a;
+})), n = o$1(((exports) => {
+	var n = t(), r = n.find, i = n.NAMESPACE;
+	function a(e) {
+		return e !== ``;
+	}
+	function o(e) {
+		return e ? e.split(/[\t\n\f\r ]+/).filter(a) : [];
+	}
+	function s(e, t) {
+		return e.hasOwnProperty(t) || (e[t] = !0), e;
+	}
+	function c(e) {
+		if (!e) return [];
+		var t = o(e);
+		return Object.keys(t.reduce(s, {}));
+	}
+	function l(e) {
+		return function(t) {
+			return e && e.indexOf(t) !== -1;
+		};
+	}
+	function u(e, t) {
+		for (var n in e) Object.prototype.hasOwnProperty.call(e, n) && (t[n] = e[n]);
+	}
+	function d(e, t) {
+		var n = e.prototype;
+		if (!(n instanceof t)) {
+			function r() {}
+			r.prototype = t.prototype, r = new r(), u(n, r), e.prototype = n = r;
+		}
+		n.constructor != e && (typeof e != `function` && console.error(`unknown Class:` + e), n.constructor = e);
+	}
+	var f = {}, p = f.ELEMENT_NODE = 1, m = f.ATTRIBUTE_NODE = 2, h = f.TEXT_NODE = 3, g = f.CDATA_SECTION_NODE = 4, _ = f.ENTITY_REFERENCE_NODE = 5, v = f.ENTITY_NODE = 6, y = f.PROCESSING_INSTRUCTION_NODE = 7, b = f.COMMENT_NODE = 8, x = f.DOCUMENT_NODE = 9, S = f.DOCUMENT_TYPE_NODE = 10, C = f.DOCUMENT_FRAGMENT_NODE = 11, w = f.NOTATION_NODE = 12, T = {}, E = {};
+	T.INDEX_SIZE_ERR = (E[1] = `Index size error`, 1), T.DOMSTRING_SIZE_ERR = (E[2] = `DOMString size error`, 2);
+	var D = T.HIERARCHY_REQUEST_ERR = (E[3] = `Hierarchy request error`, 3);
+	T.WRONG_DOCUMENT_ERR = (E[4] = `Wrong document`, 4), T.INVALID_CHARACTER_ERR = (E[5] = `Invalid character`, 5), T.NO_DATA_ALLOWED_ERR = (E[6] = `No data allowed`, 6), T.NO_MODIFICATION_ALLOWED_ERR = (E[7] = `No modification allowed`, 7);
+	var O = T.NOT_FOUND_ERR = (E[8] = `Not found`, 8);
+	T.NOT_SUPPORTED_ERR = (E[9] = `Not supported`, 9);
+	var k = T.INUSE_ATTRIBUTE_ERR = (E[10] = `Attribute in use`, 10);
+	T.INVALID_STATE_ERR = (E[11] = `Invalid state`, 11), T.SYNTAX_ERR = (E[12] = `Syntax error`, 12), T.INVALID_MODIFICATION_ERR = (E[13] = `Invalid modification`, 13), T.NAMESPACE_ERR = (E[14] = `Invalid namespace`, 14), T.INVALID_ACCESS_ERR = (E[15] = `Invalid access`, 15);
+	function A(e, t) {
+		if (t instanceof Error) var n = t;
+		else n = this, Error.call(this, E[e]), this.message = E[e], Error.captureStackTrace && Error.captureStackTrace(this, A);
+		return n.code = e, t && (this.message = this.message + `: ` + t), n;
+	}
+	A.prototype = Error.prototype, u(T, A);
+	function j() {}
+	j.prototype = {
+		length: 0,
+		item: function(e) {
+			return e >= 0 && e < this.length ? this[e] : null;
+		},
+		toString: function(e, t) {
+			for (var n = [], r = 0; r < this.length; r++) $(this[r], n, e, t);
+			return n.join(``);
+		},
+		filter: function(e) {
+			return Array.prototype.filter.call(this, e);
+		},
+		indexOf: function(e) {
+			return Array.prototype.indexOf.call(this, e);
+		}
+	};
+	function M(e, t) {
+		this._node = e, this._refresh = t, N(this);
+	}
+	function N(e) {
+		var t = e._node._inc || e._node.ownerDocument._inc;
+		if (e._inc !== t) {
+			var n = e._refresh(e._node);
+			if (Te(e, `length`, n.length), !e.$$length || n.length < e.$$length) for (var r = n.length; r in e; r++) Object.prototype.hasOwnProperty.call(e, r) && delete e[r];
+			u(n, e), e._inc = t;
+		}
+	}
+	M.prototype.item = function(e) {
+		return N(this), this[e] || null;
+	}, d(M, j);
+	function P() {}
+	function F(e, t) {
+		for (var n = e.length; n--;) if (e[n] === t) return n;
+	}
+	function I(e, t, n, r) {
+		if (r ? t[F(t, r)] = n : t[t.length++] = n, e) {
+			n.ownerElement = e;
+			var i = e.ownerDocument;
+			i && (r && te(i, e, r), ee(i, e, n));
+		}
+	}
+	function L(e, t, n) {
+		var r = F(t, n);
+		if (r >= 0) {
+			for (var i = t.length - 1; r < i;) t[r] = t[++r];
+			if (t.length = i, e) {
+				var a = e.ownerDocument;
+				a && (te(a, e, n), n.ownerElement = null);
+			}
+		} else throw new A(O, Error(e.tagName + `@` + n));
+	}
+	P.prototype = {
+		length: 0,
+		item: j.prototype.item,
+		getNamedItem: function(e) {
+			for (var t = this.length; t--;) {
+				var n = this[t];
+				if (n.nodeName == e) return n;
+			}
+		},
+		setNamedItem: function(e) {
+			var t = e.ownerElement;
+			if (t && t != this._ownerElement) throw new A(k);
+			var n = this.getNamedItem(e.nodeName);
+			return I(this._ownerElement, this, e, n), n;
+		},
+		setNamedItemNS: function(e) {
+			var t = e.ownerElement, n;
+			if (t && t != this._ownerElement) throw new A(k);
+			return n = this.getNamedItemNS(e.namespaceURI, e.localName), I(this._ownerElement, this, e, n), n;
+		},
+		removeNamedItem: function(e) {
+			var t = this.getNamedItem(e);
+			return L(this._ownerElement, this, t), t;
+		},
+		removeNamedItemNS: function(e, t) {
+			var n = this.getNamedItemNS(e, t);
+			return L(this._ownerElement, this, n), n;
+		},
+		getNamedItemNS: function(e, t) {
+			for (var n = this.length; n--;) {
+				var r = this[n];
+				if (r.localName == t && r.namespaceURI == e) return r;
+			}
+			return null;
+		}
+	};
+	function R() {}
+	R.prototype = {
+		hasFeature: function(e, t) {
+			return !0;
+		},
+		createDocument: function(e, t, n) {
+			var r = new H();
+			if (r.implementation = this, r.childNodes = new j(), r.doctype = n || null, n && r.appendChild(n), t) {
+				var i = r.createElementNS(e, t);
+				r.appendChild(i);
+			}
+			return r;
+		},
+		createDocumentType: function(e, t, n) {
+			var r = new Z();
+			return r.name = e, r.nodeName = e, r.publicId = t || ``, r.systemId = n || ``, r;
+		}
+	};
+	function z() {}
+	z.prototype = {
+		firstChild: null,
+		lastChild: null,
+		previousSibling: null,
+		nextSibling: null,
+		attributes: null,
+		parentNode: null,
+		childNodes: null,
+		ownerDocument: null,
+		nodeValue: null,
+		namespaceURI: null,
+		prefix: null,
+		localName: null,
+		insertBefore: function(e, t) {
+			return K(this, e, t);
+		},
+		replaceChild: function(e, t) {
+			K(this, e, t, ue), t && this.removeChild(t);
+		},
+		removeChild: function(e) {
+			return ne(this, e);
+		},
+		appendChild: function(e) {
+			return this.insertBefore(e, null);
+		},
+		hasChildNodes: function() {
+			return this.firstChild != null;
+		},
+		cloneNode: function(e) {
+			return we(this.ownerDocument || this, this, e);
+		},
+		normalize: function() {
+			for (var e = this.firstChild; e;) {
+				var t = e.nextSibling;
+				t && t.nodeType == h && e.nodeType == h ? (this.removeChild(t), e.appendData(t.data)) : (e.normalize(), e = t);
+			}
+		},
+		isSupported: function(e, t) {
+			return this.ownerDocument.implementation.hasFeature(e, t);
+		},
+		hasAttributes: function() {
+			return this.attributes.length > 0;
+		},
+		lookupPrefix: function(e) {
+			for (var t = this; t;) {
+				var n = t._nsMap;
+				if (n) {
+					for (var r in n) if (Object.prototype.hasOwnProperty.call(n, r) && n[r] === e) return r;
+				}
+				t = t.nodeType == m ? t.ownerDocument : t.parentNode;
+			}
+			return null;
+		},
+		lookupNamespaceURI: function(e) {
+			for (var t = this; t;) {
+				var n = t._nsMap;
+				if (n && Object.prototype.hasOwnProperty.call(n, e)) return n[e];
+				t = t.nodeType == m ? t.ownerDocument : t.parentNode;
+			}
+			return null;
+		},
+		isDefaultNamespace: function(e) {
+			return this.lookupPrefix(e) == null;
+		}
+	};
+	function B(e) {
+		return e == `<` && `&lt;` || e == `>` && `&gt;` || e == `&` && `&amp;` || e == `"` && `&quot;` || `&#` + e.charCodeAt() + `;`;
+	}
+	u(f, z), u(f, z.prototype);
+	function V(e, t) {
+		if (t(e)) return !0;
+		if (e = e.firstChild) do
+			if (V(e, t)) return !0;
+		while (e = e.nextSibling);
+	}
+	function H() {
+		this.ownerDocument = this;
+	}
+	function ee(e, t, n) {
+		e && e._inc++, n.namespaceURI === i.XMLNS && (t._nsMap[n.prefix ? n.localName : ``] = n.value);
+	}
+	function te(e, t, n, r) {
+		e && e._inc++, n.namespaceURI === i.XMLNS && delete t._nsMap[n.prefix ? n.localName : ``];
+	}
+	function U(e, t, n) {
+		if (e && e._inc) {
+			e._inc++;
+			var r = t.childNodes;
+			if (n) r[r.length++] = n;
+			else {
+				for (var i = t.firstChild, a = 0; i;) r[a++] = i, i = i.nextSibling;
+				r.length = a, delete r[r.length];
+			}
+		}
+	}
+	function ne(e, t) {
+		var n = t.previousSibling, r = t.nextSibling;
+		return n ? n.nextSibling = r : e.firstChild = r, r ? r.previousSibling = n : e.lastChild = n, t.parentNode = null, t.previousSibling = null, t.nextSibling = null, U(e.ownerDocument, e), t;
+	}
+	function re(e) {
+		return e && (e.nodeType === z.DOCUMENT_NODE || e.nodeType === z.DOCUMENT_FRAGMENT_NODE || e.nodeType === z.ELEMENT_NODE);
+	}
+	function ie(e) {
+		return e && (G(e) || ae(e) || W(e) || e.nodeType === z.DOCUMENT_FRAGMENT_NODE || e.nodeType === z.COMMENT_NODE || e.nodeType === z.PROCESSING_INSTRUCTION_NODE);
+	}
+	function W(e) {
+		return e && e.nodeType === z.DOCUMENT_TYPE_NODE;
+	}
+	function G(e) {
+		return e && e.nodeType === z.ELEMENT_NODE;
+	}
+	function ae(e) {
+		return e && e.nodeType === z.TEXT_NODE;
+	}
+	function oe(e, t) {
+		var n = e.childNodes || [];
+		if (r(n, G) || W(t)) return !1;
+		var i = r(n, W);
+		return !(t && i && n.indexOf(i) > n.indexOf(t));
+	}
+	function se(e, t) {
+		var n = e.childNodes || [];
+		function i(e) {
+			return G(e) && e !== t;
+		}
+		if (r(n, i)) return !1;
+		var a = r(n, W);
+		return !(t && a && n.indexOf(a) > n.indexOf(t));
+	}
+	function ce(e, t, n) {
+		if (!re(e)) throw new A(D, `Unexpected parent node type ` + e.nodeType);
+		if (n && n.parentNode !== e) throw new A(O, `child not in parent`);
+		if (!ie(t) || W(t) && e.nodeType !== z.DOCUMENT_NODE) throw new A(D, `Unexpected node type ` + t.nodeType + ` for parent node type ` + e.nodeType);
+	}
+	function le(e, t, n) {
+		var i = e.childNodes || [], a = t.childNodes || [];
+		if (t.nodeType === z.DOCUMENT_FRAGMENT_NODE) {
+			var o = a.filter(G);
+			if (o.length > 1 || r(a, ae)) throw new A(D, `More than one element or text in fragment`);
+			if (o.length === 1 && !oe(e, n)) throw new A(D, `Element in fragment can not be inserted before doctype`);
+		}
+		if (G(t) && !oe(e, n)) throw new A(D, `Only one element can be added and only after doctype`);
+		if (W(t)) {
+			if (r(i, W)) throw new A(D, `Only one doctype is allowed`);
+			var s = r(i, G);
+			if (n && i.indexOf(s) < i.indexOf(n)) throw new A(D, `Doctype can only be inserted before an element`);
+			if (!n && s) throw new A(D, `Doctype can not be appended since element is present`);
+		}
+	}
+	function ue(e, t, n) {
+		var i = e.childNodes || [], a = t.childNodes || [];
+		if (t.nodeType === z.DOCUMENT_FRAGMENT_NODE) {
+			var o = a.filter(G);
+			if (o.length > 1 || r(a, ae)) throw new A(D, `More than one element or text in fragment`);
+			if (o.length === 1 && !se(e, n)) throw new A(D, `Element in fragment can not be inserted before doctype`);
+		}
+		if (G(t) && !se(e, n)) throw new A(D, `Only one element can be added and only after doctype`);
+		if (W(t)) {
+			function e(e) {
+				return W(e) && e !== n;
+			}
+			if (r(i, e)) throw new A(D, `Only one doctype is allowed`);
+			var s = r(i, G);
+			if (n && i.indexOf(s) < i.indexOf(n)) throw new A(D, `Doctype can only be inserted before an element`);
+		}
+	}
+	function K(e, t, n, r) {
+		ce(e, t, n), e.nodeType === z.DOCUMENT_NODE && (r || le)(e, t, n);
+		var i = t.parentNode;
+		if (i && i.removeChild(t), t.nodeType === C) {
+			var a = t.firstChild;
+			if (a == null) return t;
+			var o = t.lastChild;
+		} else a = o = t;
+		var s = n ? n.previousSibling : e.lastChild;
+		a.previousSibling = s, o.nextSibling = n, s ? s.nextSibling = a : e.firstChild = a, n == null ? e.lastChild = o : n.previousSibling = o;
+		do {
+			a.parentNode = e;
+			var c = e.ownerDocument || e;
+			q(a, c);
+		} while (a !== o && (a = a.nextSibling));
+		return U(e.ownerDocument || e, e), t.nodeType == C && (t.firstChild = t.lastChild = null), t;
+	}
+	function q(e, t) {
+		if (e.ownerDocument !== t) {
+			if (e.ownerDocument = t, e.nodeType === p && e.attributes) for (var n = 0; n < e.attributes.length; n++) {
+				var r = e.attributes.item(n);
+				r && (r.ownerDocument = t);
+			}
+			for (var i = e.firstChild; i;) q(i, t), i = i.nextSibling;
+		}
+	}
+	function de(e, t) {
+		t.parentNode && t.parentNode.removeChild(t), t.parentNode = e, t.previousSibling = e.lastChild, t.nextSibling = null, t.previousSibling ? t.previousSibling.nextSibling = t : e.firstChild = t, e.lastChild = t, U(e.ownerDocument, e, t);
+		return q(t, e.ownerDocument || e), t;
+	}
+	H.prototype = {
+		nodeName: `#document`,
+		nodeType: x,
+		doctype: null,
+		documentElement: null,
+		_inc: 1,
+		insertBefore: function(e, t) {
+			if (e.nodeType == C) {
+				for (var n = e.firstChild; n;) {
+					var r = n.nextSibling;
+					this.insertBefore(n, t), n = r;
+				}
+				return e;
+			}
+			return K(this, e, t), q(e, this), this.documentElement === null && e.nodeType === p && (this.documentElement = e), e;
+		},
+		removeChild: function(e) {
+			return this.documentElement == e && (this.documentElement = null), ne(this, e);
+		},
+		replaceChild: function(e, t) {
+			K(this, e, t, ue), q(e, this), t && this.removeChild(t), G(e) && (this.documentElement = e);
+		},
+		importNode: function(e, t) {
+			return Ce(this, e, t);
+		},
+		getElementById: function(e) {
+			var t = null;
+			return V(this.documentElement, function(n) {
+				if (n.nodeType == p && n.getAttribute(`id`) == e) return t = n, !0;
+			}), t;
+		},
+		getElementsByClassName: function(e) {
+			var t = c(e);
+			return new M(this, function(n) {
+				var r = [];
+				return t.length > 0 && V(n.documentElement, function(i) {
+					if (i !== n && i.nodeType === p) {
+						var a = i.getAttribute(`class`);
+						if (a) {
+							var o = e === a;
+							if (!o) {
+								var s = c(a);
+								o = t.every(l(s));
+							}
+							o && r.push(i);
+						}
+					}
+				}), r;
+			});
+		},
+		createElement: function(e) {
+			var t = new J();
+			t.ownerDocument = this, t.nodeName = e, t.tagName = e, t.localName = e, t.childNodes = new j();
+			var n = t.attributes = new P();
+			return n._ownerElement = t, t;
+		},
+		createDocumentFragment: function() {
+			var e = new Q();
+			return e.ownerDocument = this, e.childNodes = new j(), e;
+		},
+		createTextNode: function(e) {
+			var t = new fe();
+			return t.ownerDocument = this, t.appendData(e), t;
+		},
+		createComment: function(e) {
+			var t = new pe();
+			return t.ownerDocument = this, t.appendData(e), t;
+		},
+		createCDATASection: function(e) {
+			var t = new me();
+			return t.ownerDocument = this, t.appendData(e), t;
+		},
+		createProcessingInstruction: function(e, t) {
+			var n = new ve();
+			return n.ownerDocument = this, n.tagName = n.nodeName = n.target = e, n.nodeValue = n.data = t, n;
+		},
+		createAttribute: function(e) {
+			var t = new Y();
+			return t.ownerDocument = this, t.name = e, t.nodeName = e, t.localName = e, t.specified = !0, t;
+		},
+		createEntityReference: function(e) {
+			var t = new _e();
+			return t.ownerDocument = this, t.nodeName = e, t;
+		},
+		createElementNS: function(e, t) {
+			var n = new J(), r = t.split(`:`), i = n.attributes = new P();
+			return n.childNodes = new j(), n.ownerDocument = this, n.nodeName = t, n.tagName = t, n.namespaceURI = e, r.length == 2 ? (n.prefix = r[0], n.localName = r[1]) : n.localName = t, i._ownerElement = n, n;
+		},
+		createAttributeNS: function(e, t) {
+			var n = new Y(), r = t.split(`:`);
+			return n.ownerDocument = this, n.nodeName = t, n.name = t, n.namespaceURI = e, n.specified = !0, r.length == 2 ? (n.prefix = r[0], n.localName = r[1]) : n.localName = t, n;
+		}
+	}, d(H, z);
+	function J() {
+		this._nsMap = {};
+	}
+	J.prototype = {
+		nodeType: p,
+		hasAttribute: function(e) {
+			return this.getAttributeNode(e) != null;
+		},
+		getAttribute: function(e) {
+			var t = this.getAttributeNode(e);
+			return t && t.value || ``;
+		},
+		getAttributeNode: function(e) {
+			return this.attributes.getNamedItem(e);
+		},
+		setAttribute: function(e, t) {
+			var n = this.ownerDocument.createAttribute(e);
+			n.value = n.nodeValue = `` + t, this.setAttributeNode(n);
+		},
+		removeAttribute: function(e) {
+			var t = this.getAttributeNode(e);
+			t && this.removeAttributeNode(t);
+		},
+		appendChild: function(e) {
+			return e.nodeType === C ? this.insertBefore(e, null) : de(this, e);
+		},
+		setAttributeNode: function(e) {
+			return this.attributes.setNamedItem(e);
+		},
+		setAttributeNodeNS: function(e) {
+			return this.attributes.setNamedItemNS(e);
+		},
+		removeAttributeNode: function(e) {
+			return this.attributes.removeNamedItem(e.nodeName);
+		},
+		removeAttributeNS: function(e, t) {
+			var n = this.getAttributeNodeNS(e, t);
+			n && this.removeAttributeNode(n);
+		},
+		hasAttributeNS: function(e, t) {
+			return this.getAttributeNodeNS(e, t) != null;
+		},
+		getAttributeNS: function(e, t) {
+			var n = this.getAttributeNodeNS(e, t);
+			return n && n.value || ``;
+		},
+		setAttributeNS: function(e, t, n) {
+			var r = this.ownerDocument.createAttributeNS(e, t);
+			r.value = r.nodeValue = `` + n, this.setAttributeNode(r);
+		},
+		getAttributeNodeNS: function(e, t) {
+			return this.attributes.getNamedItemNS(e, t);
+		},
+		getElementsByTagName: function(e) {
+			return new M(this, function(t) {
+				var n = [];
+				return V(t, function(r) {
+					r !== t && r.nodeType == p && (e === `*` || r.tagName == e) && n.push(r);
+				}), n;
+			});
+		},
+		getElementsByTagNameNS: function(e, t) {
+			return new M(this, function(n) {
+				var r = [];
+				return V(n, function(i) {
+					i !== n && i.nodeType === p && (e === `*` || i.namespaceURI === e) && (t === `*` || i.localName == t) && r.push(i);
+				}), r;
+			});
+		}
+	}, H.prototype.getElementsByTagName = J.prototype.getElementsByTagName, H.prototype.getElementsByTagNameNS = J.prototype.getElementsByTagNameNS, d(J, z);
+	function Y() {}
+	Y.prototype.nodeType = m, d(Y, z);
+	function X() {}
+	X.prototype = {
+		data: ``,
+		substringData: function(e, t) {
+			return this.data.substring(e, e + t);
+		},
+		appendData: function(e) {
+			e = this.data + e, this.nodeValue = this.data = e, this.length = e.length;
+		},
+		insertData: function(e, t) {
+			this.replaceData(e, 0, t);
+		},
+		appendChild: function(e) {
+			throw Error(E[D]);
+		},
+		deleteData: function(e, t) {
+			this.replaceData(e, t, ``);
+		},
+		replaceData: function(e, t, n) {
+			var r = this.data.substring(0, e), i = this.data.substring(e + t);
+			n = r + n + i, this.nodeValue = this.data = n, this.length = n.length;
+		}
+	}, d(X, z);
+	function fe() {}
+	fe.prototype = {
+		nodeName: `#text`,
+		nodeType: h,
+		splitText: function(e) {
+			var t = this.data, n = t.substring(e);
+			t = t.substring(0, e), this.data = this.nodeValue = t, this.length = t.length;
+			var r = this.ownerDocument.createTextNode(n);
+			return this.parentNode && this.parentNode.insertBefore(r, this.nextSibling), r;
+		}
+	}, d(fe, X);
+	function pe() {}
+	pe.prototype = {
+		nodeName: `#comment`,
+		nodeType: b
+	}, d(pe, X);
+	function me() {}
+	me.prototype = {
+		nodeName: `#cdata-section`,
+		nodeType: g
+	}, d(me, X);
+	function Z() {}
+	Z.prototype.nodeType = S, d(Z, z);
+	function he() {}
+	he.prototype.nodeType = w, d(he, z);
+	function ge() {}
+	ge.prototype.nodeType = v, d(ge, z);
+	function _e() {}
+	_e.prototype.nodeType = _, d(_e, z);
+	function Q() {}
+	Q.prototype.nodeName = `#document-fragment`, Q.prototype.nodeType = C, d(Q, z);
+	function ve() {}
+	ve.prototype.nodeType = y, d(ve, z);
+	function ye() {}
+	ye.prototype.serializeToString = function(e, t, n) {
+		return be.call(e, t, n);
+	}, z.prototype.toString = be;
+	function be(e, t) {
+		var n = [], r = this.nodeType == 9 && this.documentElement || this, i = r.prefix, a = r.namespaceURI;
+		if (a && i == null) {
+			var i = r.lookupPrefix(a);
+			if (i == null) var o = [{
+				namespace: a,
+				prefix: null
+			}];
+		}
+		return $(this, n, e, t, o), n.join(``);
+	}
+	function xe(e, t, n) {
+		var r = e.prefix || ``, a = e.namespaceURI;
+		if (!a || r === `xml` && a === i.XML || a === i.XMLNS) return !1;
+		for (var o = n.length; o--;) {
+			var s = n[o];
+			if (s.prefix === r) return s.namespace !== a;
+		}
+		return !0;
+	}
+	function Se(e, t, n) {
+		e.push(` `, t, `="`, n.replace(/[<>&"\t\n\r]/g, B), `"`);
+	}
+	function $(e, t, n, r, a) {
+		if (a ||= [], r) if (e = r(e), e) {
+			if (typeof e == `string`) {
+				t.push(e);
+				return;
+			}
+		} else return;
+		switch (e.nodeType) {
+			case p:
+				var o = e.attributes, s = o.length, c = e.firstChild, l = e.tagName;
+				n = i.isHTML(e.namespaceURI) || n;
+				var u = l;
+				if (!n && !e.prefix && e.namespaceURI) {
+					for (var d, f = 0; f < o.length; f++) if (o.item(f).name === `xmlns`) {
+						d = o.item(f).value;
+						break;
+					}
+					if (!d) for (var v = a.length - 1; v >= 0; v--) {
+						var w = a[v];
+						if (w.prefix === `` && w.namespace === e.namespaceURI) {
+							d = w.namespace;
+							break;
+						}
+					}
+					if (d !== e.namespaceURI) for (var v = a.length - 1; v >= 0; v--) {
+						var w = a[v];
+						if (w.namespace === e.namespaceURI) {
+							w.prefix && (u = w.prefix + `:` + l);
+							break;
+						}
+					}
+				}
+				t.push(`<`, u);
+				for (var T = 0; T < s; T++) {
+					var E = o.item(T);
+					E.prefix == `xmlns` ? a.push({
+						prefix: E.localName,
+						namespace: E.value
+					}) : E.nodeName == `xmlns` && a.push({
+						prefix: ``,
+						namespace: E.value
+					});
+				}
+				for (var T = 0; T < s; T++) {
+					var E = o.item(T);
+					if (xe(E, n, a)) {
+						var D = E.prefix || ``, O = E.namespaceURI;
+						Se(t, D ? `xmlns:` + D : `xmlns`, O), a.push({
+							prefix: D,
+							namespace: O
+						});
+					}
+					$(E, t, n, r, a);
+				}
+				if (l === u && xe(e, n, a)) {
+					var D = e.prefix || ``, O = e.namespaceURI;
+					Se(t, D ? `xmlns:` + D : `xmlns`, O), a.push({
+						prefix: D,
+						namespace: O
+					});
+				}
+				if (c || n && !/^(?:meta|link|img|br|hr|input)$/i.test(l)) {
+					if (t.push(`>`), n && /^script$/i.test(l)) for (; c;) c.data ? t.push(c.data) : $(c, t, n, r, a.slice()), c = c.nextSibling;
+					else for (; c;) $(c, t, n, r, a.slice()), c = c.nextSibling;
+					t.push(`</`, u, `>`);
+				} else t.push(`/>`);
+				return;
+			case x:
+			case C:
+				for (var c = e.firstChild; c;) $(c, t, n, r, a.slice()), c = c.nextSibling;
+				return;
+			case m: return Se(t, e.name, e.value);
+			case h: return t.push(e.data.replace(/[<&>]/g, B));
+			case g: return t.push(`<![CDATA[`, e.data, `]]>`);
+			case b: return t.push(`<!--`, e.data, `-->`);
+			case S:
+				var k = e.publicId, A = e.systemId;
+				if (t.push(`<!DOCTYPE `, e.name), k) t.push(` PUBLIC `, k), A && A != `.` && t.push(` `, A), t.push(`>`);
+				else if (A && A != `.`) t.push(` SYSTEM `, A, `>`);
+				else {
+					var j = e.internalSubset;
+					j && t.push(` [`, j, `]`), t.push(`>`);
+				}
+				return;
+			case y: return t.push(`<?`, e.target, ` `, e.data, `?>`);
+			case _: return t.push(`&`, e.nodeName, `;`);
+			default: t.push(`??`, e.nodeName);
+		}
+	}
+	function Ce(e, t, n) {
+		var r;
+		switch (t.nodeType) {
+			case p: r = t.cloneNode(!1), r.ownerDocument = e;
+			case C: break;
+			case m:
+				n = !0;
+				break;
+		}
+		if (r ||= t.cloneNode(!1), r.ownerDocument = e, r.parentNode = null, n) for (var i = t.firstChild; i;) r.appendChild(Ce(e, i, n)), i = i.nextSibling;
+		return r;
+	}
+	function we(e, t, n) {
+		var r = new t.constructor();
+		for (var i in t) if (Object.prototype.hasOwnProperty.call(t, i)) {
+			var a = t[i];
+			typeof a != `object` && a != r[i] && (r[i] = a);
+		}
+		switch (t.childNodes && (r.childNodes = new j()), r.ownerDocument = e, r.nodeType) {
+			case p:
+				var o = t.attributes, s = r.attributes = new P(), c = o.length;
+				s._ownerElement = r;
+				for (var l = 0; l < c; l++) r.setAttributeNode(we(e, o.item(l), !0));
+				break;
+			case m: n = !0;
+		}
+		if (n) for (var u = t.firstChild; u;) r.appendChild(we(e, u, n)), u = u.nextSibling;
+		return r;
+	}
+	function Te(e, t, n) {
+		e[t] = n;
+	}
+	try {
+		if (Object.defineProperty) {
+			Object.defineProperty(M.prototype, `length`, { get: function() {
+				return N(this), this.$$length;
+			} }), Object.defineProperty(z.prototype, `textContent`, {
+				get: function() {
+					return e(this);
+				},
+				set: function(e) {
+					switch (this.nodeType) {
+						case p:
+						case C:
+							for (; this.firstChild;) this.removeChild(this.firstChild);
+							(e || String(e)) && this.appendChild(this.ownerDocument.createTextNode(e));
+							break;
+						default: this.data = e, this.value = e, this.nodeValue = e;
+					}
+				}
+			});
+			function e(t) {
+				switch (t.nodeType) {
+					case p:
+					case C:
+						var n = [];
+						for (t = t.firstChild; t;) t.nodeType !== 7 && t.nodeType !== 8 && n.push(e(t)), t = t.nextSibling;
+						return n.join(``);
+					default: return t.nodeValue;
+				}
+			}
+			Te = function(e, t, n) {
+				e[`$$` + t] = n;
+			};
+		}
+	} catch {}
+	exports.DocumentType = Z, exports.DOMException = A, exports.DOMImplementation = R, exports.Element = J, exports.Node = z, exports.NodeList = j, exports.XMLSerializer = ye;
+})), r = o$1(((exports) => {
+	var n = t().freeze;
+	exports.XML_ENTITIES = n({
+		amp: `&`,
+		apos: `'`,
+		gt: `>`,
+		lt: `<`,
+		quot: `"`
+	}), exports.HTML_ENTITIES = n({
+		Aacute: `Á`,
+		aacute: `á`,
+		Abreve: `Ă`,
+		abreve: `ă`,
+		ac: `∾`,
+		acd: `∿`,
+		acE: `∾̳`,
+		Acirc: `Â`,
+		acirc: `â`,
+		acute: `´`,
+		Acy: `А`,
+		acy: `а`,
+		AElig: `Æ`,
+		aelig: `æ`,
+		af: `⁡`,
+		Afr: `𝔄`,
+		afr: `𝔞`,
+		Agrave: `À`,
+		agrave: `à`,
+		alefsym: `ℵ`,
+		aleph: `ℵ`,
+		Alpha: `Α`,
+		alpha: `α`,
+		Amacr: `Ā`,
+		amacr: `ā`,
+		amalg: `⨿`,
+		AMP: `&`,
+		amp: `&`,
+		And: `⩓`,
+		and: `∧`,
+		andand: `⩕`,
+		andd: `⩜`,
+		andslope: `⩘`,
+		andv: `⩚`,
+		ang: `∠`,
+		ange: `⦤`,
+		angle: `∠`,
+		angmsd: `∡`,
+		angmsdaa: `⦨`,
+		angmsdab: `⦩`,
+		angmsdac: `⦪`,
+		angmsdad: `⦫`,
+		angmsdae: `⦬`,
+		angmsdaf: `⦭`,
+		angmsdag: `⦮`,
+		angmsdah: `⦯`,
+		angrt: `∟`,
+		angrtvb: `⊾`,
+		angrtvbd: `⦝`,
+		angsph: `∢`,
+		angst: `Å`,
+		angzarr: `⍼`,
+		Aogon: `Ą`,
+		aogon: `ą`,
+		Aopf: `𝔸`,
+		aopf: `𝕒`,
+		ap: `≈`,
+		apacir: `⩯`,
+		apE: `⩰`,
+		ape: `≊`,
+		apid: `≋`,
+		apos: `'`,
+		ApplyFunction: `⁡`,
+		approx: `≈`,
+		approxeq: `≊`,
+		Aring: `Å`,
+		aring: `å`,
+		Ascr: `𝒜`,
+		ascr: `𝒶`,
+		Assign: `≔`,
+		ast: `*`,
+		asymp: `≈`,
+		asympeq: `≍`,
+		Atilde: `Ã`,
+		atilde: `ã`,
+		Auml: `Ä`,
+		auml: `ä`,
+		awconint: `∳`,
+		awint: `⨑`,
+		backcong: `≌`,
+		backepsilon: `϶`,
+		backprime: `‵`,
+		backsim: `∽`,
+		backsimeq: `⋍`,
+		Backslash: `∖`,
+		Barv: `⫧`,
+		barvee: `⊽`,
+		Barwed: `⌆`,
+		barwed: `⌅`,
+		barwedge: `⌅`,
+		bbrk: `⎵`,
+		bbrktbrk: `⎶`,
+		bcong: `≌`,
+		Bcy: `Б`,
+		bcy: `б`,
+		bdquo: `„`,
+		becaus: `∵`,
+		Because: `∵`,
+		because: `∵`,
+		bemptyv: `⦰`,
+		bepsi: `϶`,
+		bernou: `ℬ`,
+		Bernoullis: `ℬ`,
+		Beta: `Β`,
+		beta: `β`,
+		beth: `ℶ`,
+		between: `≬`,
+		Bfr: `𝔅`,
+		bfr: `𝔟`,
+		bigcap: `⋂`,
+		bigcirc: `◯`,
+		bigcup: `⋃`,
+		bigodot: `⨀`,
+		bigoplus: `⨁`,
+		bigotimes: `⨂`,
+		bigsqcup: `⨆`,
+		bigstar: `★`,
+		bigtriangledown: `▽`,
+		bigtriangleup: `△`,
+		biguplus: `⨄`,
+		bigvee: `⋁`,
+		bigwedge: `⋀`,
+		bkarow: `⤍`,
+		blacklozenge: `⧫`,
+		blacksquare: `▪`,
+		blacktriangle: `▴`,
+		blacktriangledown: `▾`,
+		blacktriangleleft: `◂`,
+		blacktriangleright: `▸`,
+		blank: `␣`,
+		blk12: `▒`,
+		blk14: `░`,
+		blk34: `▓`,
+		block: `█`,
+		bne: `=⃥`,
+		bnequiv: `≡⃥`,
+		bNot: `⫭`,
+		bnot: `⌐`,
+		Bopf: `𝔹`,
+		bopf: `𝕓`,
+		bot: `⊥`,
+		bottom: `⊥`,
+		bowtie: `⋈`,
+		boxbox: `⧉`,
+		boxDL: `╗`,
+		boxDl: `╖`,
+		boxdL: `╕`,
+		boxdl: `┐`,
+		boxDR: `╔`,
+		boxDr: `╓`,
+		boxdR: `╒`,
+		boxdr: `┌`,
+		boxH: `═`,
+		boxh: `─`,
+		boxHD: `╦`,
+		boxHd: `╤`,
+		boxhD: `╥`,
+		boxhd: `┬`,
+		boxHU: `╩`,
+		boxHu: `╧`,
+		boxhU: `╨`,
+		boxhu: `┴`,
+		boxminus: `⊟`,
+		boxplus: `⊞`,
+		boxtimes: `⊠`,
+		boxUL: `╝`,
+		boxUl: `╜`,
+		boxuL: `╛`,
+		boxul: `┘`,
+		boxUR: `╚`,
+		boxUr: `╙`,
+		boxuR: `╘`,
+		boxur: `└`,
+		boxV: `║`,
+		boxv: `│`,
+		boxVH: `╬`,
+		boxVh: `╫`,
+		boxvH: `╪`,
+		boxvh: `┼`,
+		boxVL: `╣`,
+		boxVl: `╢`,
+		boxvL: `╡`,
+		boxvl: `┤`,
+		boxVR: `╠`,
+		boxVr: `╟`,
+		boxvR: `╞`,
+		boxvr: `├`,
+		bprime: `‵`,
+		Breve: `˘`,
+		breve: `˘`,
+		brvbar: `¦`,
+		Bscr: `ℬ`,
+		bscr: `𝒷`,
+		bsemi: `⁏`,
+		bsim: `∽`,
+		bsime: `⋍`,
+		bsol: `\\`,
+		bsolb: `⧅`,
+		bsolhsub: `⟈`,
+		bull: `•`,
+		bullet: `•`,
+		bump: `≎`,
+		bumpE: `⪮`,
+		bumpe: `≏`,
+		Bumpeq: `≎`,
+		bumpeq: `≏`,
+		Cacute: `Ć`,
+		cacute: `ć`,
+		Cap: `⋒`,
+		cap: `∩`,
+		capand: `⩄`,
+		capbrcup: `⩉`,
+		capcap: `⩋`,
+		capcup: `⩇`,
+		capdot: `⩀`,
+		CapitalDifferentialD: `ⅅ`,
+		caps: `∩︀`,
+		caret: `⁁`,
+		caron: `ˇ`,
+		Cayleys: `ℭ`,
+		ccaps: `⩍`,
+		Ccaron: `Č`,
+		ccaron: `č`,
+		Ccedil: `Ç`,
+		ccedil: `ç`,
+		Ccirc: `Ĉ`,
+		ccirc: `ĉ`,
+		Cconint: `∰`,
+		ccups: `⩌`,
+		ccupssm: `⩐`,
+		Cdot: `Ċ`,
+		cdot: `ċ`,
+		cedil: `¸`,
+		Cedilla: `¸`,
+		cemptyv: `⦲`,
+		cent: `¢`,
+		CenterDot: `·`,
+		centerdot: `·`,
+		Cfr: `ℭ`,
+		cfr: `𝔠`,
+		CHcy: `Ч`,
+		chcy: `ч`,
+		check: `✓`,
+		checkmark: `✓`,
+		Chi: `Χ`,
+		chi: `χ`,
+		cir: `○`,
+		circ: `ˆ`,
+		circeq: `≗`,
+		circlearrowleft: `↺`,
+		circlearrowright: `↻`,
+		circledast: `⊛`,
+		circledcirc: `⊚`,
+		circleddash: `⊝`,
+		CircleDot: `⊙`,
+		circledR: `®`,
+		circledS: `Ⓢ`,
+		CircleMinus: `⊖`,
+		CirclePlus: `⊕`,
+		CircleTimes: `⊗`,
+		cirE: `⧃`,
+		cire: `≗`,
+		cirfnint: `⨐`,
+		cirmid: `⫯`,
+		cirscir: `⧂`,
+		ClockwiseContourIntegral: `∲`,
+		CloseCurlyDoubleQuote: `”`,
+		CloseCurlyQuote: `’`,
+		clubs: `♣`,
+		clubsuit: `♣`,
+		Colon: `∷`,
+		colon: `:`,
+		Colone: `⩴`,
+		colone: `≔`,
+		coloneq: `≔`,
+		comma: `,`,
+		commat: `@`,
+		comp: `∁`,
+		compfn: `∘`,
+		complement: `∁`,
+		complexes: `ℂ`,
+		cong: `≅`,
+		congdot: `⩭`,
+		Congruent: `≡`,
+		Conint: `∯`,
+		conint: `∮`,
+		ContourIntegral: `∮`,
+		Copf: `ℂ`,
+		copf: `𝕔`,
+		coprod: `∐`,
+		Coproduct: `∐`,
+		COPY: `©`,
+		copy: `©`,
+		copysr: `℗`,
+		CounterClockwiseContourIntegral: `∳`,
+		crarr: `↵`,
+		Cross: `⨯`,
+		cross: `✗`,
+		Cscr: `𝒞`,
+		cscr: `𝒸`,
+		csub: `⫏`,
+		csube: `⫑`,
+		csup: `⫐`,
+		csupe: `⫒`,
+		ctdot: `⋯`,
+		cudarrl: `⤸`,
+		cudarrr: `⤵`,
+		cuepr: `⋞`,
+		cuesc: `⋟`,
+		cularr: `↶`,
+		cularrp: `⤽`,
+		Cup: `⋓`,
+		cup: `∪`,
+		cupbrcap: `⩈`,
+		CupCap: `≍`,
+		cupcap: `⩆`,
+		cupcup: `⩊`,
+		cupdot: `⊍`,
+		cupor: `⩅`,
+		cups: `∪︀`,
+		curarr: `↷`,
+		curarrm: `⤼`,
+		curlyeqprec: `⋞`,
+		curlyeqsucc: `⋟`,
+		curlyvee: `⋎`,
+		curlywedge: `⋏`,
+		curren: `¤`,
+		curvearrowleft: `↶`,
+		curvearrowright: `↷`,
+		cuvee: `⋎`,
+		cuwed: `⋏`,
+		cwconint: `∲`,
+		cwint: `∱`,
+		cylcty: `⌭`,
+		Dagger: `‡`,
+		dagger: `†`,
+		daleth: `ℸ`,
+		Darr: `↡`,
+		dArr: `⇓`,
+		darr: `↓`,
+		dash: `‐`,
+		Dashv: `⫤`,
+		dashv: `⊣`,
+		dbkarow: `⤏`,
+		dblac: `˝`,
+		Dcaron: `Ď`,
+		dcaron: `ď`,
+		Dcy: `Д`,
+		dcy: `д`,
+		DD: `ⅅ`,
+		dd: `ⅆ`,
+		ddagger: `‡`,
+		ddarr: `⇊`,
+		DDotrahd: `⤑`,
+		ddotseq: `⩷`,
+		deg: `°`,
+		Del: `∇`,
+		Delta: `Δ`,
+		delta: `δ`,
+		demptyv: `⦱`,
+		dfisht: `⥿`,
+		Dfr: `𝔇`,
+		dfr: `𝔡`,
+		dHar: `⥥`,
+		dharl: `⇃`,
+		dharr: `⇂`,
+		DiacriticalAcute: `´`,
+		DiacriticalDot: `˙`,
+		DiacriticalDoubleAcute: `˝`,
+		DiacriticalGrave: "`",
+		DiacriticalTilde: `˜`,
+		diam: `⋄`,
+		Diamond: `⋄`,
+		diamond: `⋄`,
+		diamondsuit: `♦`,
+		diams: `♦`,
+		die: `¨`,
+		DifferentialD: `ⅆ`,
+		digamma: `ϝ`,
+		disin: `⋲`,
+		div: `÷`,
+		divide: `÷`,
+		divideontimes: `⋇`,
+		divonx: `⋇`,
+		DJcy: `Ђ`,
+		djcy: `ђ`,
+		dlcorn: `⌞`,
+		dlcrop: `⌍`,
+		dollar: `$`,
+		Dopf: `𝔻`,
+		dopf: `𝕕`,
+		Dot: `¨`,
+		dot: `˙`,
+		DotDot: `⃜`,
+		doteq: `≐`,
+		doteqdot: `≑`,
+		DotEqual: `≐`,
+		dotminus: `∸`,
+		dotplus: `∔`,
+		dotsquare: `⊡`,
+		doublebarwedge: `⌆`,
+		DoubleContourIntegral: `∯`,
+		DoubleDot: `¨`,
+		DoubleDownArrow: `⇓`,
+		DoubleLeftArrow: `⇐`,
+		DoubleLeftRightArrow: `⇔`,
+		DoubleLeftTee: `⫤`,
+		DoubleLongLeftArrow: `⟸`,
+		DoubleLongLeftRightArrow: `⟺`,
+		DoubleLongRightArrow: `⟹`,
+		DoubleRightArrow: `⇒`,
+		DoubleRightTee: `⊨`,
+		DoubleUpArrow: `⇑`,
+		DoubleUpDownArrow: `⇕`,
+		DoubleVerticalBar: `∥`,
+		DownArrow: `↓`,
+		Downarrow: `⇓`,
+		downarrow: `↓`,
+		DownArrowBar: `⤓`,
+		DownArrowUpArrow: `⇵`,
+		DownBreve: `̑`,
+		downdownarrows: `⇊`,
+		downharpoonleft: `⇃`,
+		downharpoonright: `⇂`,
+		DownLeftRightVector: `⥐`,
+		DownLeftTeeVector: `⥞`,
+		DownLeftVector: `↽`,
+		DownLeftVectorBar: `⥖`,
+		DownRightTeeVector: `⥟`,
+		DownRightVector: `⇁`,
+		DownRightVectorBar: `⥗`,
+		DownTee: `⊤`,
+		DownTeeArrow: `↧`,
+		drbkarow: `⤐`,
+		drcorn: `⌟`,
+		drcrop: `⌌`,
+		Dscr: `𝒟`,
+		dscr: `𝒹`,
+		DScy: `Ѕ`,
+		dscy: `ѕ`,
+		dsol: `⧶`,
+		Dstrok: `Đ`,
+		dstrok: `đ`,
+		dtdot: `⋱`,
+		dtri: `▿`,
+		dtrif: `▾`,
+		duarr: `⇵`,
+		duhar: `⥯`,
+		dwangle: `⦦`,
+		DZcy: `Џ`,
+		dzcy: `џ`,
+		dzigrarr: `⟿`,
+		Eacute: `É`,
+		eacute: `é`,
+		easter: `⩮`,
+		Ecaron: `Ě`,
+		ecaron: `ě`,
+		ecir: `≖`,
+		Ecirc: `Ê`,
+		ecirc: `ê`,
+		ecolon: `≕`,
+		Ecy: `Э`,
+		ecy: `э`,
+		eDDot: `⩷`,
+		Edot: `Ė`,
+		eDot: `≑`,
+		edot: `ė`,
+		ee: `ⅇ`,
+		efDot: `≒`,
+		Efr: `𝔈`,
+		efr: `𝔢`,
+		eg: `⪚`,
+		Egrave: `È`,
+		egrave: `è`,
+		egs: `⪖`,
+		egsdot: `⪘`,
+		el: `⪙`,
+		Element: `∈`,
+		elinters: `⏧`,
+		ell: `ℓ`,
+		els: `⪕`,
+		elsdot: `⪗`,
+		Emacr: `Ē`,
+		emacr: `ē`,
+		empty: `∅`,
+		emptyset: `∅`,
+		EmptySmallSquare: `◻`,
+		emptyv: `∅`,
+		EmptyVerySmallSquare: `▫`,
+		emsp: ` `,
+		emsp13: ` `,
+		emsp14: ` `,
+		ENG: `Ŋ`,
+		eng: `ŋ`,
+		ensp: ` `,
+		Eogon: `Ę`,
+		eogon: `ę`,
+		Eopf: `𝔼`,
+		eopf: `𝕖`,
+		epar: `⋕`,
+		eparsl: `⧣`,
+		eplus: `⩱`,
+		epsi: `ε`,
+		Epsilon: `Ε`,
+		epsilon: `ε`,
+		epsiv: `ϵ`,
+		eqcirc: `≖`,
+		eqcolon: `≕`,
+		eqsim: `≂`,
+		eqslantgtr: `⪖`,
+		eqslantless: `⪕`,
+		Equal: `⩵`,
+		equals: `=`,
+		EqualTilde: `≂`,
+		equest: `≟`,
+		Equilibrium: `⇌`,
+		equiv: `≡`,
+		equivDD: `⩸`,
+		eqvparsl: `⧥`,
+		erarr: `⥱`,
+		erDot: `≓`,
+		Escr: `ℰ`,
+		escr: `ℯ`,
+		esdot: `≐`,
+		Esim: `⩳`,
+		esim: `≂`,
+		Eta: `Η`,
+		eta: `η`,
+		ETH: `Ð`,
+		eth: `ð`,
+		Euml: `Ë`,
+		euml: `ë`,
+		euro: `€`,
+		excl: `!`,
+		exist: `∃`,
+		Exists: `∃`,
+		expectation: `ℰ`,
+		ExponentialE: `ⅇ`,
+		exponentiale: `ⅇ`,
+		fallingdotseq: `≒`,
+		Fcy: `Ф`,
+		fcy: `ф`,
+		female: `♀`,
+		ffilig: `ffi`,
+		fflig: `ff`,
+		ffllig: `ffl`,
+		Ffr: `𝔉`,
+		ffr: `𝔣`,
+		filig: `fi`,
+		FilledSmallSquare: `◼`,
+		FilledVerySmallSquare: `▪`,
+		fjlig: `fj`,
+		flat: `♭`,
+		fllig: `fl`,
+		fltns: `▱`,
+		fnof: `ƒ`,
+		Fopf: `𝔽`,
+		fopf: `𝕗`,
+		ForAll: `∀`,
+		forall: `∀`,
+		fork: `⋔`,
+		forkv: `⫙`,
+		Fouriertrf: `ℱ`,
+		fpartint: `⨍`,
+		frac12: `½`,
+		frac13: `⅓`,
+		frac14: `¼`,
+		frac15: `⅕`,
+		frac16: `⅙`,
+		frac18: `⅛`,
+		frac23: `⅔`,
+		frac25: `⅖`,
+		frac34: `¾`,
+		frac35: `⅗`,
+		frac38: `⅜`,
+		frac45: `⅘`,
+		frac56: `⅚`,
+		frac58: `⅝`,
+		frac78: `⅞`,
+		frasl: `⁄`,
+		frown: `⌢`,
+		Fscr: `ℱ`,
+		fscr: `𝒻`,
+		gacute: `ǵ`,
+		Gamma: `Γ`,
+		gamma: `γ`,
+		Gammad: `Ϝ`,
+		gammad: `ϝ`,
+		gap: `⪆`,
+		Gbreve: `Ğ`,
+		gbreve: `ğ`,
+		Gcedil: `Ģ`,
+		Gcirc: `Ĝ`,
+		gcirc: `ĝ`,
+		Gcy: `Г`,
+		gcy: `г`,
+		Gdot: `Ġ`,
+		gdot: `ġ`,
+		gE: `≧`,
+		ge: `≥`,
+		gEl: `⪌`,
+		gel: `⋛`,
+		geq: `≥`,
+		geqq: `≧`,
+		geqslant: `⩾`,
+		ges: `⩾`,
+		gescc: `⪩`,
+		gesdot: `⪀`,
+		gesdoto: `⪂`,
+		gesdotol: `⪄`,
+		gesl: `⋛︀`,
+		gesles: `⪔`,
+		Gfr: `𝔊`,
+		gfr: `𝔤`,
+		Gg: `⋙`,
+		gg: `≫`,
+		ggg: `⋙`,
+		gimel: `ℷ`,
+		GJcy: `Ѓ`,
+		gjcy: `ѓ`,
+		gl: `≷`,
+		gla: `⪥`,
+		glE: `⪒`,
+		glj: `⪤`,
+		gnap: `⪊`,
+		gnapprox: `⪊`,
+		gnE: `≩`,
+		gne: `⪈`,
+		gneq: `⪈`,
+		gneqq: `≩`,
+		gnsim: `⋧`,
+		Gopf: `𝔾`,
+		gopf: `𝕘`,
+		grave: "`",
+		GreaterEqual: `≥`,
+		GreaterEqualLess: `⋛`,
+		GreaterFullEqual: `≧`,
+		GreaterGreater: `⪢`,
+		GreaterLess: `≷`,
+		GreaterSlantEqual: `⩾`,
+		GreaterTilde: `≳`,
+		Gscr: `𝒢`,
+		gscr: `ℊ`,
+		gsim: `≳`,
+		gsime: `⪎`,
+		gsiml: `⪐`,
+		Gt: `≫`,
+		GT: `>`,
+		gt: `>`,
+		gtcc: `⪧`,
+		gtcir: `⩺`,
+		gtdot: `⋗`,
+		gtlPar: `⦕`,
+		gtquest: `⩼`,
+		gtrapprox: `⪆`,
+		gtrarr: `⥸`,
+		gtrdot: `⋗`,
+		gtreqless: `⋛`,
+		gtreqqless: `⪌`,
+		gtrless: `≷`,
+		gtrsim: `≳`,
+		gvertneqq: `≩︀`,
+		gvnE: `≩︀`,
+		Hacek: `ˇ`,
+		hairsp: ` `,
+		half: `½`,
+		hamilt: `ℋ`,
+		HARDcy: `Ъ`,
+		hardcy: `ъ`,
+		hArr: `⇔`,
+		harr: `↔`,
+		harrcir: `⥈`,
+		harrw: `↭`,
+		Hat: `^`,
+		hbar: `ℏ`,
+		Hcirc: `Ĥ`,
+		hcirc: `ĥ`,
+		hearts: `♥`,
+		heartsuit: `♥`,
+		hellip: `…`,
+		hercon: `⊹`,
+		Hfr: `ℌ`,
+		hfr: `𝔥`,
+		HilbertSpace: `ℋ`,
+		hksearow: `⤥`,
+		hkswarow: `⤦`,
+		hoarr: `⇿`,
+		homtht: `∻`,
+		hookleftarrow: `↩`,
+		hookrightarrow: `↪`,
+		Hopf: `ℍ`,
+		hopf: `𝕙`,
+		horbar: `―`,
+		HorizontalLine: `─`,
+		Hscr: `ℋ`,
+		hscr: `𝒽`,
+		hslash: `ℏ`,
+		Hstrok: `Ħ`,
+		hstrok: `ħ`,
+		HumpDownHump: `≎`,
+		HumpEqual: `≏`,
+		hybull: `⁃`,
+		hyphen: `‐`,
+		Iacute: `Í`,
+		iacute: `í`,
+		ic: `⁣`,
+		Icirc: `Î`,
+		icirc: `î`,
+		Icy: `И`,
+		icy: `и`,
+		Idot: `İ`,
+		IEcy: `Е`,
+		iecy: `е`,
+		iexcl: `¡`,
+		iff: `⇔`,
+		Ifr: `ℑ`,
+		ifr: `𝔦`,
+		Igrave: `Ì`,
+		igrave: `ì`,
+		ii: `ⅈ`,
+		iiiint: `⨌`,
+		iiint: `∭`,
+		iinfin: `⧜`,
+		iiota: `℩`,
+		IJlig: `IJ`,
+		ijlig: `ij`,
+		Im: `ℑ`,
+		Imacr: `Ī`,
+		imacr: `ī`,
+		image: `ℑ`,
+		ImaginaryI: `ⅈ`,
+		imagline: `ℐ`,
+		imagpart: `ℑ`,
+		imath: `ı`,
+		imof: `⊷`,
+		imped: `Ƶ`,
+		Implies: `⇒`,
+		in: `∈`,
+		incare: `℅`,
+		infin: `∞`,
+		infintie: `⧝`,
+		inodot: `ı`,
+		Int: `∬`,
+		int: `∫`,
+		intcal: `⊺`,
+		integers: `ℤ`,
+		Integral: `∫`,
+		intercal: `⊺`,
+		Intersection: `⋂`,
+		intlarhk: `⨗`,
+		intprod: `⨼`,
+		InvisibleComma: `⁣`,
+		InvisibleTimes: `⁢`,
+		IOcy: `Ё`,
+		iocy: `ё`,
+		Iogon: `Į`,
+		iogon: `į`,
+		Iopf: `𝕀`,
+		iopf: `𝕚`,
+		Iota: `Ι`,
+		iota: `ι`,
+		iprod: `⨼`,
+		iquest: `¿`,
+		Iscr: `ℐ`,
+		iscr: `𝒾`,
+		isin: `∈`,
+		isindot: `⋵`,
+		isinE: `⋹`,
+		isins: `⋴`,
+		isinsv: `⋳`,
+		isinv: `∈`,
+		it: `⁢`,
+		Itilde: `Ĩ`,
+		itilde: `ĩ`,
+		Iukcy: `І`,
+		iukcy: `і`,
+		Iuml: `Ï`,
+		iuml: `ï`,
+		Jcirc: `Ĵ`,
+		jcirc: `ĵ`,
+		Jcy: `Й`,
+		jcy: `й`,
+		Jfr: `𝔍`,
+		jfr: `𝔧`,
+		jmath: `ȷ`,
+		Jopf: `𝕁`,
+		jopf: `𝕛`,
+		Jscr: `𝒥`,
+		jscr: `𝒿`,
+		Jsercy: `Ј`,
+		jsercy: `ј`,
+		Jukcy: `Є`,
+		jukcy: `є`,
+		Kappa: `Κ`,
+		kappa: `κ`,
+		kappav: `ϰ`,
+		Kcedil: `Ķ`,
+		kcedil: `ķ`,
+		Kcy: `К`,
+		kcy: `к`,
+		Kfr: `𝔎`,
+		kfr: `𝔨`,
+		kgreen: `ĸ`,
+		KHcy: `Х`,
+		khcy: `х`,
+		KJcy: `Ќ`,
+		kjcy: `ќ`,
+		Kopf: `𝕂`,
+		kopf: `𝕜`,
+		Kscr: `𝒦`,
+		kscr: `𝓀`,
+		lAarr: `⇚`,
+		Lacute: `Ĺ`,
+		lacute: `ĺ`,
+		laemptyv: `⦴`,
+		lagran: `ℒ`,
+		Lambda: `Λ`,
+		lambda: `λ`,
+		Lang: `⟪`,
+		lang: `⟨`,
+		langd: `⦑`,
+		langle: `⟨`,
+		lap: `⪅`,
+		Laplacetrf: `ℒ`,
+		laquo: `«`,
+		Larr: `↞`,
+		lArr: `⇐`,
+		larr: `←`,
+		larrb: `⇤`,
+		larrbfs: `⤟`,
+		larrfs: `⤝`,
+		larrhk: `↩`,
+		larrlp: `↫`,
+		larrpl: `⤹`,
+		larrsim: `⥳`,
+		larrtl: `↢`,
+		lat: `⪫`,
+		lAtail: `⤛`,
+		latail: `⤙`,
+		late: `⪭`,
+		lates: `⪭︀`,
+		lBarr: `⤎`,
+		lbarr: `⤌`,
+		lbbrk: `❲`,
+		lbrace: `{`,
+		lbrack: `[`,
+		lbrke: `⦋`,
+		lbrksld: `⦏`,
+		lbrkslu: `⦍`,
+		Lcaron: `Ľ`,
+		lcaron: `ľ`,
+		Lcedil: `Ļ`,
+		lcedil: `ļ`,
+		lceil: `⌈`,
+		lcub: `{`,
+		Lcy: `Л`,
+		lcy: `л`,
+		ldca: `⤶`,
+		ldquo: `“`,
+		ldquor: `„`,
+		ldrdhar: `⥧`,
+		ldrushar: `⥋`,
+		ldsh: `↲`,
+		lE: `≦`,
+		le: `≤`,
+		LeftAngleBracket: `⟨`,
+		LeftArrow: `←`,
+		Leftarrow: `⇐`,
+		leftarrow: `←`,
+		LeftArrowBar: `⇤`,
+		LeftArrowRightArrow: `⇆`,
+		leftarrowtail: `↢`,
+		LeftCeiling: `⌈`,
+		LeftDoubleBracket: `⟦`,
+		LeftDownTeeVector: `⥡`,
+		LeftDownVector: `⇃`,
+		LeftDownVectorBar: `⥙`,
+		LeftFloor: `⌊`,
+		leftharpoondown: `↽`,
+		leftharpoonup: `↼`,
+		leftleftarrows: `⇇`,
+		LeftRightArrow: `↔`,
+		Leftrightarrow: `⇔`,
+		leftrightarrow: `↔`,
+		leftrightarrows: `⇆`,
+		leftrightharpoons: `⇋`,
+		leftrightsquigarrow: `↭`,
+		LeftRightVector: `⥎`,
+		LeftTee: `⊣`,
+		LeftTeeArrow: `↤`,
+		LeftTeeVector: `⥚`,
+		leftthreetimes: `⋋`,
+		LeftTriangle: `⊲`,
+		LeftTriangleBar: `⧏`,
+		LeftTriangleEqual: `⊴`,
+		LeftUpDownVector: `⥑`,
+		LeftUpTeeVector: `⥠`,
+		LeftUpVector: `↿`,
+		LeftUpVectorBar: `⥘`,
+		LeftVector: `↼`,
+		LeftVectorBar: `⥒`,
+		lEg: `⪋`,
+		leg: `⋚`,
+		leq: `≤`,
+		leqq: `≦`,
+		leqslant: `⩽`,
+		les: `⩽`,
+		lescc: `⪨`,
+		lesdot: `⩿`,
+		lesdoto: `⪁`,
+		lesdotor: `⪃`,
+		lesg: `⋚︀`,
+		lesges: `⪓`,
+		lessapprox: `⪅`,
+		lessdot: `⋖`,
+		lesseqgtr: `⋚`,
+		lesseqqgtr: `⪋`,
+		LessEqualGreater: `⋚`,
+		LessFullEqual: `≦`,
+		LessGreater: `≶`,
+		lessgtr: `≶`,
+		LessLess: `⪡`,
+		lesssim: `≲`,
+		LessSlantEqual: `⩽`,
+		LessTilde: `≲`,
+		lfisht: `⥼`,
+		lfloor: `⌊`,
+		Lfr: `𝔏`,
+		lfr: `𝔩`,
+		lg: `≶`,
+		lgE: `⪑`,
+		lHar: `⥢`,
+		lhard: `↽`,
+		lharu: `↼`,
+		lharul: `⥪`,
+		lhblk: `▄`,
+		LJcy: `Љ`,
+		ljcy: `љ`,
+		Ll: `⋘`,
+		ll: `≪`,
+		llarr: `⇇`,
+		llcorner: `⌞`,
+		Lleftarrow: `⇚`,
+		llhard: `⥫`,
+		lltri: `◺`,
+		Lmidot: `Ŀ`,
+		lmidot: `ŀ`,
+		lmoust: `⎰`,
+		lmoustache: `⎰`,
+		lnap: `⪉`,
+		lnapprox: `⪉`,
+		lnE: `≨`,
+		lne: `⪇`,
+		lneq: `⪇`,
+		lneqq: `≨`,
+		lnsim: `⋦`,
+		loang: `⟬`,
+		loarr: `⇽`,
+		lobrk: `⟦`,
+		LongLeftArrow: `⟵`,
+		Longleftarrow: `⟸`,
+		longleftarrow: `⟵`,
+		LongLeftRightArrow: `⟷`,
+		Longleftrightarrow: `⟺`,
+		longleftrightarrow: `⟷`,
+		longmapsto: `⟼`,
+		LongRightArrow: `⟶`,
+		Longrightarrow: `⟹`,
+		longrightarrow: `⟶`,
+		looparrowleft: `↫`,
+		looparrowright: `↬`,
+		lopar: `⦅`,
+		Lopf: `𝕃`,
+		lopf: `𝕝`,
+		loplus: `⨭`,
+		lotimes: `⨴`,
+		lowast: `∗`,
+		lowbar: `_`,
+		LowerLeftArrow: `↙`,
+		LowerRightArrow: `↘`,
+		loz: `◊`,
+		lozenge: `◊`,
+		lozf: `⧫`,
+		lpar: `(`,
+		lparlt: `⦓`,
+		lrarr: `⇆`,
+		lrcorner: `⌟`,
+		lrhar: `⇋`,
+		lrhard: `⥭`,
+		lrm: `‎`,
+		lrtri: `⊿`,
+		lsaquo: `‹`,
+		Lscr: `ℒ`,
+		lscr: `𝓁`,
+		Lsh: `↰`,
+		lsh: `↰`,
+		lsim: `≲`,
+		lsime: `⪍`,
+		lsimg: `⪏`,
+		lsqb: `[`,
+		lsquo: `‘`,
+		lsquor: `‚`,
+		Lstrok: `Ł`,
+		lstrok: `ł`,
+		Lt: `≪`,
+		LT: `<`,
+		lt: `<`,
+		ltcc: `⪦`,
+		ltcir: `⩹`,
+		ltdot: `⋖`,
+		lthree: `⋋`,
+		ltimes: `⋉`,
+		ltlarr: `⥶`,
+		ltquest: `⩻`,
+		ltri: `◃`,
+		ltrie: `⊴`,
+		ltrif: `◂`,
+		ltrPar: `⦖`,
+		lurdshar: `⥊`,
+		luruhar: `⥦`,
+		lvertneqq: `≨︀`,
+		lvnE: `≨︀`,
+		macr: `¯`,
+		male: `♂`,
+		malt: `✠`,
+		maltese: `✠`,
+		Map: `⤅`,
+		map: `↦`,
+		mapsto: `↦`,
+		mapstodown: `↧`,
+		mapstoleft: `↤`,
+		mapstoup: `↥`,
+		marker: `▮`,
+		mcomma: `⨩`,
+		Mcy: `М`,
+		mcy: `м`,
+		mdash: `—`,
+		mDDot: `∺`,
+		measuredangle: `∡`,
+		MediumSpace: ` `,
+		Mellintrf: `ℳ`,
+		Mfr: `𝔐`,
+		mfr: `𝔪`,
+		mho: `℧`,
+		micro: `µ`,
+		mid: `∣`,
+		midast: `*`,
+		midcir: `⫰`,
+		middot: `·`,
+		minus: `−`,
+		minusb: `⊟`,
+		minusd: `∸`,
+		minusdu: `⨪`,
+		MinusPlus: `∓`,
+		mlcp: `⫛`,
+		mldr: `…`,
+		mnplus: `∓`,
+		models: `⊧`,
+		Mopf: `𝕄`,
+		mopf: `𝕞`,
+		mp: `∓`,
+		Mscr: `ℳ`,
+		mscr: `𝓂`,
+		mstpos: `∾`,
+		Mu: `Μ`,
+		mu: `μ`,
+		multimap: `⊸`,
+		mumap: `⊸`,
+		nabla: `∇`,
+		Nacute: `Ń`,
+		nacute: `ń`,
+		nang: `∠⃒`,
+		nap: `≉`,
+		napE: `⩰̸`,
+		napid: `≋̸`,
+		napos: `ʼn`,
+		napprox: `≉`,
+		natur: `♮`,
+		natural: `♮`,
+		naturals: `ℕ`,
+		nbsp: `\xA0`,
+		nbump: `≎̸`,
+		nbumpe: `≏̸`,
+		ncap: `⩃`,
+		Ncaron: `Ň`,
+		ncaron: `ň`,
+		Ncedil: `Ņ`,
+		ncedil: `ņ`,
+		ncong: `≇`,
+		ncongdot: `⩭̸`,
+		ncup: `⩂`,
+		Ncy: `Н`,
+		ncy: `н`,
+		ndash: `–`,
+		ne: `≠`,
+		nearhk: `⤤`,
+		neArr: `⇗`,
+		nearr: `↗`,
+		nearrow: `↗`,
+		nedot: `≐̸`,
+		NegativeMediumSpace: `​`,
+		NegativeThickSpace: `​`,
+		NegativeThinSpace: `​`,
+		NegativeVeryThinSpace: `​`,
+		nequiv: `≢`,
+		nesear: `⤨`,
+		nesim: `≂̸`,
+		NestedGreaterGreater: `≫`,
+		NestedLessLess: `≪`,
+		NewLine: `
+`,
+		nexist: `∄`,
+		nexists: `∄`,
+		Nfr: `𝔑`,
+		nfr: `𝔫`,
+		ngE: `≧̸`,
+		nge: `≱`,
+		ngeq: `≱`,
+		ngeqq: `≧̸`,
+		ngeqslant: `⩾̸`,
+		nges: `⩾̸`,
+		nGg: `⋙̸`,
+		ngsim: `≵`,
+		nGt: `≫⃒`,
+		ngt: `≯`,
+		ngtr: `≯`,
+		nGtv: `≫̸`,
+		nhArr: `⇎`,
+		nharr: `↮`,
+		nhpar: `⫲`,
+		ni: `∋`,
+		nis: `⋼`,
+		nisd: `⋺`,
+		niv: `∋`,
+		NJcy: `Њ`,
+		njcy: `њ`,
+		nlArr: `⇍`,
+		nlarr: `↚`,
+		nldr: `‥`,
+		nlE: `≦̸`,
+		nle: `≰`,
+		nLeftarrow: `⇍`,
+		nleftarrow: `↚`,
+		nLeftrightarrow: `⇎`,
+		nleftrightarrow: `↮`,
+		nleq: `≰`,
+		nleqq: `≦̸`,
+		nleqslant: `⩽̸`,
+		nles: `⩽̸`,
+		nless: `≮`,
+		nLl: `⋘̸`,
+		nlsim: `≴`,
+		nLt: `≪⃒`,
+		nlt: `≮`,
+		nltri: `⋪`,
+		nltrie: `⋬`,
+		nLtv: `≪̸`,
+		nmid: `∤`,
+		NoBreak: `⁠`,
+		NonBreakingSpace: `\xA0`,
+		Nopf: `ℕ`,
+		nopf: `𝕟`,
+		Not: `⫬`,
+		not: `¬`,
+		NotCongruent: `≢`,
+		NotCupCap: `≭`,
+		NotDoubleVerticalBar: `∦`,
+		NotElement: `∉`,
+		NotEqual: `≠`,
+		NotEqualTilde: `≂̸`,
+		NotExists: `∄`,
+		NotGreater: `≯`,
+		NotGreaterEqual: `≱`,
+		NotGreaterFullEqual: `≧̸`,
+		NotGreaterGreater: `≫̸`,
+		NotGreaterLess: `≹`,
+		NotGreaterSlantEqual: `⩾̸`,
+		NotGreaterTilde: `≵`,
+		NotHumpDownHump: `≎̸`,
+		NotHumpEqual: `≏̸`,
+		notin: `∉`,
+		notindot: `⋵̸`,
+		notinE: `⋹̸`,
+		notinva: `∉`,
+		notinvb: `⋷`,
+		notinvc: `⋶`,
+		NotLeftTriangle: `⋪`,
+		NotLeftTriangleBar: `⧏̸`,
+		NotLeftTriangleEqual: `⋬`,
+		NotLess: `≮`,
+		NotLessEqual: `≰`,
+		NotLessGreater: `≸`,
+		NotLessLess: `≪̸`,
+		NotLessSlantEqual: `⩽̸`,
+		NotLessTilde: `≴`,
+		NotNestedGreaterGreater: `⪢̸`,
+		NotNestedLessLess: `⪡̸`,
+		notni: `∌`,
+		notniva: `∌`,
+		notnivb: `⋾`,
+		notnivc: `⋽`,
+		NotPrecedes: `⊀`,
+		NotPrecedesEqual: `⪯̸`,
+		NotPrecedesSlantEqual: `⋠`,
+		NotReverseElement: `∌`,
+		NotRightTriangle: `⋫`,
+		NotRightTriangleBar: `⧐̸`,
+		NotRightTriangleEqual: `⋭`,
+		NotSquareSubset: `⊏̸`,
+		NotSquareSubsetEqual: `⋢`,
+		NotSquareSuperset: `⊐̸`,
+		NotSquareSupersetEqual: `⋣`,
+		NotSubset: `⊂⃒`,
+		NotSubsetEqual: `⊈`,
+		NotSucceeds: `⊁`,
+		NotSucceedsEqual: `⪰̸`,
+		NotSucceedsSlantEqual: `⋡`,
+		NotSucceedsTilde: `≿̸`,
+		NotSuperset: `⊃⃒`,
+		NotSupersetEqual: `⊉`,
+		NotTilde: `≁`,
+		NotTildeEqual: `≄`,
+		NotTildeFullEqual: `≇`,
+		NotTildeTilde: `≉`,
+		NotVerticalBar: `∤`,
+		npar: `∦`,
+		nparallel: `∦`,
+		nparsl: `⫽⃥`,
+		npart: `∂̸`,
+		npolint: `⨔`,
+		npr: `⊀`,
+		nprcue: `⋠`,
+		npre: `⪯̸`,
+		nprec: `⊀`,
+		npreceq: `⪯̸`,
+		nrArr: `⇏`,
+		nrarr: `↛`,
+		nrarrc: `⤳̸`,
+		nrarrw: `↝̸`,
+		nRightarrow: `⇏`,
+		nrightarrow: `↛`,
+		nrtri: `⋫`,
+		nrtrie: `⋭`,
+		nsc: `⊁`,
+		nsccue: `⋡`,
+		nsce: `⪰̸`,
+		Nscr: `𝒩`,
+		nscr: `𝓃`,
+		nshortmid: `∤`,
+		nshortparallel: `∦`,
+		nsim: `≁`,
+		nsime: `≄`,
+		nsimeq: `≄`,
+		nsmid: `∤`,
+		nspar: `∦`,
+		nsqsube: `⋢`,
+		nsqsupe: `⋣`,
+		nsub: `⊄`,
+		nsubE: `⫅̸`,
+		nsube: `⊈`,
+		nsubset: `⊂⃒`,
+		nsubseteq: `⊈`,
+		nsubseteqq: `⫅̸`,
+		nsucc: `⊁`,
+		nsucceq: `⪰̸`,
+		nsup: `⊅`,
+		nsupE: `⫆̸`,
+		nsupe: `⊉`,
+		nsupset: `⊃⃒`,
+		nsupseteq: `⊉`,
+		nsupseteqq: `⫆̸`,
+		ntgl: `≹`,
+		Ntilde: `Ñ`,
+		ntilde: `ñ`,
+		ntlg: `≸`,
+		ntriangleleft: `⋪`,
+		ntrianglelefteq: `⋬`,
+		ntriangleright: `⋫`,
+		ntrianglerighteq: `⋭`,
+		Nu: `Ν`,
+		nu: `ν`,
+		num: `#`,
+		numero: `№`,
+		numsp: ` `,
+		nvap: `≍⃒`,
+		nVDash: `⊯`,
+		nVdash: `⊮`,
+		nvDash: `⊭`,
+		nvdash: `⊬`,
+		nvge: `≥⃒`,
+		nvgt: `>⃒`,
+		nvHarr: `⤄`,
+		nvinfin: `⧞`,
+		nvlArr: `⤂`,
+		nvle: `≤⃒`,
+		nvlt: `<⃒`,
+		nvltrie: `⊴⃒`,
+		nvrArr: `⤃`,
+		nvrtrie: `⊵⃒`,
+		nvsim: `∼⃒`,
+		nwarhk: `⤣`,
+		nwArr: `⇖`,
+		nwarr: `↖`,
+		nwarrow: `↖`,
+		nwnear: `⤧`,
+		Oacute: `Ó`,
+		oacute: `ó`,
+		oast: `⊛`,
+		ocir: `⊚`,
+		Ocirc: `Ô`,
+		ocirc: `ô`,
+		Ocy: `О`,
+		ocy: `о`,
+		odash: `⊝`,
+		Odblac: `Ő`,
+		odblac: `ő`,
+		odiv: `⨸`,
+		odot: `⊙`,
+		odsold: `⦼`,
+		OElig: `Œ`,
+		oelig: `œ`,
+		ofcir: `⦿`,
+		Ofr: `𝔒`,
+		ofr: `𝔬`,
+		ogon: `˛`,
+		Ograve: `Ò`,
+		ograve: `ò`,
+		ogt: `⧁`,
+		ohbar: `⦵`,
+		ohm: `Ω`,
+		oint: `∮`,
+		olarr: `↺`,
+		olcir: `⦾`,
+		olcross: `⦻`,
+		oline: `‾`,
+		olt: `⧀`,
+		Omacr: `Ō`,
+		omacr: `ō`,
+		Omega: `Ω`,
+		omega: `ω`,
+		Omicron: `Ο`,
+		omicron: `ο`,
+		omid: `⦶`,
+		ominus: `⊖`,
+		Oopf: `𝕆`,
+		oopf: `𝕠`,
+		opar: `⦷`,
+		OpenCurlyDoubleQuote: `“`,
+		OpenCurlyQuote: `‘`,
+		operp: `⦹`,
+		oplus: `⊕`,
+		Or: `⩔`,
+		or: `∨`,
+		orarr: `↻`,
+		ord: `⩝`,
+		order: `ℴ`,
+		orderof: `ℴ`,
+		ordf: `ª`,
+		ordm: `º`,
+		origof: `⊶`,
+		oror: `⩖`,
+		orslope: `⩗`,
+		orv: `⩛`,
+		oS: `Ⓢ`,
+		Oscr: `𝒪`,
+		oscr: `ℴ`,
+		Oslash: `Ø`,
+		oslash: `ø`,
+		osol: `⊘`,
+		Otilde: `Õ`,
+		otilde: `õ`,
+		Otimes: `⨷`,
+		otimes: `⊗`,
+		otimesas: `⨶`,
+		Ouml: `Ö`,
+		ouml: `ö`,
+		ovbar: `⌽`,
+		OverBar: `‾`,
+		OverBrace: `⏞`,
+		OverBracket: `⎴`,
+		OverParenthesis: `⏜`,
+		par: `∥`,
+		para: `¶`,
+		parallel: `∥`,
+		parsim: `⫳`,
+		parsl: `⫽`,
+		part: `∂`,
+		PartialD: `∂`,
+		Pcy: `П`,
+		pcy: `п`,
+		percnt: `%`,
+		period: `.`,
+		permil: `‰`,
+		perp: `⊥`,
+		pertenk: `‱`,
+		Pfr: `𝔓`,
+		pfr: `𝔭`,
+		Phi: `Φ`,
+		phi: `φ`,
+		phiv: `ϕ`,
+		phmmat: `ℳ`,
+		phone: `☎`,
+		Pi: `Π`,
+		pi: `π`,
+		pitchfork: `⋔`,
+		piv: `ϖ`,
+		planck: `ℏ`,
+		planckh: `ℎ`,
+		plankv: `ℏ`,
+		plus: `+`,
+		plusacir: `⨣`,
+		plusb: `⊞`,
+		pluscir: `⨢`,
+		plusdo: `∔`,
+		plusdu: `⨥`,
+		pluse: `⩲`,
+		PlusMinus: `±`,
+		plusmn: `±`,
+		plussim: `⨦`,
+		plustwo: `⨧`,
+		pm: `±`,
+		Poincareplane: `ℌ`,
+		pointint: `⨕`,
+		Popf: `ℙ`,
+		popf: `𝕡`,
+		pound: `£`,
+		Pr: `⪻`,
+		pr: `≺`,
+		prap: `⪷`,
+		prcue: `≼`,
+		prE: `⪳`,
+		pre: `⪯`,
+		prec: `≺`,
+		precapprox: `⪷`,
+		preccurlyeq: `≼`,
+		Precedes: `≺`,
+		PrecedesEqual: `⪯`,
+		PrecedesSlantEqual: `≼`,
+		PrecedesTilde: `≾`,
+		preceq: `⪯`,
+		precnapprox: `⪹`,
+		precneqq: `⪵`,
+		precnsim: `⋨`,
+		precsim: `≾`,
+		Prime: `″`,
+		prime: `′`,
+		primes: `ℙ`,
+		prnap: `⪹`,
+		prnE: `⪵`,
+		prnsim: `⋨`,
+		prod: `∏`,
+		Product: `∏`,
+		profalar: `⌮`,
+		profline: `⌒`,
+		profsurf: `⌓`,
+		prop: `∝`,
+		Proportion: `∷`,
+		Proportional: `∝`,
+		propto: `∝`,
+		prsim: `≾`,
+		prurel: `⊰`,
+		Pscr: `𝒫`,
+		pscr: `𝓅`,
+		Psi: `Ψ`,
+		psi: `ψ`,
+		puncsp: ` `,
+		Qfr: `𝔔`,
+		qfr: `𝔮`,
+		qint: `⨌`,
+		Qopf: `ℚ`,
+		qopf: `𝕢`,
+		qprime: `⁗`,
+		Qscr: `𝒬`,
+		qscr: `𝓆`,
+		quaternions: `ℍ`,
+		quatint: `⨖`,
+		quest: `?`,
+		questeq: `≟`,
+		QUOT: `"`,
+		quot: `"`,
+		rAarr: `⇛`,
+		race: `∽̱`,
+		Racute: `Ŕ`,
+		racute: `ŕ`,
+		radic: `√`,
+		raemptyv: `⦳`,
+		Rang: `⟫`,
+		rang: `⟩`,
+		rangd: `⦒`,
+		range: `⦥`,
+		rangle: `⟩`,
+		raquo: `»`,
+		Rarr: `↠`,
+		rArr: `⇒`,
+		rarr: `→`,
+		rarrap: `⥵`,
+		rarrb: `⇥`,
+		rarrbfs: `⤠`,
+		rarrc: `⤳`,
+		rarrfs: `⤞`,
+		rarrhk: `↪`,
+		rarrlp: `↬`,
+		rarrpl: `⥅`,
+		rarrsim: `⥴`,
+		Rarrtl: `⤖`,
+		rarrtl: `↣`,
+		rarrw: `↝`,
+		rAtail: `⤜`,
+		ratail: `⤚`,
+		ratio: `∶`,
+		rationals: `ℚ`,
+		RBarr: `⤐`,
+		rBarr: `⤏`,
+		rbarr: `⤍`,
+		rbbrk: `❳`,
+		rbrace: `}`,
+		rbrack: `]`,
+		rbrke: `⦌`,
+		rbrksld: `⦎`,
+		rbrkslu: `⦐`,
+		Rcaron: `Ř`,
+		rcaron: `ř`,
+		Rcedil: `Ŗ`,
+		rcedil: `ŗ`,
+		rceil: `⌉`,
+		rcub: `}`,
+		Rcy: `Р`,
+		rcy: `р`,
+		rdca: `⤷`,
+		rdldhar: `⥩`,
+		rdquo: `”`,
+		rdquor: `”`,
+		rdsh: `↳`,
+		Re: `ℜ`,
+		real: `ℜ`,
+		realine: `ℛ`,
+		realpart: `ℜ`,
+		reals: `ℝ`,
+		rect: `▭`,
+		REG: `®`,
+		reg: `®`,
+		ReverseElement: `∋`,
+		ReverseEquilibrium: `⇋`,
+		ReverseUpEquilibrium: `⥯`,
+		rfisht: `⥽`,
+		rfloor: `⌋`,
+		Rfr: `ℜ`,
+		rfr: `𝔯`,
+		rHar: `⥤`,
+		rhard: `⇁`,
+		rharu: `⇀`,
+		rharul: `⥬`,
+		Rho: `Ρ`,
+		rho: `ρ`,
+		rhov: `ϱ`,
+		RightAngleBracket: `⟩`,
+		RightArrow: `→`,
+		Rightarrow: `⇒`,
+		rightarrow: `→`,
+		RightArrowBar: `⇥`,
+		RightArrowLeftArrow: `⇄`,
+		rightarrowtail: `↣`,
+		RightCeiling: `⌉`,
+		RightDoubleBracket: `⟧`,
+		RightDownTeeVector: `⥝`,
+		RightDownVector: `⇂`,
+		RightDownVectorBar: `⥕`,
+		RightFloor: `⌋`,
+		rightharpoondown: `⇁`,
+		rightharpoonup: `⇀`,
+		rightleftarrows: `⇄`,
+		rightleftharpoons: `⇌`,
+		rightrightarrows: `⇉`,
+		rightsquigarrow: `↝`,
+		RightTee: `⊢`,
+		RightTeeArrow: `↦`,
+		RightTeeVector: `⥛`,
+		rightthreetimes: `⋌`,
+		RightTriangle: `⊳`,
+		RightTriangleBar: `⧐`,
+		RightTriangleEqual: `⊵`,
+		RightUpDownVector: `⥏`,
+		RightUpTeeVector: `⥜`,
+		RightUpVector: `↾`,
+		RightUpVectorBar: `⥔`,
+		RightVector: `⇀`,
+		RightVectorBar: `⥓`,
+		ring: `˚`,
+		risingdotseq: `≓`,
+		rlarr: `⇄`,
+		rlhar: `⇌`,
+		rlm: `‏`,
+		rmoust: `⎱`,
+		rmoustache: `⎱`,
+		rnmid: `⫮`,
+		roang: `⟭`,
+		roarr: `⇾`,
+		robrk: `⟧`,
+		ropar: `⦆`,
+		Ropf: `ℝ`,
+		ropf: `𝕣`,
+		roplus: `⨮`,
+		rotimes: `⨵`,
+		RoundImplies: `⥰`,
+		rpar: `)`,
+		rpargt: `⦔`,
+		rppolint: `⨒`,
+		rrarr: `⇉`,
+		Rrightarrow: `⇛`,
+		rsaquo: `›`,
+		Rscr: `ℛ`,
+		rscr: `𝓇`,
+		Rsh: `↱`,
+		rsh: `↱`,
+		rsqb: `]`,
+		rsquo: `’`,
+		rsquor: `’`,
+		rthree: `⋌`,
+		rtimes: `⋊`,
+		rtri: `▹`,
+		rtrie: `⊵`,
+		rtrif: `▸`,
+		rtriltri: `⧎`,
+		RuleDelayed: `⧴`,
+		ruluhar: `⥨`,
+		rx: `℞`,
+		Sacute: `Ś`,
+		sacute: `ś`,
+		sbquo: `‚`,
+		Sc: `⪼`,
+		sc: `≻`,
+		scap: `⪸`,
+		Scaron: `Š`,
+		scaron: `š`,
+		sccue: `≽`,
+		scE: `⪴`,
+		sce: `⪰`,
+		Scedil: `Ş`,
+		scedil: `ş`,
+		Scirc: `Ŝ`,
+		scirc: `ŝ`,
+		scnap: `⪺`,
+		scnE: `⪶`,
+		scnsim: `⋩`,
+		scpolint: `⨓`,
+		scsim: `≿`,
+		Scy: `С`,
+		scy: `с`,
+		sdot: `⋅`,
+		sdotb: `⊡`,
+		sdote: `⩦`,
+		searhk: `⤥`,
+		seArr: `⇘`,
+		searr: `↘`,
+		searrow: `↘`,
+		sect: `§`,
+		semi: `;`,
+		seswar: `⤩`,
+		setminus: `∖`,
+		setmn: `∖`,
+		sext: `✶`,
+		Sfr: `𝔖`,
+		sfr: `𝔰`,
+		sfrown: `⌢`,
+		sharp: `♯`,
+		SHCHcy: `Щ`,
+		shchcy: `щ`,
+		SHcy: `Ш`,
+		shcy: `ш`,
+		ShortDownArrow: `↓`,
+		ShortLeftArrow: `←`,
+		shortmid: `∣`,
+		shortparallel: `∥`,
+		ShortRightArrow: `→`,
+		ShortUpArrow: `↑`,
+		shy: `­`,
+		Sigma: `Σ`,
+		sigma: `σ`,
+		sigmaf: `ς`,
+		sigmav: `ς`,
+		sim: `∼`,
+		simdot: `⩪`,
+		sime: `≃`,
+		simeq: `≃`,
+		simg: `⪞`,
+		simgE: `⪠`,
+		siml: `⪝`,
+		simlE: `⪟`,
+		simne: `≆`,
+		simplus: `⨤`,
+		simrarr: `⥲`,
+		slarr: `←`,
+		SmallCircle: `∘`,
+		smallsetminus: `∖`,
+		smashp: `⨳`,
+		smeparsl: `⧤`,
+		smid: `∣`,
+		smile: `⌣`,
+		smt: `⪪`,
+		smte: `⪬`,
+		smtes: `⪬︀`,
+		SOFTcy: `Ь`,
+		softcy: `ь`,
+		sol: `/`,
+		solb: `⧄`,
+		solbar: `⌿`,
+		Sopf: `𝕊`,
+		sopf: `𝕤`,
+		spades: `♠`,
+		spadesuit: `♠`,
+		spar: `∥`,
+		sqcap: `⊓`,
+		sqcaps: `⊓︀`,
+		sqcup: `⊔`,
+		sqcups: `⊔︀`,
+		Sqrt: `√`,
+		sqsub: `⊏`,
+		sqsube: `⊑`,
+		sqsubset: `⊏`,
+		sqsubseteq: `⊑`,
+		sqsup: `⊐`,
+		sqsupe: `⊒`,
+		sqsupset: `⊐`,
+		sqsupseteq: `⊒`,
+		squ: `□`,
+		Square: `□`,
+		square: `□`,
+		SquareIntersection: `⊓`,
+		SquareSubset: `⊏`,
+		SquareSubsetEqual: `⊑`,
+		SquareSuperset: `⊐`,
+		SquareSupersetEqual: `⊒`,
+		SquareUnion: `⊔`,
+		squarf: `▪`,
+		squf: `▪`,
+		srarr: `→`,
+		Sscr: `𝒮`,
+		sscr: `𝓈`,
+		ssetmn: `∖`,
+		ssmile: `⌣`,
+		sstarf: `⋆`,
+		Star: `⋆`,
+		star: `☆`,
+		starf: `★`,
+		straightepsilon: `ϵ`,
+		straightphi: `ϕ`,
+		strns: `¯`,
+		Sub: `⋐`,
+		sub: `⊂`,
+		subdot: `⪽`,
+		subE: `⫅`,
+		sube: `⊆`,
+		subedot: `⫃`,
+		submult: `⫁`,
+		subnE: `⫋`,
+		subne: `⊊`,
+		subplus: `⪿`,
+		subrarr: `⥹`,
+		Subset: `⋐`,
+		subset: `⊂`,
+		subseteq: `⊆`,
+		subseteqq: `⫅`,
+		SubsetEqual: `⊆`,
+		subsetneq: `⊊`,
+		subsetneqq: `⫋`,
+		subsim: `⫇`,
+		subsub: `⫕`,
+		subsup: `⫓`,
+		succ: `≻`,
+		succapprox: `⪸`,
+		succcurlyeq: `≽`,
+		Succeeds: `≻`,
+		SucceedsEqual: `⪰`,
+		SucceedsSlantEqual: `≽`,
+		SucceedsTilde: `≿`,
+		succeq: `⪰`,
+		succnapprox: `⪺`,
+		succneqq: `⪶`,
+		succnsim: `⋩`,
+		succsim: `≿`,
+		SuchThat: `∋`,
+		Sum: `∑`,
+		sum: `∑`,
+		sung: `♪`,
+		Sup: `⋑`,
+		sup: `⊃`,
+		sup1: `¹`,
+		sup2: `²`,
+		sup3: `³`,
+		supdot: `⪾`,
+		supdsub: `⫘`,
+		supE: `⫆`,
+		supe: `⊇`,
+		supedot: `⫄`,
+		Superset: `⊃`,
+		SupersetEqual: `⊇`,
+		suphsol: `⟉`,
+		suphsub: `⫗`,
+		suplarr: `⥻`,
+		supmult: `⫂`,
+		supnE: `⫌`,
+		supne: `⊋`,
+		supplus: `⫀`,
+		Supset: `⋑`,
+		supset: `⊃`,
+		supseteq: `⊇`,
+		supseteqq: `⫆`,
+		supsetneq: `⊋`,
+		supsetneqq: `⫌`,
+		supsim: `⫈`,
+		supsub: `⫔`,
+		supsup: `⫖`,
+		swarhk: `⤦`,
+		swArr: `⇙`,
+		swarr: `↙`,
+		swarrow: `↙`,
+		swnwar: `⤪`,
+		szlig: `ß`,
+		Tab: `	`,
+		target: `⌖`,
+		Tau: `Τ`,
+		tau: `τ`,
+		tbrk: `⎴`,
+		Tcaron: `Ť`,
+		tcaron: `ť`,
+		Tcedil: `Ţ`,
+		tcedil: `ţ`,
+		Tcy: `Т`,
+		tcy: `т`,
+		tdot: `⃛`,
+		telrec: `⌕`,
+		Tfr: `𝔗`,
+		tfr: `𝔱`,
+		there4: `∴`,
+		Therefore: `∴`,
+		therefore: `∴`,
+		Theta: `Θ`,
+		theta: `θ`,
+		thetasym: `ϑ`,
+		thetav: `ϑ`,
+		thickapprox: `≈`,
+		thicksim: `∼`,
+		ThickSpace: `  `,
+		thinsp: ` `,
+		ThinSpace: ` `,
+		thkap: `≈`,
+		thksim: `∼`,
+		THORN: `Þ`,
+		thorn: `þ`,
+		Tilde: `∼`,
+		tilde: `˜`,
+		TildeEqual: `≃`,
+		TildeFullEqual: `≅`,
+		TildeTilde: `≈`,
+		times: `×`,
+		timesb: `⊠`,
+		timesbar: `⨱`,
+		timesd: `⨰`,
+		tint: `∭`,
+		toea: `⤨`,
+		top: `⊤`,
+		topbot: `⌶`,
+		topcir: `⫱`,
+		Topf: `𝕋`,
+		topf: `𝕥`,
+		topfork: `⫚`,
+		tosa: `⤩`,
+		tprime: `‴`,
+		TRADE: `™`,
+		trade: `™`,
+		triangle: `▵`,
+		triangledown: `▿`,
+		triangleleft: `◃`,
+		trianglelefteq: `⊴`,
+		triangleq: `≜`,
+		triangleright: `▹`,
+		trianglerighteq: `⊵`,
+		tridot: `◬`,
+		trie: `≜`,
+		triminus: `⨺`,
+		TripleDot: `⃛`,
+		triplus: `⨹`,
+		trisb: `⧍`,
+		tritime: `⨻`,
+		trpezium: `⏢`,
+		Tscr: `𝒯`,
+		tscr: `𝓉`,
+		TScy: `Ц`,
+		tscy: `ц`,
+		TSHcy: `Ћ`,
+		tshcy: `ћ`,
+		Tstrok: `Ŧ`,
+		tstrok: `ŧ`,
+		twixt: `≬`,
+		twoheadleftarrow: `↞`,
+		twoheadrightarrow: `↠`,
+		Uacute: `Ú`,
+		uacute: `ú`,
+		Uarr: `↟`,
+		uArr: `⇑`,
+		uarr: `↑`,
+		Uarrocir: `⥉`,
+		Ubrcy: `Ў`,
+		ubrcy: `ў`,
+		Ubreve: `Ŭ`,
+		ubreve: `ŭ`,
+		Ucirc: `Û`,
+		ucirc: `û`,
+		Ucy: `У`,
+		ucy: `у`,
+		udarr: `⇅`,
+		Udblac: `Ű`,
+		udblac: `ű`,
+		udhar: `⥮`,
+		ufisht: `⥾`,
+		Ufr: `𝔘`,
+		ufr: `𝔲`,
+		Ugrave: `Ù`,
+		ugrave: `ù`,
+		uHar: `⥣`,
+		uharl: `↿`,
+		uharr: `↾`,
+		uhblk: `▀`,
+		ulcorn: `⌜`,
+		ulcorner: `⌜`,
+		ulcrop: `⌏`,
+		ultri: `◸`,
+		Umacr: `Ū`,
+		umacr: `ū`,
+		uml: `¨`,
+		UnderBar: `_`,
+		UnderBrace: `⏟`,
+		UnderBracket: `⎵`,
+		UnderParenthesis: `⏝`,
+		Union: `⋃`,
+		UnionPlus: `⊎`,
+		Uogon: `Ų`,
+		uogon: `ų`,
+		Uopf: `𝕌`,
+		uopf: `𝕦`,
+		UpArrow: `↑`,
+		Uparrow: `⇑`,
+		uparrow: `↑`,
+		UpArrowBar: `⤒`,
+		UpArrowDownArrow: `⇅`,
+		UpDownArrow: `↕`,
+		Updownarrow: `⇕`,
+		updownarrow: `↕`,
+		UpEquilibrium: `⥮`,
+		upharpoonleft: `↿`,
+		upharpoonright: `↾`,
+		uplus: `⊎`,
+		UpperLeftArrow: `↖`,
+		UpperRightArrow: `↗`,
+		Upsi: `ϒ`,
+		upsi: `υ`,
+		upsih: `ϒ`,
+		Upsilon: `Υ`,
+		upsilon: `υ`,
+		UpTee: `⊥`,
+		UpTeeArrow: `↥`,
+		upuparrows: `⇈`,
+		urcorn: `⌝`,
+		urcorner: `⌝`,
+		urcrop: `⌎`,
+		Uring: `Ů`,
+		uring: `ů`,
+		urtri: `◹`,
+		Uscr: `𝒰`,
+		uscr: `𝓊`,
+		utdot: `⋰`,
+		Utilde: `Ũ`,
+		utilde: `ũ`,
+		utri: `▵`,
+		utrif: `▴`,
+		uuarr: `⇈`,
+		Uuml: `Ü`,
+		uuml: `ü`,
+		uwangle: `⦧`,
+		vangrt: `⦜`,
+		varepsilon: `ϵ`,
+		varkappa: `ϰ`,
+		varnothing: `∅`,
+		varphi: `ϕ`,
+		varpi: `ϖ`,
+		varpropto: `∝`,
+		vArr: `⇕`,
+		varr: `↕`,
+		varrho: `ϱ`,
+		varsigma: `ς`,
+		varsubsetneq: `⊊︀`,
+		varsubsetneqq: `⫋︀`,
+		varsupsetneq: `⊋︀`,
+		varsupsetneqq: `⫌︀`,
+		vartheta: `ϑ`,
+		vartriangleleft: `⊲`,
+		vartriangleright: `⊳`,
+		Vbar: `⫫`,
+		vBar: `⫨`,
+		vBarv: `⫩`,
+		Vcy: `В`,
+		vcy: `в`,
+		VDash: `⊫`,
+		Vdash: `⊩`,
+		vDash: `⊨`,
+		vdash: `⊢`,
+		Vdashl: `⫦`,
+		Vee: `⋁`,
+		vee: `∨`,
+		veebar: `⊻`,
+		veeeq: `≚`,
+		vellip: `⋮`,
+		Verbar: `‖`,
+		verbar: `|`,
+		Vert: `‖`,
+		vert: `|`,
+		VerticalBar: `∣`,
+		VerticalLine: `|`,
+		VerticalSeparator: `❘`,
+		VerticalTilde: `≀`,
+		VeryThinSpace: ` `,
+		Vfr: `𝔙`,
+		vfr: `𝔳`,
+		vltri: `⊲`,
+		vnsub: `⊂⃒`,
+		vnsup: `⊃⃒`,
+		Vopf: `𝕍`,
+		vopf: `𝕧`,
+		vprop: `∝`,
+		vrtri: `⊳`,
+		Vscr: `𝒱`,
+		vscr: `𝓋`,
+		vsubnE: `⫋︀`,
+		vsubne: `⊊︀`,
+		vsupnE: `⫌︀`,
+		vsupne: `⊋︀`,
+		Vvdash: `⊪`,
+		vzigzag: `⦚`,
+		Wcirc: `Ŵ`,
+		wcirc: `ŵ`,
+		wedbar: `⩟`,
+		Wedge: `⋀`,
+		wedge: `∧`,
+		wedgeq: `≙`,
+		weierp: `℘`,
+		Wfr: `𝔚`,
+		wfr: `𝔴`,
+		Wopf: `𝕎`,
+		wopf: `𝕨`,
+		wp: `℘`,
+		wr: `≀`,
+		wreath: `≀`,
+		Wscr: `𝒲`,
+		wscr: `𝓌`,
+		xcap: `⋂`,
+		xcirc: `◯`,
+		xcup: `⋃`,
+		xdtri: `▽`,
+		Xfr: `𝔛`,
+		xfr: `𝔵`,
+		xhArr: `⟺`,
+		xharr: `⟷`,
+		Xi: `Ξ`,
+		xi: `ξ`,
+		xlArr: `⟸`,
+		xlarr: `⟵`,
+		xmap: `⟼`,
+		xnis: `⋻`,
+		xodot: `⨀`,
+		Xopf: `𝕏`,
+		xopf: `𝕩`,
+		xoplus: `⨁`,
+		xotime: `⨂`,
+		xrArr: `⟹`,
+		xrarr: `⟶`,
+		Xscr: `𝒳`,
+		xscr: `𝓍`,
+		xsqcup: `⨆`,
+		xuplus: `⨄`,
+		xutri: `△`,
+		xvee: `⋁`,
+		xwedge: `⋀`,
+		Yacute: `Ý`,
+		yacute: `ý`,
+		YAcy: `Я`,
+		yacy: `я`,
+		Ycirc: `Ŷ`,
+		ycirc: `ŷ`,
+		Ycy: `Ы`,
+		ycy: `ы`,
+		yen: `¥`,
+		Yfr: `𝔜`,
+		yfr: `𝔶`,
+		YIcy: `Ї`,
+		yicy: `ї`,
+		Yopf: `𝕐`,
+		yopf: `𝕪`,
+		Yscr: `𝒴`,
+		yscr: `𝓎`,
+		YUcy: `Ю`,
+		yucy: `ю`,
+		Yuml: `Ÿ`,
+		yuml: `ÿ`,
+		Zacute: `Ź`,
+		zacute: `ź`,
+		Zcaron: `Ž`,
+		zcaron: `ž`,
+		Zcy: `З`,
+		zcy: `з`,
+		Zdot: `Ż`,
+		zdot: `ż`,
+		zeetrf: `ℨ`,
+		ZeroWidthSpace: `​`,
+		Zeta: `Ζ`,
+		zeta: `ζ`,
+		Zfr: `ℨ`,
+		zfr: `𝔷`,
+		ZHcy: `Ж`,
+		zhcy: `ж`,
+		zigrarr: `⇝`,
+		Zopf: `ℤ`,
+		zopf: `𝕫`,
+		Zscr: `𝒵`,
+		zscr: `𝓏`,
+		zwj: `‍`,
+		zwnj: `‌`
+	}), exports.entityMap = exports.HTML_ENTITIES;
+})), i = o$1(((exports) => {
+	var n = t().NAMESPACE, r = /[A-Z_a-z\xC0-\xD6\xD8-\xF6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]/, i = RegExp(`[\\-\\.0-9` + r.source.slice(1, -1) + `\\u00B7\\u0300-\\u036F\\u203F-\\u2040]`), a = RegExp(`^` + r.source + i.source + `*(?::` + r.source + i.source + `*)?$`), o = 0, s = 1, c = 2, l = 3, u = 4, d = 5, f = 6, p = 7;
+	function m(e, t) {
+		this.message = e, this.locator = t, Error.captureStackTrace && Error.captureStackTrace(this, m);
+	}
+	m.prototype = Error(), m.prototype.name = m.name;
+	function h() {}
+	h.prototype = { parse: function(e, t, n) {
+		var r = this.domBuilder;
+		r.startDocument(), S(t, t = {}), g(e, t, n, r, this.errorHandler), r.endDocument();
+	} };
+	function g(e, t, r, i, a) {
+		function o(e) {
+			if (e > 65535) {
+				e -= 65536;
+				var t = 55296 + (e >> 10), n = 56320 + (e & 1023);
+				return String.fromCharCode(t, n);
+			} else return String.fromCharCode(e);
+		}
+		function s(e) {
+			var t = e.slice(1, -1);
+			return Object.hasOwnProperty.call(r, t) ? r[t] : t.charAt(0) === `#` ? o(parseInt(t.substr(1).replace(`x`, `0x`))) : (a.error(`entity not found:` + e), e);
+		}
+		function c(t) {
+			if (t > S) {
+				var n = e.substring(S, t).replace(/&#?\w+;/g, s);
+				p && l(S), i.characters(n, 0, t - S), S = t;
+			}
+		}
+		function l(t, n) {
+			for (; t >= d && (n = f.exec(e));) u = n.index, d = u + n[0].length, p.lineNumber++;
+			p.columnNumber = t - u + 1;
+		}
+		for (var u = 0, d = 0, f = /.*(?:\r\n?|\n)|.*$/g, p = i.locator, h = [{ currentNSMap: t }], g = {}, S = 0;;) {
+			try {
+				var E = e.indexOf(`<`, S);
+				if (E < 0) {
+					if (!e.substr(S).match(/^\s*$/)) {
+						var D = i.doc, O = D.createTextNode(e.substr(S));
+						D.appendChild(O), i.currentElement = O;
+					}
+					return;
+				}
+				switch (E > S && c(E), e.charAt(E + 1)) {
+					case `/`:
+						var k = e.indexOf(`>`, E + 3), A = e.substring(E + 2, k).replace(/[ \t\n\r]+$/g, ``), j = h.pop();
+						k < 0 ? (A = e.substring(E + 2).replace(/[\s<].*/, ``), a.error(`end tag name: ` + A + ` is not complete:` + j.tagName), k = E + 1 + A.length) : A.match(/\s</) && (A = A.replace(/[\s<].*/, ``), a.error(`end tag name: ` + A + ` maybe not complete`), k = E + 1 + A.length);
+						var M = j.localNSMap, N = j.tagName == A;
+						if (N || j.tagName && j.tagName.toLowerCase() == A.toLowerCase()) {
+							if (i.endElement(j.uri, j.localName, A), M) for (var P in M) Object.prototype.hasOwnProperty.call(M, P) && i.endPrefixMapping(P);
+							N || a.fatalError(`end tag name: ` + A + ` is not match the current start tagName:` + j.tagName);
+						} else h.push(j);
+						k++;
+						break;
+					case `?`:
+						p && l(E), k = w(e, E, i);
+						break;
+					case `!`:
+						p && l(E), k = C(e, E, i, a);
+						break;
+					default:
+						p && l(E);
+						var F = new T(), I = h[h.length - 1].currentNSMap, k = v(e, E, F, I, s, a), L = F.length;
+						if (!F.closed && x(e, k, F.tagName, g) && (F.closed = !0, r.nbsp || a.warning(`unclosed xml attribute`)), p && L) {
+							for (var R = _(p, {}), z = 0; z < L; z++) {
+								var B = F[z];
+								l(B.offset), B.locator = _(p, {});
+							}
+							i.locator = R, y(F, i, I) && h.push(F), i.locator = p;
+						} else y(F, i, I) && h.push(F);
+						n.isHTML(F.uri) && !F.closed ? k = b(e, k, F.tagName, s, i) : k++;
+				}
+			} catch (e) {
+				if (e instanceof m) throw e;
+				a.error(`element parse error: ` + e), k = -1;
+			}
+			k > S ? S = k : c(Math.max(E, S) + 1);
+		}
+	}
+	function _(e, t) {
+		return t.lineNumber = e.lineNumber, t.columnNumber = e.columnNumber, t;
+	}
+	function v(e, t, r, i, a, m) {
+		function h(e, t, n) {
+			r.attributeNames.hasOwnProperty(e) && m.fatalError(`Attribute ` + e + ` redefined`), r.addValue(e, t.replace(/[\t\n\r]/g, ` `).replace(/&#?\w+;/g, a), n);
+		}
+		for (var g, _, v = ++t, y = o;;) {
+			var b = e.charAt(v);
+			switch (b) {
+				case `=`:
+					if (y === s) g = e.slice(t, v), y = l;
+					else if (y === c) y = l;
+					else throw Error(`attribute equal must after attrName`);
+					break;
+				case `'`:
+				case `"`:
+					if (y === l || y === s) if (y === s && (m.warning(`attribute value must after "="`), g = e.slice(t, v)), t = v + 1, v = e.indexOf(b, t), v > 0) _ = e.slice(t, v), h(g, _, t - 1), y = d;
+					else throw Error(`attribute value no end '` + b + `' match`);
+					else if (y == u) _ = e.slice(t, v), h(g, _, t), m.warning(`attribute "` + g + `" missed start quot(` + b + `)!!`), t = v + 1, y = d;
+					else throw Error(`attribute value must after "="`);
+					break;
+				case `/`:
+					switch (y) {
+						case o: r.setTagName(e.slice(t, v));
+						case d:
+						case f:
+						case p: y = p, r.closed = !0;
+						case u:
+						case s: break;
+						case c:
+							r.closed = !0;
+							break;
+						default: throw Error(`attribute invalid close char('/')`);
+					}
+					break;
+				case ``: return m.error(`unexpected end of input`), y == o && r.setTagName(e.slice(t, v)), v;
+				case `>`:
+					switch (y) {
+						case o: r.setTagName(e.slice(t, v));
+						case d:
+						case f:
+						case p: break;
+						case u:
+						case s: _ = e.slice(t, v), _.slice(-1) === `/` && (r.closed = !0, _ = _.slice(0, -1));
+						case c:
+							y === c && (_ = g), y == u ? (m.warning(`attribute "` + _ + `" missed quot(")!`), h(g, _, t)) : ((!n.isHTML(i[``]) || !_.match(/^(?:disabled|checked|selected)$/i)) && m.warning(`attribute "` + _ + `" missed value!! "` + _ + `" instead!!`), h(_, _, t));
+							break;
+						case l: throw Error(`attribute value missed!!`);
+					}
+					return v;
+				case `€`: b = ` `;
+				default: if (b <= ` `) switch (y) {
+					case o:
+						r.setTagName(e.slice(t, v)), y = f;
+						break;
+					case s:
+						g = e.slice(t, v), y = c;
+						break;
+					case u:
+						var _ = e.slice(t, v);
+						m.warning(`attribute "` + _ + `" missed quot(")!!`), h(g, _, t);
+					case d:
+						y = f;
+						break;
+				}
+				else switch (y) {
+					case c:
+						r.tagName, (!n.isHTML(i[``]) || !g.match(/^(?:disabled|checked|selected)$/i)) && m.warning(`attribute "` + g + `" missed value!! "` + g + `" instead2!!`), h(g, g, t), t = v, y = s;
+						break;
+					case d: m.warning(`attribute space is required"` + g + `"!!`);
+					case f:
+						y = s, t = v;
+						break;
+					case l:
+						y = u, t = v;
+						break;
+					case p: throw Error(`elements closed character '/' and '>' must be connected to`);
+				}
+			}
+			v++;
+		}
+	}
+	function y(e, t, r) {
+		for (var i = e.tagName, a = null, o = e.length; o--;) {
+			var s = e[o], c = s.qName, l = s.value, u = c.indexOf(`:`);
+			if (u > 0) var d = s.prefix = c.slice(0, u), f = c.slice(u + 1), p = d === `xmlns` && f;
+			else f = c, d = null, p = c === `xmlns` && ``;
+			s.localName = f, p !== !1 && (a ?? (a = {}, S(r, r = {})), r[p] = a[p] = l, s.uri = n.XMLNS, t.startPrefixMapping(p, l));
+		}
+		for (var o = e.length; o--;) {
+			s = e[o];
+			var d = s.prefix;
+			d && (d === `xml` && (s.uri = n.XML), d !== `xmlns` && (s.uri = r[d || ``]));
+		}
+		var u = i.indexOf(`:`);
+		u > 0 ? (d = e.prefix = i.slice(0, u), f = e.localName = i.slice(u + 1)) : (d = null, f = e.localName = i);
+		var m = e.uri = r[d || ``];
+		if (t.startElement(m, f, i, e), e.closed) {
+			if (t.endElement(m, f, i), a) for (d in a) Object.prototype.hasOwnProperty.call(a, d) && t.endPrefixMapping(d);
+		} else return e.currentNSMap = r, e.localNSMap = a, !0;
+	}
+	function b(e, t, n, r, i) {
+		if (/^(?:script|textarea)$/i.test(n)) {
+			var a = e.indexOf(`</` + n + `>`, t), o = e.substring(t + 1, a);
+			if (/[&<]/.test(o)) return /^script$/i.test(n) ? (i.characters(o, 0, o.length), a) : (o = o.replace(/&#?\w+;/g, r), i.characters(o, 0, o.length), a);
+		}
+		return t + 1;
+	}
+	function x(e, t, n, r) {
+		var i = r[n];
+		return i ?? (i = e.lastIndexOf(`</` + n + `>`), i < t && (i = e.lastIndexOf(`</` + n)), r[n] = i), i < t;
+	}
+	function S(e, t) {
+		for (var n in e) Object.prototype.hasOwnProperty.call(e, n) && (t[n] = e[n]);
+	}
+	function C(e, t, n, r) {
+		switch (e.charAt(t + 2)) {
+			case `-`: if (e.charAt(t + 3) === `-`) {
+				var i = e.indexOf(`-->`, t + 4);
+				return i > t ? (n.comment(e, t + 4, i - t - 4), i + 3) : (r.error(`Unclosed comment`), -1);
+			} else return -1;
+			default:
+				if (e.substr(t + 3, 6) == `CDATA[`) {
+					var i = e.indexOf(`]]>`, t + 9);
+					return n.startCDATA(), n.characters(e, t + 9, i - t - 9), n.endCDATA(), i + 3;
+				}
+				var a = E(e, t), o = a.length;
+				if (o > 1 && /!doctype/i.test(a[0][0])) {
+					var s = a[1][0], c = !1, l = !1;
+					o > 3 && (/^public$/i.test(a[2][0]) ? (c = a[3][0], l = o > 4 && a[4][0]) : /^system$/i.test(a[2][0]) && (l = a[3][0]));
+					var u = a[o - 1];
+					return n.startDTD(s, c, l), n.endDTD(), u.index + u[0].length;
+				}
+		}
+		return -1;
+	}
+	function w(e, t, n) {
+		var r = e.indexOf(`?>`, t);
+		if (r) {
+			var i = e.substring(t, r).match(/^<\?(\S*)\s*([\s\S]*?)\s*$/);
+			return i ? (i[0].length, n.processingInstruction(i[1], i[2]), r + 2) : -1;
+		}
+		return -1;
+	}
+	function T() {
+		this.attributeNames = {};
+	}
+	T.prototype = {
+		setTagName: function(e) {
+			if (!a.test(e)) throw Error(`invalid tagName:` + e);
+			this.tagName = e;
+		},
+		addValue: function(e, t, n) {
+			if (!a.test(e)) throw Error(`invalid attribute:` + e);
+			this.attributeNames[e] = this.length, this[this.length++] = {
+				qName: e,
+				value: t,
+				offset: n
+			};
+		},
+		length: 0,
+		getLocalName: function(e) {
+			return this[e].localName;
+		},
+		getLocator: function(e) {
+			return this[e].locator;
+		},
+		getQName: function(e) {
+			return this[e].qName;
+		},
+		getURI: function(e) {
+			return this[e].uri;
+		},
+		getValue: function(e) {
+			return this[e].value;
+		}
+	};
+	function E(e, t) {
+		var n, r = [], i = /'[^']+'|"[^"]+"|[^\s<>\/=]+=?|(\/?\s*>|<)/g;
+		for (i.lastIndex = t, i.exec(e); n = i.exec(e);) if (r.push(n), n[1]) return r;
+	}
+	exports.XMLReader = h, exports.ParseError = m;
+})), a = o$1(((exports) => {
+	var a = t(), o = n(), s = r(), c = i(), l = o.DOMImplementation, u = a.NAMESPACE, d = c.ParseError, f = c.XMLReader;
+	function p(e) {
+		return e.replace(/\r[\n\u0085]/g, `
+`).replace(/[\r\u0085\u2028]/g, `
+`);
+	}
+	function m(e) {
+		this.options = e || { locator: {} };
+	}
+	m.prototype.parseFromString = function(e, t) {
+		var n = this.options, r = new f(), i = n.domBuilder || new g(), a = n.errorHandler, o = n.locator, c = n.xmlns || {}, l = /\/x?html?$/.test(t), d = l ? s.HTML_ENTITIES : s.XML_ENTITIES;
+		o && i.setDocumentLocator(o), r.errorHandler = h(a, i, o), r.domBuilder = n.domBuilder || i, l && (c[``] = u.HTML), c.xml = c.xml || u.XML;
+		var m = n.normalizeLineEndings || p;
+		return e && typeof e == `string` ? r.parse(m(e), c, d) : r.errorHandler.error(`invalid doc source`), i.doc;
+	};
+	function h(e, t, n) {
+		if (!e) {
+			if (t instanceof g) return t;
+			e = t;
+		}
+		var r = {}, i = e instanceof Function;
+		n ||= {};
+		function a(t) {
+			var a = e[t];
+			!a && i && (a = e.length == 2 ? function(n) {
+				e(t, n);
+			} : e), r[t] = a && function(e) {
+				a(`[xmldom ` + t + `]	` + e + v(n));
+			} || function() {};
+		}
+		return a(`warning`), a(`error`), a(`fatalError`), r;
+	}
+	function g() {
+		this.cdata = !1;
+	}
+	function _(e, t) {
+		t.lineNumber = e.lineNumber, t.columnNumber = e.columnNumber;
+	}
+	g.prototype = {
+		startDocument: function() {
+			this.doc = new l().createDocument(null, null, null), this.locator && (this.doc.documentURI = this.locator.systemId);
+		},
+		startElement: function(e, t, n, r) {
+			var i = this.doc, a = i.createElementNS(e, n || t), o = r.length;
+			b(this, a), this.currentElement = a, this.locator && _(this.locator, a);
+			for (var s = 0; s < o; s++) {
+				var e = r.getURI(s), c = r.getValue(s), n = r.getQName(s), l = i.createAttributeNS(e, n);
+				this.locator && _(r.getLocator(s), l), l.value = l.nodeValue = c, a.setAttributeNode(l);
+			}
+		},
+		endElement: function(e, t, n) {
+			var r = this.currentElement;
+			r.tagName, this.currentElement = r.parentNode;
+		},
+		startPrefixMapping: function(e, t) {},
+		endPrefixMapping: function(e) {},
+		processingInstruction: function(e, t) {
+			var n = this.doc.createProcessingInstruction(e, t);
+			this.locator && _(this.locator, n), b(this, n);
+		},
+		ignorableWhitespace: function(e, t, n) {},
+		characters: function(e, t, n) {
+			if (e = y.apply(this, arguments), e) {
+				if (this.cdata) var r = this.doc.createCDATASection(e);
+				else var r = this.doc.createTextNode(e);
+				this.currentElement ? this.currentElement.appendChild(r) : /^\s*$/.test(e) && this.doc.appendChild(r), this.locator && _(this.locator, r);
+			}
+		},
+		skippedEntity: function(e) {},
+		endDocument: function() {
+			this.doc.normalize();
+		},
+		setDocumentLocator: function(e) {
+			(this.locator = e) && (e.lineNumber = 0);
+		},
+		comment: function(e, t, n) {
+			e = y.apply(this, arguments);
+			var r = this.doc.createComment(e);
+			this.locator && _(this.locator, r), b(this, r);
+		},
+		startCDATA: function() {
+			this.cdata = !0;
+		},
+		endCDATA: function() {
+			this.cdata = !1;
+		},
+		startDTD: function(e, t, n) {
+			var r = this.doc.implementation;
+			if (r && r.createDocumentType) {
+				var i = r.createDocumentType(e, t, n);
+				this.locator && _(this.locator, i), b(this, i), this.doc.doctype = i;
+			}
+		},
+		warning: function(e) {
+			console.warn(`[xmldom warning]	` + e, v(this.locator));
+		},
+		error: function(e) {
+			console.error(`[xmldom error]	` + e, v(this.locator));
+		},
+		fatalError: function(e) {
+			throw new d(e, this.locator);
+		}
+	};
+	function v(e) {
+		if (e) return `
+@` + (e.systemId || ``) + `#[line:` + e.lineNumber + `,col:` + e.columnNumber + `]`;
+	}
+	function y(e, t, n) {
+		return typeof e == `string` ? e.substr(t, n) : e.length >= t + n || t ? new java.lang.String(e, t, n) + `` : e;
+	}
+	`endDTD,startEntity,endEntity,attributeDecl,elementDecl,externalEntityDecl,internalEntityDecl,resolveEntity,getExternalSubset,notationDecl,unparsedEntityDecl`.replace(/\w+/g, function(e) {
+		g.prototype[e] = function() {
+			return null;
+		};
+	});
+	function b(e, t) {
+		e.currentElement ? e.currentElement.appendChild(t) : e.doc.appendChild(t);
+	}
+	exports.__DOMHandler = g, exports.normalizeLineEndings = p, exports.DOMParser = m;
+}));
+var lib_CBtriEt5_default = o$1(((exports) => {
+	var t = n();
+	exports.DOMImplementation = t.DOMImplementation, exports.XMLSerializer = t.XMLSerializer, exports.DOMParser = a().DOMParser;
+}))();
+//#endregion
+export { lib_CBtriEt5_default as default };
+
+//# sourceMappingURL=lib-CBtriEt5.js.map

viewer/assets/liteDOM-Cp0aN3bP.js

@@ -0,0 +1,747 @@
+//#region ../../node_modules/.pnpm/mathxyjax3@0.8.3/node_modules/mathxyjax3/dist/liteDOM-Cp0aN3bP.js
+(function() {
+	var e, t, n, r, i, a, o, s, c, l = {
+		244: function(e, t, n) {
+			var r, i = this && this.__extends || (r = function(e, t) {
+				return r = Object.setPrototypeOf || { __proto__: [] } instanceof Array && function(e, t) {
+					e.__proto__ = t;
+				} || function(e, t) {
+					for (var n in t) Object.prototype.hasOwnProperty.call(t, n) && (e[n] = t[n]);
+				}, r(e, t);
+			}, function(e, t) {
+				if (typeof t != `function` && t !== null) throw TypeError(`Class extends value ` + String(t) + ` is not a constructor or null`);
+				function n() {
+					this.constructor = e;
+				}
+				r(e, t), e.prototype = t === null ? Object.create(t) : (n.prototype = t.prototype, new n());
+			}), a = this && this.__assign || function() {
+				return a = Object.assign || function(e) {
+					for (var t, n = 1, r = arguments.length; n < r; n++) for (var i in t = arguments[n]) Object.prototype.hasOwnProperty.call(t, i) && (e[i] = t[i]);
+					return e;
+				}, a.apply(this, arguments);
+			};
+			Object.defineProperty(t, `__esModule`, { value: !0 }), t.NodeMixin = t.NodeMixinOptions = void 0;
+			var o = n(77);
+			t.NodeMixinOptions = {
+				badCSS: !0,
+				badSizes: !0
+			}, t.NodeMixin = function(e, n) {
+				var r;
+				return n === void 0 && (n = {}), n = (0, o.userOptions)((0, o.defaultOptions)({}, t.NodeMixinOptions), n), r = function(e) {
+					function t() {
+						var t = [...arguments], n = e.call(this, t[0]) || this, r = n.constructor;
+						return n.options = (0, o.userOptions)((0, o.defaultOptions)({}, r.OPTIONS), t[1]), n;
+					}
+					return i(t, e), t.prototype.fontSize = function(t) {
+						return n.badCSS ? this.options.fontSize : e.prototype.fontSize.call(this, t);
+					}, t.prototype.fontFamily = function(t) {
+						return n.badCSS ? this.options.fontFamily : e.prototype.fontFamily.call(this, t);
+					}, t.prototype.nodeSize = function(r, i, a) {
+						if (i === void 0 && (i = 1), a === void 0 && (a = null), !n.badSizes) return e.prototype.nodeSize.call(this, r, i, a);
+						var o = this.textContent(r), s = Array.from(o.replace(t.cjkPattern, ``)).length;
+						return [(Array.from(o).length - s) * this.options.cjkCharWidth + s * this.options.unknownCharWidth, this.options.unknownCharHeight];
+					}, t.prototype.nodeBBox = function(t) {
+						return n.badSizes ? {
+							left: 0,
+							right: 0,
+							top: 0,
+							bottom: 0
+						} : e.prototype.nodeBBox.call(this, t);
+					}, t;
+				}(e), r.OPTIONS = a(a({}, n.badCSS ? {
+					fontSize: 16,
+					fontFamily: `Times`
+				} : {}), n.badSizes ? {
+					cjkCharWidth: 1,
+					unknownCharWidth: .6,
+					unknownCharHeight: .8
+				} : {}), r.cjkPattern = new RegExp([
+					`[`,
+					`ᄀ-ᅟ`,
+					`〈〉`,
+					`⺀-〾`,
+					`぀-㉇`,
+					`㉐-䶿`,
+					`一-꓆`,
+					`ꥠ-ꥼ`,
+					`가-힣`,
+					`豈-﫿`,
+					`︐-︙`,
+					`︰-﹫`,
+					`！-｠￠-￦`,
+					`𛀀-𛀁`,
+					`🈀-🉑`,
+					`𠀀-𿿽`,
+					`]`
+				].join(``), `gu`), r;
+			};
+		},
+		877: function(e, t, n) {
+			Object.defineProperty(t, `__esModule`, { value: !0 }), t.LiteDocument = void 0;
+			var r = n(946);
+			t.LiteDocument = function() {
+				function e() {
+					this.root = new r.LiteElement(`html`, {}, [this.head = new r.LiteElement(`head`), this.body = new r.LiteElement(`body`)]), this.type = ``;
+				}
+				return Object.defineProperty(e.prototype, `kind`, {
+					get: function() {
+						return `#document`;
+					},
+					enumerable: !1,
+					configurable: !0
+				}), e;
+			}();
+		},
+		946: function(e, t) {
+			var n = this && this.__assign || function() {
+				return n = Object.assign || function(e) {
+					for (var t, n = 1, r = arguments.length; n < r; n++) for (var i in t = arguments[n]) Object.prototype.hasOwnProperty.call(t, i) && (e[i] = t[i]);
+					return e;
+				}, n.apply(this, arguments);
+			}, r = this && this.__read || function(e, t) {
+				var n = typeof Symbol == `function` && e[Symbol.iterator];
+				if (!n) return e;
+				var r, i, a = n.call(e), o = [];
+				try {
+					for (; (t === void 0 || t-- > 0) && !(r = a.next()).done;) o.push(r.value);
+				} catch (e) {
+					i = { error: e };
+				} finally {
+					try {
+						r && !r.done && (n = a.return) && n.call(a);
+					} finally {
+						if (i) throw i.error;
+					}
+				}
+				return o;
+			}, i = this && this.__spreadArray || function(e, t, n) {
+				if (n || arguments.length === 2) for (var r, i = 0, a = t.length; i < a; i++) !r && i in t || (r ||= Array.prototype.slice.call(t, 0, i), r[i] = t[i]);
+				return e.concat(r || Array.prototype.slice.call(t));
+			}, a = this && this.__values || function(e) {
+				var t = typeof Symbol == `function` && Symbol.iterator, n = t && e[t], r = 0;
+				if (n) return n.call(e);
+				if (e && typeof e.length == `number`) return { next: function() {
+					return e && r >= e.length && (e = void 0), {
+						value: e && e[r++],
+						done: !e
+					};
+				} };
+				throw TypeError(t ? `Object is not iterable.` : `Symbol.iterator is not defined.`);
+			};
+			Object.defineProperty(t, `__esModule`, { value: !0 }), t.LiteElement = void 0, t.LiteElement = function(e, t, o) {
+				var s, c;
+				t === void 0 && (t = {}), o === void 0 && (o = []), this.kind = e, this.attributes = n({}, t), this.children = i([], r(o), !1);
+				try {
+					for (var l = a(this.children), u = l.next(); !u.done; u = l.next()) u.value.parent = this;
+				} catch (e) {
+					s = { error: e };
+				} finally {
+					try {
+						u && !u.done && (c = l.return) && c.call(l);
+					} finally {
+						if (s) throw s.error;
+					}
+				}
+				this.styles = null;
+			};
+		},
+		6: function(e, t) {
+			var n = this && this.__read || function(e, t) {
+				var n = typeof Symbol == `function` && e[Symbol.iterator];
+				if (!n) return e;
+				var r, i, a = n.call(e), o = [];
+				try {
+					for (; (t === void 0 || t-- > 0) && !(r = a.next()).done;) o.push(r.value);
+				} catch (e) {
+					i = { error: e };
+				} finally {
+					try {
+						r && !r.done && (n = a.return) && n.call(a);
+					} finally {
+						if (i) throw i.error;
+					}
+				}
+				return o;
+			}, r = this && this.__spreadArray || function(e, t, n) {
+				if (n || arguments.length === 2) for (var r, i = 0, a = t.length; i < a; i++) !r && i in t || (r ||= Array.prototype.slice.call(t, 0, i), r[i] = t[i]);
+				return e.concat(r || Array.prototype.slice.call(t));
+			};
+			Object.defineProperty(t, `__esModule`, { value: !0 }), t.LiteList = void 0, t.LiteList = function() {
+				function e(e) {
+					this.nodes = [], this.nodes = r([], n(e), !1);
+				}
+				return e.prototype.append = function(e) {
+					this.nodes.push(e);
+				}, e.prototype[Symbol.iterator] = function() {
+					var e = 0;
+					return { next: function() {
+						return e === this.nodes.length ? {
+							value: null,
+							done: !0
+						} : {
+							value: this.nodes[e++],
+							done: !1
+						};
+					} };
+				}, e;
+			}();
+		},
+		246: function(e, t, n) {
+			var r = this && this.__createBinding || (Object.create ? function(e, t, n, r) {
+				r === void 0 && (r = n);
+				var i = Object.getOwnPropertyDescriptor(t, n);
+				i && !(`get` in i ? !t.__esModule : i.writable || i.configurable) || (i = {
+					enumerable: !0,
+					get: function() {
+						return t[n];
+					}
+				}), Object.defineProperty(e, r, i);
+			} : function(e, t, n, r) {
+				r === void 0 && (r = n), e[r] = t[n];
+			}), i = this && this.__setModuleDefault || (Object.create ? function(e, t) {
+				Object.defineProperty(e, `default`, {
+					enumerable: !0,
+					value: t
+				});
+			} : function(e, t) {
+				e.default = t;
+			}), a = this && this.__importStar || function(e) {
+				if (e && e.__esModule) return e;
+				var t = {};
+				if (e != null) for (var n in e) n !== `default` && Object.prototype.hasOwnProperty.call(e, n) && r(t, e, n);
+				return i(t, e), t;
+			}, o = this && this.__read || function(e, t) {
+				var n = typeof Symbol == `function` && e[Symbol.iterator];
+				if (!n) return e;
+				var r, i, a = n.call(e), o = [];
+				try {
+					for (; (t === void 0 || t-- > 0) && !(r = a.next()).done;) o.push(r.value);
+				} catch (e) {
+					i = { error: e };
+				} finally {
+					try {
+						r && !r.done && (n = a.return) && n.call(a);
+					} finally {
+						if (i) throw i.error;
+					}
+				}
+				return o;
+			}, s = this && this.__values || function(e) {
+				var t = typeof Symbol == `function` && Symbol.iterator, n = t && e[t], r = 0;
+				if (n) return n.call(e);
+				if (e && typeof e.length == `number`) return { next: function() {
+					return e && r >= e.length && (e = void 0), {
+						value: e && e[r++],
+						done: !e
+					};
+				} };
+				throw TypeError(t ? `Object is not iterable.` : `Symbol.iterator is not defined.`);
+			};
+			Object.defineProperty(t, `__esModule`, { value: !0 }), t.LiteParser = t.PATTERNS = void 0;
+			var c, l = a(n(29)), u = n(946), d = n(735);
+			(function(e) {
+				e.TAGNAME = `[a-z][^\\s\\n>]*`, e.ATTNAME = `[a-z][^\\s\\n>=]*`, e.VALUE = `(?:'[^']*'|"[^"]*"|[^\\s\\n]+)`, e.VALUESPLIT = `(?:'([^']*)'|"([^"]*)"|([^\\s\\n]+))`, e.SPACE = `(?:\\s|\\n)+`, e.OPTIONALSPACE = `(?:\\s|\\n)*`, e.ATTRIBUTE = e.ATTNAME + `(?:` + e.OPTIONALSPACE + `=` + e.OPTIONALSPACE + e.VALUE + `)?`, e.ATTRIBUTESPLIT = `(` + e.ATTNAME + `)(?:` + e.OPTIONALSPACE + `=` + e.OPTIONALSPACE + e.VALUESPLIT + `)?`, e.TAG = `(<(?:` + e.TAGNAME + `(?:` + e.SPACE + e.ATTRIBUTE + `)*` + e.OPTIONALSPACE + `/?|/` + e.TAGNAME + `|!--[^]*?--|![^]*?)(?:>|$))`, e.tag = new RegExp(e.TAG, `i`), e.attr = new RegExp(e.ATTRIBUTE, `i`), e.attrsplit = new RegExp(e.ATTRIBUTESPLIT, `i`);
+			})(c = t.PATTERNS ||= {}), t.LiteParser = function() {
+				function e() {}
+				return e.prototype.parseFromString = function(e, t, n) {
+					t === void 0 && (t = `text/html`), n === void 0 && (n = null);
+					for (var r = n.createDocument(), i = n.body(r), a = e.replace(/<\?.*?\?>/g, ``).split(c.tag); a.length;) {
+						var o = a.shift(), s = a.shift();
+						o && this.addText(n, i, o), s && s.charAt(s.length - 1) === `>` && (s.charAt(1) === `!` ? this.addComment(n, i, s) : i = s.charAt(1) === `/` ? this.closeTag(n, i, s) : this.openTag(n, i, s, a));
+					}
+					return this.checkDocument(n, r), r;
+				}, e.prototype.addText = function(e, t, n) {
+					return n = l.translate(n), e.append(t, e.text(n));
+				}, e.prototype.addComment = function(e, t, n) {
+					return e.append(t, new d.LiteComment(n));
+				}, e.prototype.closeTag = function(e, t, n) {
+					for (var r = n.slice(2, n.length - 1).toLowerCase(); e.parent(t) && e.kind(t) !== r;) t = e.parent(t);
+					return e.parent(t);
+				}, e.prototype.openTag = function(e, t, n, r) {
+					var i = this.constructor.PCDATA, a = this.constructor.SELF_CLOSING, o = n.match(/<(.*?)[\s\n>\/]/)[1].toLowerCase(), s = e.node(o), l = n.replace(/^<.*?[\s\n>]/, ``).split(c.attrsplit);
+					return (l.pop().match(/>$/) || l.length < 5) && (this.addAttributes(e, s, l), e.append(t, s), a[o] || n.match(/\/>$/) || (i[o] ? this.handlePCDATA(e, s, o, r) : t = s)), t;
+				}, e.prototype.addAttributes = function(e, t, n) {
+					for (var r = this.constructor.CDATA_ATTR; n.length;) {
+						var i = o(n.splice(0, 5), 5), a = i[1], s = i[2], c = i[3], u = i[4], d = s || c || u || ``;
+						r[a] || (d = l.translate(d)), e.setAttribute(t, a, d);
+					}
+				}, e.prototype.handlePCDATA = function(e, t, n, r) {
+					for (var i = [], a = `</` + n + `>`, o = ``; r.length && o !== a;) i.push(o), i.push(r.shift()), o = r.shift();
+					e.append(t, e.text(i.join(``)));
+				}, e.prototype.checkDocument = function(e, t) {
+					var n, r, i, a, o = this.getOnlyChild(e, e.body(t));
+					if (o) {
+						try {
+							for (var c = s(e.childNodes(e.body(t))), l = c.next(); !l.done && (p = l.value) !== o; l = c.next()) p instanceof d.LiteComment && p.value.match(/^<!DOCTYPE/) && (t.type = p.value);
+						} catch (e) {
+							n = { error: e };
+						} finally {
+							try {
+								l && !l.done && (r = c.return) && r.call(c);
+							} finally {
+								if (n) throw n.error;
+							}
+						}
+						switch (e.kind(o)) {
+							case `html`:
+								try {
+									for (var u = s(o.children), f = u.next(); !f.done; f = u.next()) {
+										var p = f.value;
+										switch (e.kind(p)) {
+											case `head`:
+												t.head = p;
+												break;
+											case `body`: t.body = p;
+										}
+									}
+								} catch (e) {
+									i = { error: e };
+								} finally {
+									try {
+										f && !f.done && (a = u.return) && a.call(u);
+									} finally {
+										if (i) throw i.error;
+									}
+								}
+								t.root = o, e.remove(o), e.parent(t.body) !== o && e.append(o, t.body), e.parent(t.head) !== o && e.insert(t.head, t.body);
+								break;
+							case `head`:
+								t.head = e.replace(o, t.head);
+								break;
+							case `body`: t.body = e.replace(o, t.body);
+						}
+					}
+				}, e.prototype.getOnlyChild = function(e, t) {
+					var n, r, i = null;
+					try {
+						for (var a = s(e.childNodes(t)), o = a.next(); !o.done; o = a.next()) {
+							var c = o.value;
+							if (c instanceof u.LiteElement) {
+								if (i) return null;
+								i = c;
+							}
+						}
+					} catch (e) {
+						n = { error: e };
+					} finally {
+						try {
+							o && !o.done && (r = a.return) && r.call(a);
+						} finally {
+							if (n) throw n.error;
+						}
+					}
+					return i;
+				}, e.prototype.serialize = function(e, t, n) {
+					var r = this;
+					n === void 0 && (n = !1);
+					var i = this.constructor.SELF_CLOSING, a = this.constructor.CDATA_ATTR, o = e.kind(t), s = e.allAttributes(t).map((function(e) {
+						return e.name + `="` + (a[e.name] ? e.value : r.protectAttribute(e.value)) + `"`;
+					})).join(` `), c = this.serializeInner(e, t, n);
+					return `<` + o + (s ? ` ` + s : ``) + (n && !c || i[o] ? n ? `/>` : `>` : `>${c}</${o}>`);
+				}, e.prototype.serializeInner = function(e, t, n) {
+					var r = this;
+					return n === void 0 && (n = !1), this.constructor.PCDATA.hasOwnProperty(t.kind) ? e.childNodes(t).map((function(t) {
+						return e.value(t);
+					})).join(``) : e.childNodes(t).map((function(t) {
+						var i = e.kind(t);
+						return i === `#text` ? r.protectHTML(e.value(t)) : i === `#comment` ? t.value : r.serialize(e, t, n);
+					})).join(``);
+				}, e.prototype.protectAttribute = function(e) {
+					return typeof e != `string` && (e = String(e)), e.replace(/"/g, `&quot;`);
+				}, e.prototype.protectHTML = function(e) {
+					return e.replace(/&/g, `&amp;`).replace(/</g, `&lt;`).replace(/>/g, `&gt;`);
+				}, e.SELF_CLOSING = {
+					area: !0,
+					base: !0,
+					br: !0,
+					col: !0,
+					command: !0,
+					embed: !0,
+					hr: !0,
+					img: !0,
+					input: !0,
+					keygen: !0,
+					link: !0,
+					menuitem: !0,
+					meta: !0,
+					param: !0,
+					source: !0,
+					track: !0,
+					wbr: !0
+				}, e.PCDATA = {
+					option: !0,
+					textarea: !0,
+					fieldset: !0,
+					title: !0,
+					style: !0,
+					script: !0
+				}, e.CDATA_ATTR = {
+					style: !0,
+					datafld: !0,
+					datasrc: !0,
+					href: !0,
+					src: !0,
+					longdesc: !0,
+					usemap: !0,
+					cite: !0,
+					datetime: !0,
+					action: !0,
+					axis: !0,
+					profile: !0,
+					content: !0,
+					scheme: !0
+				}, e;
+			}();
+		},
+		735: function(e, t) {
+			var n, r = this && this.__extends || (n = function(e, t) {
+				return n = Object.setPrototypeOf || { __proto__: [] } instanceof Array && function(e, t) {
+					e.__proto__ = t;
+				} || function(e, t) {
+					for (var n in t) Object.prototype.hasOwnProperty.call(t, n) && (e[n] = t[n]);
+				}, n(e, t);
+			}, function(e, t) {
+				if (typeof t != `function` && t !== null) throw TypeError(`Class extends value ` + String(t) + ` is not a constructor or null`);
+				function r() {
+					this.constructor = e;
+				}
+				n(e, t), e.prototype = t === null ? Object.create(t) : (r.prototype = t.prototype, new r());
+			});
+			Object.defineProperty(t, `__esModule`, { value: !0 }), t.LiteComment = t.LiteText = void 0;
+			var i = function() {
+				function e(e) {
+					e === void 0 && (e = ``), this.value = e;
+				}
+				return Object.defineProperty(e.prototype, `kind`, {
+					get: function() {
+						return `#text`;
+					},
+					enumerable: !1,
+					configurable: !0
+				}), e;
+			}();
+			t.LiteText = i, t.LiteComment = function(e) {
+				function t() {
+					return e !== null && e.apply(this, arguments) || this;
+				}
+				return r(t, e), Object.defineProperty(t.prototype, `kind`, {
+					get: function() {
+						return `#comment`;
+					},
+					enumerable: !1,
+					configurable: !0
+				}), t;
+			}(i);
+		},
+		492: function(e, t, n) {
+			Object.defineProperty(t, `__esModule`, { value: !0 }), t.LiteWindow = void 0;
+			var r = n(946), i = n(877), a = n(6), o = n(246);
+			t.LiteWindow = function() {
+				this.DOMParser = o.LiteParser, this.NodeList = a.LiteList, this.HTMLCollection = a.LiteList, this.HTMLElement = r.LiteElement, this.DocumentFragment = a.LiteList, this.Document = i.LiteDocument, this.document = new i.LiteDocument();
+			};
+		},
+		250: function(e, t, n) {
+			var r, i = this && this.__extends || (r = function(e, t) {
+				return r = Object.setPrototypeOf || { __proto__: [] } instanceof Array && function(e, t) {
+					e.__proto__ = t;
+				} || function(e, t) {
+					for (var n in t) Object.prototype.hasOwnProperty.call(t, n) && (e[n] = t[n]);
+				}, r(e, t);
+			}, function(e, t) {
+				if (typeof t != `function` && t !== null) throw TypeError(`Class extends value ` + String(t) + ` is not a constructor or null`);
+				function n() {
+					this.constructor = e;
+				}
+				r(e, t), e.prototype = t === null ? Object.create(t) : (n.prototype = t.prototype, new n());
+			}), a = this && this.__assign || function() {
+				return a = Object.assign || function(e) {
+					for (var t, n = 1, r = arguments.length; n < r; n++) for (var i in t = arguments[n]) Object.prototype.hasOwnProperty.call(t, i) && (e[i] = t[i]);
+					return e;
+				}, a.apply(this, arguments);
+			}, o = this && this.__values || function(e) {
+				var t = typeof Symbol == `function` && Symbol.iterator, n = t && e[t], r = 0;
+				if (n) return n.call(e);
+				if (e && typeof e.length == `number`) return { next: function() {
+					return e && r >= e.length && (e = void 0), {
+						value: e && e[r++],
+						done: !e
+					};
+				} };
+				throw TypeError(t ? `Object is not iterable.` : `Symbol.iterator is not defined.`);
+			}, s = this && this.__read || function(e, t) {
+				var n = typeof Symbol == `function` && e[Symbol.iterator];
+				if (!n) return e;
+				var r, i, a = n.call(e), o = [];
+				try {
+					for (; (t === void 0 || t-- > 0) && !(r = a.next()).done;) o.push(r.value);
+				} catch (e) {
+					i = { error: e };
+				} finally {
+					try {
+						r && !r.done && (n = a.return) && n.call(a);
+					} finally {
+						if (i) throw i.error;
+					}
+				}
+				return o;
+			}, c = this && this.__spreadArray || function(e, t, n) {
+				if (n || arguments.length === 2) for (var r, i = 0, a = t.length; i < a; i++) !r && i in t || (r ||= Array.prototype.slice.call(t, 0, i), r[i] = t[i]);
+				return e.concat(r || Array.prototype.slice.call(t));
+			};
+			Object.defineProperty(t, `__esModule`, { value: !0 }), t.liteAdaptor = t.LiteAdaptor = t.LiteBase = void 0;
+			var l = n(857), u = n(244), d = n(877), f = n(946), p = n(735), m = n(492), h = n(246), g = n(878), _ = function(e) {
+				function t() {
+					var t = e.call(this) || this;
+					return t.parser = new h.LiteParser(), t.window = new m.LiteWindow(), t;
+				}
+				return i(t, e), t.prototype.parse = function(e, t) {
+					return this.parser.parseFromString(e, t, this);
+				}, t.prototype.create = function(e, t) {
+					return t === void 0 && (t = null), new f.LiteElement(e);
+				}, t.prototype.text = function(e) {
+					return new p.LiteText(e);
+				}, t.prototype.comment = function(e) {
+					return new p.LiteComment(e);
+				}, t.prototype.createDocument = function() {
+					return new d.LiteDocument();
+				}, t.prototype.head = function(e) {
+					return e.head;
+				}, t.prototype.body = function(e) {
+					return e.body;
+				}, t.prototype.root = function(e) {
+					return e.root;
+				}, t.prototype.doctype = function(e) {
+					return e.type;
+				}, t.prototype.tags = function(e, t, n) {
+					n === void 0 && (n = null);
+					var r = [], i = [];
+					if (n) return i;
+					for (var a = e; a;) {
+						var o = a.kind;
+						o !== `#text` && o !== `#comment` && (a = a, o === t && i.push(a), a.children.length && (r = a.children.concat(r))), a = r.shift();
+					}
+					return i;
+				}, t.prototype.elementById = function(e, t) {
+					for (var n = [], r = e; r;) {
+						if (r.kind !== `#text` && r.kind !== `#comment`) {
+							if ((r = r).attributes.id === t) return r;
+							r.children.length && (n = r.children.concat(n));
+						}
+						r = n.shift();
+					}
+					return null;
+				}, t.prototype.elementsByClass = function(e, t) {
+					for (var n = [], r = [], i = e; i;) i.kind !== `#text` && i.kind !== `#comment` && (((i = i).attributes.class || ``).trim().split(/ +/).includes(t) && r.push(i), i.children.length && (n = i.children.concat(n))), i = n.shift();
+					return r;
+				}, t.prototype.getElements = function(e, t) {
+					var n, r, i = [], a = this.body(t);
+					try {
+						for (var s = o(e), c = s.next(); !c.done; c = s.next()) {
+							var l = c.value;
+							if (typeof l == `string`) if (l.charAt(0) === `#`) {
+								var u = this.elementById(a, l.slice(1));
+								u && i.push(u);
+							} else l.charAt(0) === `.` ? i = i.concat(this.elementsByClass(a, l.slice(1))) : l.match(/^[-a-z][-a-z0-9]*$/i) && (i = i.concat(this.tags(a, l)));
+							else Array.isArray(l) ? i = i.concat(l) : l instanceof this.window.NodeList || l instanceof this.window.HTMLCollection ? i = i.concat(l.nodes) : i.push(l);
+						}
+					} catch (e) {
+						n = { error: e };
+					} finally {
+						try {
+							c && !c.done && (r = s.return) && r.call(s);
+						} finally {
+							if (n) throw n.error;
+						}
+					}
+					return i;
+				}, t.prototype.contains = function(e, t) {
+					for (; t && t !== e;) t = this.parent(t);
+					return !!t;
+				}, t.prototype.parent = function(e) {
+					return e.parent;
+				}, t.prototype.childIndex = function(e) {
+					return e.parent ? e.parent.children.findIndex((function(t) {
+						return t === e;
+					})) : -1;
+				}, t.prototype.append = function(e, t) {
+					return t.parent && this.remove(t), e.children.push(t), t.parent = e, t;
+				}, t.prototype.insert = function(e, t) {
+					if (e.parent && this.remove(e), t && t.parent) {
+						var n = this.childIndex(t);
+						t.parent.children.splice(n, 0, e), e.parent = t.parent;
+					}
+				}, t.prototype.remove = function(e) {
+					var t = this.childIndex(e);
+					return t >= 0 && e.parent.children.splice(t, 1), e.parent = null, e;
+				}, t.prototype.replace = function(e, t) {
+					var n = this.childIndex(t);
+					return n >= 0 && (t.parent.children[n] = e, e.parent = t.parent, t.parent = null), t;
+				}, t.prototype.clone = function(e) {
+					var t = this, n = new f.LiteElement(e.kind);
+					return n.attributes = a({}, e.attributes), n.children = e.children.map((function(e) {
+						if (e.kind === `#text`) return new p.LiteText(e.value);
+						if (e.kind === `#comment`) return new p.LiteComment(e.value);
+						var r = t.clone(e);
+						return r.parent = n, r;
+					})), n;
+				}, t.prototype.split = function(e, t) {
+					var n = new p.LiteText(e.value.slice(t));
+					return e.value = e.value.slice(0, t), e.parent.children.splice(this.childIndex(e) + 1, 0, n), n.parent = e.parent, n;
+				}, t.prototype.next = function(e) {
+					var t = e.parent;
+					if (!t) return null;
+					var n = this.childIndex(e) + 1;
+					return n >= 0 && n < t.children.length ? t.children[n] : null;
+				}, t.prototype.previous = function(e) {
+					var t = e.parent;
+					if (!t) return null;
+					var n = this.childIndex(e) - 1;
+					return n >= 0 ? t.children[n] : null;
+				}, t.prototype.firstChild = function(e) {
+					return e.children[0];
+				}, t.prototype.lastChild = function(e) {
+					return e.children[e.children.length - 1];
+				}, t.prototype.childNodes = function(e) {
+					return c([], s(e.children), !1);
+				}, t.prototype.childNode = function(e, t) {
+					return e.children[t];
+				}, t.prototype.kind = function(e) {
+					return e.kind;
+				}, t.prototype.value = function(e) {
+					return e.kind === `#text` ? e.value : e.kind === `#comment` ? e.value.replace(/^<!(--)?((?:.|\n)*)\1>$/, `$2`) : ``;
+				}, t.prototype.textContent = function(e) {
+					var t = this;
+					return e.children.reduce((function(e, n) {
+						return e + (n.kind === `#text` ? n.value : n.kind === `#comment` ? `` : t.textContent(n));
+					}), ``);
+				}, t.prototype.innerHTML = function(e) {
+					return this.parser.serializeInner(this, e);
+				}, t.prototype.outerHTML = function(e) {
+					return this.parser.serialize(this, e);
+				}, t.prototype.serializeXML = function(e) {
+					return this.parser.serialize(this, e, !0);
+				}, t.prototype.setAttribute = function(e, t, n, r) {
+					r === void 0 && (r = null), typeof n != `string` && (n = String(n)), r && (t = r.replace(/.*\//, ``) + `:` + t.replace(/^.*:/, ``)), e.attributes[t] = n, t === `style` && (e.styles = null);
+				}, t.prototype.getAttribute = function(e, t) {
+					return e.attributes[t];
+				}, t.prototype.removeAttribute = function(e, t) {
+					delete e.attributes[t];
+				}, t.prototype.hasAttribute = function(e, t) {
+					return e.attributes.hasOwnProperty(t);
+				}, t.prototype.allAttributes = function(e) {
+					var t, n, r = e.attributes, i = [];
+					try {
+						for (var a = o(Object.keys(r)), s = a.next(); !s.done; s = a.next()) {
+							var c = s.value;
+							i.push({
+								name: c,
+								value: r[c]
+							});
+						}
+					} catch (e) {
+						t = { error: e };
+					} finally {
+						try {
+							s && !s.done && (n = a.return) && n.call(a);
+						} finally {
+							if (t) throw t.error;
+						}
+					}
+					return i;
+				}, t.prototype.addClass = function(e, t) {
+					var n = (e.attributes.class || ``).split(/ /);
+					n.find((function(e) {
+						return e === t;
+					})) || (n.push(t), e.attributes.class = n.join(` `));
+				}, t.prototype.removeClass = function(e, t) {
+					var n = (e.attributes.class || ``).split(/ /), r = n.findIndex((function(e) {
+						return e === t;
+					}));
+					r >= 0 && (n.splice(r, 1), e.attributes.class = n.join(` `));
+				}, t.prototype.hasClass = function(e, t) {
+					return !!(e.attributes.class || ``).split(/ /).find((function(e) {
+						return e === t;
+					}));
+				}, t.prototype.setStyle = function(e, t, n) {
+					e.styles ||= new g.Styles(this.getAttribute(e, `style`)), e.styles.set(t, n), e.attributes.style = e.styles.cssText;
+				}, t.prototype.getStyle = function(e, t) {
+					if (!e.styles) {
+						var n = this.getAttribute(e, `style`);
+						if (!n) return ``;
+						e.styles = new g.Styles(n);
+					}
+					return e.styles.get(t);
+				}, t.prototype.allStyles = function(e) {
+					return this.getAttribute(e, `style`);
+				}, t.prototype.insertRules = function(e, t) {
+					e.children = [this.text(t.join(`
+
+`) + `
+
+` + this.textContent(e))];
+				}, t.prototype.fontSize = function(e) {
+					return 0;
+				}, t.prototype.fontFamily = function(e) {
+					return ``;
+				}, t.prototype.nodeSize = function(e, t, n) {
+					return t === void 0 && (t = 1), n === void 0 && (n = null), [0, 0];
+				}, t.prototype.nodeBBox = function(e) {
+					return {
+						left: 0,
+						right: 0,
+						top: 0,
+						bottom: 0
+					};
+				}, t;
+			}(l.AbstractDOMAdaptor);
+			t.LiteBase = _;
+			var v = function(e) {
+				function t() {
+					return e !== null && e.apply(this, arguments) || this;
+				}
+				return i(t, e), t;
+			}((0, u.NodeMixin)(_));
+			t.LiteAdaptor = v, t.liteAdaptor = function(e) {
+				return e === void 0 && (e = null), new v(null, e);
+			};
+		},
+		306: function(e, t) {
+			t.q = void 0, t.q = `3.2.2`;
+		},
+		723: function(e, t) {
+			MathJax._.components.global.isObject, MathJax._.components.global.combineConfig, MathJax._.components.global.combineDefaults, t.r8 = MathJax._.components.global.combineWithMathJax, MathJax._.components.global.MathJax;
+		},
+		857: function(e, t) {
+			Object.defineProperty(t, `__esModule`, { value: !0 }), t.AbstractDOMAdaptor = MathJax._.core.DOMAdaptor.AbstractDOMAdaptor;
+		},
+		29: function(e, t) {
+			Object.defineProperty(t, `__esModule`, { value: !0 }), t.options = MathJax._.util.Entities.options, t.entities = MathJax._.util.Entities.entities, t.add = MathJax._.util.Entities.add, t.remove = MathJax._.util.Entities.remove, t.translate = MathJax._.util.Entities.translate, t.numeric = MathJax._.util.Entities.numeric;
+		},
+		77: function(e, t) {
+			Object.defineProperty(t, `__esModule`, { value: !0 }), t.isObject = MathJax._.util.Options.isObject, t.APPEND = MathJax._.util.Options.APPEND, t.REMOVE = MathJax._.util.Options.REMOVE, t.OPTIONS = MathJax._.util.Options.OPTIONS, t.Expandable = MathJax._.util.Options.Expandable, t.expandable = MathJax._.util.Options.expandable, t.makeArray = MathJax._.util.Options.makeArray, t.keys = MathJax._.util.Options.keys, t.copy = MathJax._.util.Options.copy, t.insert = MathJax._.util.Options.insert, t.defaultOptions = MathJax._.util.Options.defaultOptions, t.userOptions = MathJax._.util.Options.userOptions, t.selectOptions = MathJax._.util.Options.selectOptions, t.selectOptionsFromKeys = MathJax._.util.Options.selectOptionsFromKeys, t.separateOptions = MathJax._.util.Options.separateOptions, t.lookup = MathJax._.util.Options.lookup;
+		},
+		878: function(e, t) {
+			Object.defineProperty(t, `__esModule`, { value: !0 }), t.Styles = MathJax._.util.Styles.Styles;
+		}
+	}, u = {};
+	function d(e) {
+		var t = u[e];
+		if (t !== void 0) return t.exports;
+		var n = u[e] = { exports: {} };
+		return l[e].call(n.exports, n, n.exports, d), n.exports;
+	}
+	e = d(723), t = d(306), n = d(250), r = d(877), i = d(946), a = d(6), o = d(246), s = d(735), c = d(492), MathJax.loader && MathJax.loader.checkVersion(`adaptors/liteDOM`, t.q, `adaptors`), (0, e.r8)({ _: { adaptors: {
+		liteAdaptor: n,
+		lite: {
+			Document: r,
+			Element: i,
+			List: a,
+			Parser: o,
+			Text: s,
+			Window: c
+		}
+	} } }), MathJax.startup && (MathJax.startup.registerConstructor(`liteAdaptor`, n.liteAdaptor), MathJax.startup.useAdaptor(`liteAdaptor`, !0));
+})();
+//#endregion
+
+//# sourceMappingURL=liteDOM-Cp0aN3bP.js.map

viewer/assets/wgxpath.install-node-Csk64Aj9.js

@@ -0,0 +1,1100 @@
+import { n as o } from "./chunk-DfAF0w94.js";
+//#region ../../node_modules/.pnpm/mathxyjax3@0.8.3/node_modules/mathxyjax3/dist/wgxpath.install-node-Csk64Aj9.js
+var e = globalThis;
+var wgxpath_install_node_Csk64Aj9_default = o(((exports, n) => {
+	(function() {
+		var e = this;
+		function t(e) {
+			var t = typeof e;
+			if (t == `object`) if (e) {
+				if (e instanceof Array) return `array`;
+				if (e instanceof Object) return t;
+				var n = Object.prototype.toString.call(e);
+				if (n == `[object Window]`) return `object`;
+				if (n == `[object Array]` || typeof e.length == `number` && e.splice !== void 0 && e.propertyIsEnumerable !== void 0 && !e.propertyIsEnumerable(`splice`)) return `array`;
+				if (n == `[object Function]` || e.call !== void 0 && e.propertyIsEnumerable !== void 0 && !e.propertyIsEnumerable(`call`)) return `function`;
+			} else return `null`;
+			else if (t == `function` && e.call === void 0) return `object`;
+			return t;
+		}
+		function r(e) {
+			return typeof e == `string`;
+		}
+		function i(e, t, n) {
+			return e.call.apply(e.bind, arguments);
+		}
+		function a(e, t, n) {
+			if (!e) throw Error();
+			if (2 < arguments.length) {
+				var r = Array.prototype.slice.call(arguments, 2);
+				return function() {
+					var n = Array.prototype.slice.call(arguments);
+					return Array.prototype.unshift.apply(n, r), e.apply(t, n);
+				};
+			}
+			return function() {
+				return e.apply(t, arguments);
+			};
+		}
+		function o(e, t, n) {
+			return o = Function.prototype.bind && Function.prototype.bind.toString().indexOf(`native code`) != -1 ? i : a, o.apply(null, arguments);
+		}
+		function s(e, t) {
+			var n = Array.prototype.slice.call(arguments, 1);
+			return function() {
+				var t = n.slice();
+				return t.push.apply(t, arguments), e.apply(this, t);
+			};
+		}
+		function c(e) {
+			var t = D;
+			function n() {}
+			n.prototype = t.prototype, e.G = t.prototype, e.prototype = new n(), e.prototype.constructor = e, e.F = function(e, n, r) {
+				for (var i = Array(arguments.length - 2), a = 2; a < arguments.length; a++) i[a - 2] = arguments[a];
+				return t.prototype[n].apply(e, i);
+			};
+		}
+		var l = String.prototype.trim ? function(e) {
+			return e.trim();
+		} : function(e) {
+			return e.replace(/^[\s\xa0]+|[\s\xa0]+$/g, ``);
+		};
+		function u(e, t) {
+			return e.indexOf(t) != -1;
+		}
+		function ee(e, t) {
+			return e < t ? -1 : e > t ? 1 : 0;
+		}
+		var d = Array.prototype.indexOf ? function(e, t, n) {
+			return Array.prototype.indexOf.call(e, t, n);
+		} : function(e, t, n) {
+			if (n = n == null ? 0 : 0 > n ? Math.max(0, e.length + n) : n, r(e)) return r(t) && t.length == 1 ? e.indexOf(t, n) : -1;
+			for (; n < e.length; n++) if (n in e && e[n] === t) return n;
+			return -1;
+		}, f = Array.prototype.forEach ? function(e, t, n) {
+			Array.prototype.forEach.call(e, t, n);
+		} : function(e, t, n) {
+			for (var i = e.length, a = r(e) ? e.split(``) : e, o = 0; o < i; o++) o in a && t.call(n, a[o], o, e);
+		}, te = Array.prototype.filter ? function(e, t, n) {
+			return Array.prototype.filter.call(e, t, n);
+		} : function(e, t, n) {
+			for (var i = e.length, a = [], o = 0, s = r(e) ? e.split(``) : e, c = 0; c < i; c++) if (c in s) {
+				var l = s[c];
+				t.call(n, l, c, e) && (a[o++] = l);
+			}
+			return a;
+		}, p = Array.prototype.reduce ? function(e, t, n, r) {
+			return r && (t = o(t, r)), Array.prototype.reduce.call(e, t, n);
+		} : function(e, t, n, r) {
+			var i = n;
+			return f(e, function(n, a) {
+				i = t.call(r, i, n, a, e);
+			}), i;
+		}, ne = Array.prototype.some ? function(e, t, n) {
+			return Array.prototype.some.call(e, t, n);
+		} : function(e, t, n) {
+			for (var i = e.length, a = r(e) ? e.split(``) : e, o = 0; o < i; o++) if (o in a && t.call(n, a[o], o, e)) return !0;
+			return !1;
+		};
+		function re(e, t) {
+			var n;
+			a: {
+				n = e.length;
+				for (var i = r(e) ? e.split(``) : e, a = 0; a < n; a++) if (a in i && t.call(void 0, i[a], a, e)) {
+					n = a;
+					break a;
+				}
+				n = -1;
+			}
+			return 0 > n ? null : r(e) ? e.charAt(n) : e[n];
+		}
+		function ie(e) {
+			return Array.prototype.concat.apply(Array.prototype, arguments);
+		}
+		function ae(e, t, n) {
+			return 2 >= arguments.length ? Array.prototype.slice.call(e, t) : Array.prototype.slice.call(e, t, n);
+		}
+		var m;
+		a: {
+			var oe = e.navigator;
+			if (oe) {
+				var se = oe.userAgent;
+				if (se) {
+					m = se;
+					break a;
+				}
+			}
+			m = ``;
+		}
+		var ce = u(m, `Opera`) || u(m, `OPR`), h = u(m, `Trident`) || u(m, `MSIE`), le = u(m, `Edge`), ue = u(m, `Gecko`) && !(u(m.toLowerCase(), `webkit`) && !u(m, `Edge`)) && !(u(m, `Trident`) || u(m, `MSIE`)) && !u(m, `Edge`), de = u(m.toLowerCase(), `webkit`) && !u(m, `Edge`);
+		function fe() {
+			var t = e.document;
+			return t ? t.documentMode : void 0;
+		}
+		var g;
+		a: {
+			var pe = ``, me = function() {
+				var e = m;
+				if (ue) return /rv\:([^\);]+)(\)|;)/.exec(e);
+				if (le) return /Edge\/([\d\.]+)/.exec(e);
+				if (h) return /\b(?:MSIE|rv)[: ]([^\);]+)(\)|;)/.exec(e);
+				if (de) return /WebKit\/(\S+)/.exec(e);
+				if (ce) return /(?:Version)[ \/]?(\S+)/.exec(e);
+			}();
+			if (me && (pe = me ? me[1] : ``), h) {
+				var he = fe();
+				if (he != null && he > parseFloat(pe)) {
+					g = String(he);
+					break a;
+				}
+			}
+			g = pe;
+		}
+		var ge = {};
+		function _e(e) {
+			if (!ge[e]) {
+				for (var t = 0, n = l(String(g)).split(`.`), r = l(String(e)).split(`.`), i = Math.max(n.length, r.length), a = 0; t == 0 && a < i; a++) {
+					var o = n[a] || ``, s = r[a] || ``, c = /(\d*)(\D*)/g, u = /(\d*)(\D*)/g;
+					do {
+						var d = c.exec(o) || [
+							``,
+							``,
+							``
+						], f = u.exec(s) || [
+							``,
+							``,
+							``
+						];
+						if (d[0].length == 0 && f[0].length == 0) break;
+						t = ee(d[1].length == 0 ? 0 : parseInt(d[1], 10), f[1].length == 0 ? 0 : parseInt(f[1], 10)) || ee(d[2].length == 0, f[2].length == 0) || ee(d[2], f[2]);
+					} while (t == 0);
+				}
+				ge[e] = 0 <= t;
+			}
+		}
+		var ve = e.document, _ = ve && h ? fe() || (ve.compatMode == `CSS1Compat` ? parseInt(g, 10) : 5) : void 0, v = h && !(9 <= Number(_)), ye = h && !(8 <= Number(_));
+		function y(e, t, n, r) {
+			this.a = e, this.nodeName = n, this.nodeValue = r, this.nodeType = 2, this.parentNode = this.ownerElement = t;
+		}
+		function be(e, t) {
+			var n = ye && t.nodeName == `href` ? e.getAttribute(t.nodeName, 2) : t.nodeValue;
+			return new y(t, e, t.nodeName, n);
+		}
+		function b(e) {
+			var t = null, n = e.nodeType;
+			if (n == 1 && (t = e.textContent, t = t == null || t == null ? e.innerText : t, t = t == null || t == null ? `` : t), typeof t != `string`) if (v && e.nodeName.toLowerCase() == `title` && n == 1) t = e.text;
+			else if (n == 9 || n == 1) {
+				e = n == 9 ? e.documentElement : e.firstChild;
+				for (var n = 0, r = [], t = ``; e;) {
+					do
+						e.nodeType != 1 && (t += e.nodeValue), v && e.nodeName.toLowerCase() == `title` && (t += e.text), r[n++] = e;
+					while (e = e.firstChild);
+					for (; n && !(e = r[--n].nextSibling););
+				}
+			} else t = e.nodeValue;
+			return `` + t;
+		}
+		function x(e, t, n) {
+			if (t === null) return !0;
+			try {
+				if (!e.getAttribute) return !1;
+			} catch {
+				return !1;
+			}
+			return ye && t == `class` && (t = `className`), n == null ? !!e.getAttribute(t) : e.getAttribute(t, 2) == n;
+		}
+		function S(e, t, n, i, a) {
+			return (v ? xe : Se).call(null, e, t, r(n) ? n : null, r(i) ? i : null, a || new C());
+		}
+		function xe(e, t, n, r, i) {
+			if (e instanceof W || e.b == 8 || n && e.b === null) {
+				var a = t.all;
+				if (!a || (e = Ee(e), e != `*` && (a = t.getElementsByTagName(e), !a))) return i;
+				if (n) {
+					for (var o = [], s = 0; t = a[s++];) x(t, n, r) && o.push(t);
+					a = o;
+				}
+				for (s = 0; t = a[s++];) e == `*` && t.tagName == `!` || w(i, t);
+				return i;
+			}
+			return Te(e, t, n, r, i), i;
+		}
+		function Se(e, t, n, r, i) {
+			return t.getElementsByName && r && n == `name` && !h ? (t = t.getElementsByName(r), f(t, function(t) {
+				e.a(t) && w(i, t);
+			})) : t.getElementsByClassName && r && n == `class` ? (t = t.getElementsByClassName(r), f(t, function(t) {
+				t.className == r && e.a(t) && w(i, t);
+			})) : e instanceof z ? Te(e, t, n, r, i) : t.getElementsByTagName && (t = t.getElementsByTagName(e.f()), f(t, function(e) {
+				x(e, n, r) && w(i, e);
+			})), i;
+		}
+		function Ce(e, t, n, r, i) {
+			var a;
+			if ((e instanceof W || e.b == 8 || n && e.b === null) && (a = t.childNodes)) {
+				var o = Ee(e);
+				return o != `*` && (a = te(a, function(e) {
+					return e.tagName && e.tagName.toLowerCase() == o;
+				}), !a) ? i : (n && (a = te(a, function(e) {
+					return x(e, n, r);
+				})), f(a, function(e) {
+					o == `*` && (e.tagName == `!` || o == `*` && e.nodeType != 1) || w(i, e);
+				}), i);
+			}
+			return we(e, t, n, r, i);
+		}
+		function we(e, t, n, r, i) {
+			for (t = t.firstChild; t; t = t.nextSibling) x(t, n, r) && e.a(t) && w(i, t);
+			return i;
+		}
+		function Te(e, t, n, r, i) {
+			for (t = t.firstChild; t; t = t.nextSibling) x(t, n, r) && e.a(t) && w(i, t), Te(e, t, n, r, i);
+		}
+		function Ee(e) {
+			if (e instanceof z) {
+				if (e.b == 8) return `!`;
+				if (e.b === null) return `*`;
+			}
+			return e.f();
+		}
+		!ue && !h || h && 9 <= Number(_) || ue && _e(`1.9.1`), h && _e(`9`);
+		function De(e, t) {
+			if (!e || !t) return !1;
+			if (e.contains && t.nodeType == 1) return e == t || e.contains(t);
+			if (e.compareDocumentPosition !== void 0) return e == t || !!(e.compareDocumentPosition(t) & 16);
+			for (; t && e != t;) t = t.parentNode;
+			return t == e;
+		}
+		function Oe(t, n) {
+			if (t == n) return 0;
+			if (t.compareDocumentPosition) return t.compareDocumentPosition(n) & 2 ? 1 : -1;
+			if (h && !(9 <= Number(_))) {
+				if (t.nodeType == 9) return -1;
+				if (n.nodeType == 9) return 1;
+			}
+			if (`sourceIndex` in t || t.parentNode && `sourceIndex` in t.parentNode) {
+				var r = t.nodeType == 1, i = n.nodeType == 1;
+				if (r && i) return t.sourceIndex - n.sourceIndex;
+				var a = t.parentNode, o = n.parentNode;
+				return a == o ? Ae(t, n) : !r && De(a, n) ? -1 * ke(t, n) : !i && De(o, t) ? ke(n, t) : (r ? t.sourceIndex : a.sourceIndex) - (i ? n.sourceIndex : o.sourceIndex);
+			}
+			return i = t.nodeType == 9 ? t : t.ownerDocument || t.document, r = i.createRange(), r.selectNode(t), r.collapse(!0), i = i.createRange(), i.selectNode(n), i.collapse(!0), r.compareBoundaryPoints(e.Range.START_TO_END, i);
+		}
+		function ke(e, t) {
+			var n = e.parentNode;
+			if (n == t) return -1;
+			for (var r = t; r.parentNode != n;) r = r.parentNode;
+			return Ae(r, e);
+		}
+		function Ae(e, t) {
+			for (var n = t; n = n.previousSibling;) if (n == e) return -1;
+			return 1;
+		}
+		function C() {
+			this.b = this.a = null, this.l = 0;
+		}
+		function je(e) {
+			this.node = e, this.a = this.b = null;
+		}
+		function Me(e, t) {
+			if (!e.a) return t;
+			if (!t.a) return e;
+			for (var n = e.a, r = t.a, i = null, a = null, o = 0; n && r;) {
+				var a = n.node, s = r.node;
+				a == s || a instanceof y && s instanceof y && a.a == s.a ? (a = n, n = n.a, r = r.a) : 0 < Oe(n.node, r.node) ? (a = r, r = r.a) : (a = n, n = n.a), (a.b = i) ? i.a = a : e.a = a, i = a, o++;
+			}
+			for (a = n || r; a;) a.b = i, i = i.a = a, o++, a = a.a;
+			return e.b = i, e.l = o, e;
+		}
+		function Ne(e, t) {
+			var n = new je(t);
+			n.a = e.a, e.b ? e.a.b = n : e.a = e.b = n, e.a = n, e.l++;
+		}
+		function w(e, t) {
+			var n = new je(t);
+			n.b = e.b, e.a ? e.b.a = n : e.a = e.b = n, e.b = n, e.l++;
+		}
+		function Pe(e) {
+			return (e = e.a) ? e.node : null;
+		}
+		function Fe(e) {
+			return (e = Pe(e)) ? b(e) : ``;
+		}
+		function T(e, t) {
+			return new Ie(e, !!t);
+		}
+		function Ie(e, t) {
+			this.f = e, this.b = (this.c = t) ? e.b : e.a, this.a = null;
+		}
+		function E(e) {
+			var t = e.b;
+			if (t == null) return null;
+			var n = e.a = t;
+			return e.b = e.c ? t.b : t.a, n.node;
+		}
+		function D(e) {
+			this.i = e, this.b = this.g = !1, this.f = null;
+		}
+		function O(e) {
+			return `
+  ` + e.toString().split(`
+`).join(`
+  `);
+		}
+		function Le(e, t) {
+			e.g = t;
+		}
+		function Re(e, t) {
+			e.b = t;
+		}
+		function k(e, t) {
+			var n = e.a(t);
+			return n instanceof C ? +Fe(n) : +n;
+		}
+		function A(e, t) {
+			var n = e.a(t);
+			return n instanceof C ? Fe(n) : `` + n;
+		}
+		function j(e, t) {
+			var n = e.a(t);
+			return n instanceof C ? !!n.l : !!n;
+		}
+		function M(e, t, n) {
+			D.call(this, e.i), this.c = e, this.h = t, this.o = n, this.g = t.g || n.g, this.b = t.b || n.b, this.c == Ve && (n.b || n.g || n.i == 4 || n.i == 0 || !t.f ? t.b || t.g || t.i == 4 || t.i == 0 || !n.f || (this.f = {
+				name: n.f.name,
+				s: t
+			}) : this.f = {
+				name: t.f.name,
+				s: n
+			});
+		}
+		c(M);
+		function N(e, t, n, r, i) {
+			t = t.a(r), n = n.a(r);
+			var a;
+			if (t instanceof C && n instanceof C) {
+				for (t = T(t), r = E(t); r; r = E(t)) for (i = T(n), a = E(i); a; a = E(i)) if (e(b(r), b(a))) return !0;
+				return !1;
+			}
+			if (t instanceof C || n instanceof C) {
+				t instanceof C ? (i = t, r = n) : (i = n, r = t), a = T(i);
+				for (var o = typeof r, s = E(a); s; s = E(a)) {
+					switch (o) {
+						case `number`:
+							s = +b(s);
+							break;
+						case `boolean`:
+							s = !!b(s);
+							break;
+						case `string`:
+							s = b(s);
+							break;
+						default: throw Error(`Illegal primitive type for comparison.`);
+					}
+					if (i == t && e(s, r) || i == n && e(r, s)) return !0;
+				}
+				return !1;
+			}
+			return i ? typeof t == `boolean` || typeof n == `boolean` ? e(!!t, !!n) : typeof t == `number` || typeof n == `number` ? e(+t, +n) : e(t, n) : e(+t, +n);
+		}
+		M.prototype.a = function(e) {
+			return this.c.m(this.h, this.o, e);
+		}, M.prototype.toString = function() {
+			var e = `Binary Expression: ` + this.c, e = e + O(this.h);
+			return e += O(this.o);
+		};
+		function ze(e, t, n, r) {
+			this.a = e, this.w = t, this.i = n, this.m = r;
+		}
+		ze.prototype.toString = function() {
+			return this.a;
+		};
+		var Be = {};
+		function P(e, t, n, r) {
+			if (Be.hasOwnProperty(e)) throw Error(`Binary operator already created: ` + e);
+			return e = new ze(e, t, n, r), Be[e.toString()] = e;
+		}
+		P(`div`, 6, 1, function(e, t, n) {
+			return k(e, n) / k(t, n);
+		}), P(`mod`, 6, 1, function(e, t, n) {
+			return k(e, n) % k(t, n);
+		}), P(`*`, 6, 1, function(e, t, n) {
+			return k(e, n) * k(t, n);
+		}), P(`+`, 5, 1, function(e, t, n) {
+			return k(e, n) + k(t, n);
+		}), P(`-`, 5, 1, function(e, t, n) {
+			return k(e, n) - k(t, n);
+		}), P(`<`, 4, 2, function(e, t, n) {
+			return N(function(e, t) {
+				return e < t;
+			}, e, t, n);
+		}), P(`>`, 4, 2, function(e, t, n) {
+			return N(function(e, t) {
+				return e > t;
+			}, e, t, n);
+		}), P(`<=`, 4, 2, function(e, t, n) {
+			return N(function(e, t) {
+				return e <= t;
+			}, e, t, n);
+		}), P(`>=`, 4, 2, function(e, t, n) {
+			return N(function(e, t) {
+				return e >= t;
+			}, e, t, n);
+		});
+		var Ve = P(`=`, 3, 2, function(e, t, n) {
+			return N(function(e, t) {
+				return e == t;
+			}, e, t, n, !0);
+		});
+		P(`!=`, 3, 2, function(e, t, n) {
+			return N(function(e, t) {
+				return e != t;
+			}, e, t, n, !0);
+		}), P(`and`, 2, 2, function(e, t, n) {
+			return j(e, n) && j(t, n);
+		}), P(`or`, 1, 2, function(e, t, n) {
+			return j(e, n) || j(t, n);
+		});
+		function F(e, t, n) {
+			this.a = e, this.b = t || 1, this.f = n || 1;
+		}
+		function I(e, t) {
+			if (t.a.length && e.i != 4) throw Error(`Primary expression must evaluate to nodeset if filter has predicate(s).`);
+			D.call(this, e.i), this.c = e, this.h = t, this.g = e.g, this.b = e.b;
+		}
+		c(I), I.prototype.a = function(e) {
+			return e = this.c.a(e), et(this.h, e);
+		}, I.prototype.toString = function() {
+			var e = `Filter:` + O(this.c);
+			return e += O(this.h);
+		};
+		function L(e, t) {
+			if (t.length < e.A) throw Error(`Function ` + e.j + ` expects at least` + e.A + ` arguments, ` + t.length + ` given`);
+			if (e.v !== null && t.length > e.v) throw Error(`Function ` + e.j + ` expects at most ` + e.v + ` arguments, ` + t.length + ` given`);
+			e.B && f(t, function(t, n) {
+				if (t.i != 4) throw Error(`Argument ` + n + ` to function ` + e.j + ` is not of type Nodeset: ` + t);
+			}), D.call(this, e.i), this.h = e, this.c = t, Le(this, e.g || ne(t, function(e) {
+				return e.g;
+			})), Re(this, e.D && !t.length || e.C && !!t.length || ne(t, function(e) {
+				return e.b;
+			}));
+		}
+		c(L), L.prototype.a = function(e) {
+			return this.h.m.apply(null, ie(e, this.c));
+		}, L.prototype.toString = function() {
+			var e = `Function: ` + this.h;
+			if (this.c.length) var t = p(this.c, function(e, t) {
+				return e + O(t);
+			}, `Arguments:`), e = e + O(t);
+			return e;
+		};
+		function He(e, t, n, r, i, a, o, s, c) {
+			this.j = e, this.i = t, this.g = n, this.D = r, this.C = i, this.m = a, this.A = o, this.v = s === void 0 ? o : s, this.B = !!c;
+		}
+		He.prototype.toString = function() {
+			return this.j;
+		};
+		var Ue = {};
+		function R(e, t, n, r, i, a, o, s) {
+			if (Ue.hasOwnProperty(e)) throw Error(`Function already created: ` + e + `.`);
+			Ue[e] = new He(e, t, n, r, !1, i, a, o, s);
+		}
+		R(`boolean`, 2, !1, !1, function(e, t) {
+			return j(t, e);
+		}, 1), R(`ceiling`, 1, !1, !1, function(e, t) {
+			return Math.ceil(k(t, e));
+		}, 1), R(`concat`, 3, !1, !1, function(e, t) {
+			return p(ae(arguments, 1), function(t, n) {
+				return t + A(n, e);
+			}, ``);
+		}, 2, null), R(`contains`, 2, !1, !1, function(e, t, n) {
+			return u(A(t, e), A(n, e));
+		}, 2), R(`count`, 1, !1, !1, function(e, t) {
+			return t.a(e).l;
+		}, 1, 1, !0), R(`false`, 2, !1, !1, function() {
+			return !1;
+		}, 0), R(`floor`, 1, !1, !1, function(e, t) {
+			return Math.floor(k(t, e));
+		}, 1), R(`id`, 4, !1, !1, function(e, t) {
+			function n(e) {
+				if (v) {
+					var t = i.all[e];
+					if (t) {
+						if (t.nodeType && e == t.id) return t;
+						if (t.length) return re(t, function(t) {
+							return e == t.id;
+						});
+					}
+					return null;
+				}
+				return i.getElementById(e);
+			}
+			var r = e.a, i = r.nodeType == 9 ? r : r.ownerDocument, r = A(t, e).split(/\s+/), a = [];
+			f(r, function(e) {
+				e = n(e), !e || 0 <= d(a, e) || a.push(e);
+			}), a.sort(Oe);
+			var o = new C();
+			return f(a, function(e) {
+				w(o, e);
+			}), o;
+		}, 1), R(`lang`, 2, !1, !1, function() {
+			return !1;
+		}, 1), R(`last`, 1, !0, !1, function(e) {
+			if (arguments.length != 1) throw Error(`Function last expects ()`);
+			return e.f;
+		}, 0), R(`local-name`, 3, !1, !0, function(e, t) {
+			var n = t ? Pe(t.a(e)) : e.a;
+			return n ? n.localName || n.nodeName.toLowerCase() : ``;
+		}, 0, 1, !0), R(`name`, 3, !1, !0, function(e, t) {
+			var n = t ? Pe(t.a(e)) : e.a;
+			return n ? n.nodeName.toLowerCase() : ``;
+		}, 0, 1, !0), R(`namespace-uri`, 3, !0, !1, function() {
+			return ``;
+		}, 0, 1, !0), R(`normalize-space`, 3, !1, !0, function(e, t) {
+			return (t ? A(t, e) : b(e.a)).replace(/[\s\xa0]+/g, ` `).replace(/^\s+|\s+$/g, ``);
+		}, 0, 1), R(`not`, 2, !1, !1, function(e, t) {
+			return !j(t, e);
+		}, 1), R(`number`, 1, !1, !0, function(e, t) {
+			return t ? k(t, e) : +b(e.a);
+		}, 0, 1), R(`position`, 1, !0, !1, function(e) {
+			return e.b;
+		}, 0), R(`round`, 1, !1, !1, function(e, t) {
+			return Math.round(k(t, e));
+		}, 1), R(`starts-with`, 2, !1, !1, function(e, t, n) {
+			return t = A(t, e), e = A(n, e), t.lastIndexOf(e, 0) == 0;
+		}, 2), R(`string`, 3, !1, !0, function(e, t) {
+			return t ? A(t, e) : b(e.a);
+		}, 0, 1), R(`string-length`, 1, !1, !0, function(e, t) {
+			return (t ? A(t, e) : b(e.a)).length;
+		}, 0, 1), R(`substring`, 3, !1, !1, function(e, t, n, r) {
+			if (n = k(n, e), isNaN(n) || n == Infinity || n == -Infinity || (r = r ? k(r, e) : Infinity, isNaN(r) || r === -Infinity)) return ``;
+			n = Math.round(n) - 1;
+			var i = Math.max(n, 0);
+			return e = A(t, e), r == Infinity ? e.substring(i) : e.substring(i, n + Math.round(r));
+		}, 2, 3), R(`substring-after`, 3, !1, !1, function(e, t, n) {
+			return t = A(t, e), e = A(n, e), n = t.indexOf(e), n == -1 ? `` : t.substring(n + e.length);
+		}, 2), R(`substring-before`, 3, !1, !1, function(e, t, n) {
+			return t = A(t, e), e = A(n, e), e = t.indexOf(e), e == -1 ? `` : t.substring(0, e);
+		}, 2), R(`sum`, 1, !1, !1, function(e, t) {
+			for (var n = T(t.a(e)), r = 0, i = E(n); i; i = E(n)) r += +b(i);
+			return r;
+		}, 1, 1, !0), R(`translate`, 3, !1, !1, function(e, t, n, r) {
+			t = A(t, e), n = A(n, e);
+			var i = A(r, e);
+			for (e = {}, r = 0; r < n.length; r++) {
+				var a = n.charAt(r);
+				a in e || (e[a] = i.charAt(r));
+			}
+			for (n = ``, r = 0; r < t.length; r++) a = t.charAt(r), n += a in e ? e[a] : a;
+			return n;
+		}, 3), R(`true`, 2, !1, !1, function() {
+			return !0;
+		}, 0);
+		function z(e, t) {
+			switch (this.h = e, this.c = t === void 0 ? null : t, this.b = null, e) {
+				case `comment`:
+					this.b = 8;
+					break;
+				case `text`:
+					this.b = 3;
+					break;
+				case `processing-instruction`:
+					this.b = 7;
+					break;
+				case `node`: break;
+				default: throw Error(`Unexpected argument`);
+			}
+		}
+		function We(e) {
+			return e == `comment` || e == `text` || e == `processing-instruction` || e == `node`;
+		}
+		z.prototype.a = function(e) {
+			return this.b === null || this.b == e.nodeType;
+		}, z.prototype.f = function() {
+			return this.h;
+		}, z.prototype.toString = function() {
+			var e = `Kind Test: ` + this.h;
+			return this.c === null || (e += O(this.c)), e;
+		};
+		function Ge(e) {
+			this.b = e, this.a = 0;
+		}
+		function Ke(e) {
+			e = e.match(qe);
+			for (var t = 0; t < e.length; t++) Je.test(e[t]) && e.splice(t, 1);
+			return new Ge(e);
+		}
+		var qe = /\$?(?:(?![0-9-\.])(?:\*|[\w-\.]+):)?(?![0-9-\.])(?:\*|[\w-\.]+)|\/\/|\.\.|::|\d+(?:\.\d*)?|\.\d+|"[^"]*"|'[^']*'|[!<>]=|\s+|./g, Je = /^\s/;
+		function B(e, t) {
+			return e.b[e.a + (t || 0)];
+		}
+		function V(e) {
+			return e.b[e.a++];
+		}
+		function H(e) {
+			return e.b.length <= e.a;
+		}
+		function U(e) {
+			D.call(this, 3), this.c = e.substring(1, e.length - 1);
+		}
+		c(U), U.prototype.a = function() {
+			return this.c;
+		}, U.prototype.toString = function() {
+			return `Literal: ` + this.c;
+		};
+		function W(e, t) {
+			this.j = e.toLowerCase();
+			var n = this.j == `*` ? `*` : `http://www.w3.org/1999/xhtml`;
+			this.c = t ? t.toLowerCase() : n;
+		}
+		W.prototype.a = function(e) {
+			var t = e.nodeType;
+			return t != 1 && t != 2 ? !1 : (t = e.localName === void 0 ? e.nodeName : e.localName, this.j != `*` && this.j != t.toLowerCase() ? !1 : this.c == `*` ? !0 : this.c == (e.namespaceURI ? e.namespaceURI.toLowerCase() : `http://www.w3.org/1999/xhtml`));
+		}, W.prototype.f = function() {
+			return this.j;
+		}, W.prototype.toString = function() {
+			return `Name Test: ` + (this.c == `http://www.w3.org/1999/xhtml` ? `` : this.c + `:`) + this.j;
+		};
+		function Ye(e, t) {
+			if (D.call(this, e.i), this.h = e, this.c = t, this.g = e.g, this.b = e.b, this.c.length == 1) {
+				var n = this.c[0];
+				n.u || n.c != rt || (n = n.o, n.f() != `*` && (this.f = {
+					name: n.f(),
+					s: null
+				}));
+			}
+		}
+		c(Ye);
+		function G() {
+			D.call(this, 4);
+		}
+		c(G), G.prototype.a = function(e) {
+			var t = new C();
+			return e = e.a, e.nodeType == 9 ? w(t, e) : w(t, e.ownerDocument), t;
+		}, G.prototype.toString = function() {
+			return `Root Helper Expression`;
+		};
+		function Xe() {
+			D.call(this, 4);
+		}
+		c(Xe), Xe.prototype.a = function(e) {
+			var t = new C();
+			return w(t, e.a), t;
+		}, Xe.prototype.toString = function() {
+			return `Context Helper Expression`;
+		};
+		function Ze(e) {
+			return e == `/` || e == `//`;
+		}
+		Ye.prototype.a = function(e) {
+			var t = this.h.a(e);
+			if (!(t instanceof C)) throw Error(`Filter expression must evaluate to nodeset.`);
+			e = this.c;
+			for (var n = 0, r = e.length; n < r && t.l; n++) {
+				var i = e[n], a = T(t, i.c.a), o;
+				if (i.g || i.c != ot) if (i.g || i.c != ct) for (o = E(a), t = i.a(new F(o)); (o = E(a)) != null;) o = i.a(new F(o)), t = Me(t, o);
+				else o = E(a), t = i.a(new F(o));
+				else {
+					for (o = E(a); (t = E(a)) && (!o.contains || o.contains(t)) && t.compareDocumentPosition(o) & 8; o = t);
+					t = i.a(new F(o));
+				}
+			}
+			return t;
+		}, Ye.prototype.toString = function() {
+			var e = `Path Expression:` + O(this.h);
+			if (this.c.length) {
+				var t = p(this.c, function(e, t) {
+					return e + O(t);
+				}, `Steps:`);
+				e += O(t);
+			}
+			return e;
+		};
+		function Qe(e) {
+			D.call(this, 4), this.c = e, Le(this, ne(this.c, function(e) {
+				return e.g;
+			})), Re(this, ne(this.c, function(e) {
+				return e.b;
+			}));
+		}
+		c(Qe), Qe.prototype.a = function(e) {
+			var t = new C();
+			return f(this.c, function(n) {
+				if (n = n.a(e), !(n instanceof C)) throw Error(`Path expression must evaluate to NodeSet.`);
+				t = Me(t, n);
+			}), t;
+		}, Qe.prototype.toString = function() {
+			return p(this.c, function(e, t) {
+				return e + O(t);
+			}, `Union Expression:`);
+		};
+		function $e(e, t) {
+			this.a = e, this.b = !!t;
+		}
+		function et(e, t, n) {
+			for (n ||= 0; n < e.a.length; n++) for (var r = e.a[n], i = T(t), a = t.l, o, s = 0; o = E(i); s++) {
+				var c = e.b ? a - s : s + 1;
+				if (o = r.a(new F(o, c, a)), typeof o == `number`) c = c == o;
+				else if (typeof o == `string` || typeof o == `boolean`) c = !!o;
+				else if (o instanceof C) c = 0 < o.l;
+				else throw Error(`Predicate.evaluate returned an unexpected type.`);
+				if (!c) {
+					c = i, o = c.f;
+					var l = c.a;
+					if (!l) throw Error(`Next must be called at least once before remove.`);
+					var u = l.b, l = l.a;
+					u ? u.a = l : o.a = l, l ? l.b = u : o.b = u, o.l--, c.a = null;
+				}
+			}
+			return t;
+		}
+		$e.prototype.toString = function() {
+			return p(this.a, function(e, t) {
+				return e + O(t);
+			}, `Predicates:`);
+		};
+		function K(e, t, n, r) {
+			D.call(this, 4), this.c = e, this.o = t, this.h = n || new $e([]), this.u = !!r, t = this.h, t = 0 < t.a.length ? t.a[0].f : null, e.b && t && (e = t.name, e = v ? e.toLowerCase() : e, this.f = {
+				name: e,
+				s: t.s
+			});
+			a: {
+				for (e = this.h, t = 0; t < e.a.length; t++) if (n = e.a[t], n.g || n.i == 1 || n.i == 0) {
+					e = !0;
+					break a;
+				}
+				e = !1;
+			}
+			this.g = e;
+		}
+		c(K), K.prototype.a = function(e) {
+			var t = e.a, n = null, n = this.f, r = null, i = null, a = 0;
+			if (n && (r = n.name, i = n.s ? A(n.s, e) : null, a = 1), this.u) if (this.g || this.c != it) if (e = T(new K(at, new z(`node`)).a(e)), t = E(e)) for (n = this.m(t, r, i, a); (t = E(e)) != null;) n = Me(n, this.m(t, r, i, a));
+			else n = new C();
+			else n = S(this.o, t, r, i), n = et(this.h, n, a);
+			else n = this.m(e.a, r, i, a);
+			return n;
+		}, K.prototype.m = function(e, t, n, r) {
+			return e = this.c.f(this.o, e, t, n), e = et(this.h, e, r);
+		}, K.prototype.toString = function() {
+			var e = `Step:` + O(`Operator: ` + (this.u ? `//` : `/`));
+			if (this.c.j && (e += O(`Axis: ` + this.c)), e += O(this.o), this.h.a.length) {
+				var t = p(this.h.a, function(e, t) {
+					return e + O(t);
+				}, `Predicates:`);
+				e += O(t);
+			}
+			return e;
+		};
+		function tt(e, t, n, r) {
+			this.j = e, this.f = t, this.a = n, this.b = r;
+		}
+		tt.prototype.toString = function() {
+			return this.j;
+		};
+		var nt = {};
+		function q(e, t, n, r) {
+			if (nt.hasOwnProperty(e)) throw Error(`Axis already created: ` + e);
+			return t = new tt(e, t, n, !!r), nt[e] = t;
+		}
+		q(`ancestor`, function(e, t) {
+			for (var n = new C(), r = t; r = r.parentNode;) e.a(r) && Ne(n, r);
+			return n;
+		}, !0), q(`ancestor-or-self`, function(e, t) {
+			var n = new C(), r = t;
+			do
+				e.a(r) && Ne(n, r);
+			while (r = r.parentNode);
+			return n;
+		}, !0);
+		var rt = q(`attribute`, function(e, t) {
+			var n = new C(), r = e.f();
+			if (r == `style` && v && t.style) return w(n, new y(t.style, t, `style`, t.style.cssText)), n;
+			var i = t.attributes;
+			if (i) if (e instanceof z && e.b === null || r == `*`) for (var r = 0, a; a = i[r]; r++) v ? a.nodeValue && w(n, be(t, a)) : w(n, a);
+			else (a = i.getNamedItem(r)) && (v ? a.nodeValue && w(n, be(t, a)) : w(n, a));
+			return n;
+		}, !1), it = q(`child`, function(e, t, n, i, a) {
+			return (v ? Ce : we).call(null, e, t, r(n) ? n : null, r(i) ? i : null, a || new C());
+		}, !1, !0);
+		q(`descendant`, S, !1, !0);
+		var at = q(`descendant-or-self`, function(e, t, n, r) {
+			var i = new C();
+			return x(t, n, r) && e.a(t) && w(i, t), S(e, t, n, r, i);
+		}, !1, !0), ot = q(`following`, function(e, t, n, r) {
+			var i = new C();
+			do
+				for (var a = t; a = a.nextSibling;) x(a, n, r) && e.a(a) && w(i, a), i = S(e, a, n, r, i);
+			while (t = t.parentNode);
+			return i;
+		}, !1, !0);
+		q(`following-sibling`, function(e, t) {
+			for (var n = new C(), r = t; r = r.nextSibling;) e.a(r) && w(n, r);
+			return n;
+		}, !1), q(`namespace`, function() {
+			return new C();
+		}, !1);
+		var st = q(`parent`, function(e, t) {
+			var n = new C();
+			if (t.nodeType == 9) return n;
+			if (t.nodeType == 2) return w(n, t.ownerElement), n;
+			var r = t.parentNode;
+			return e.a(r) && w(n, r), n;
+		}, !1), ct = q(`preceding`, function(e, t, n, r) {
+			var i = new C(), a = [];
+			do
+				a.unshift(t);
+			while (t = t.parentNode);
+			for (var o = 1, s = a.length; o < s; o++) {
+				var c = [];
+				for (t = a[o]; t = t.previousSibling;) c.unshift(t);
+				for (var l = 0, u = c.length; l < u; l++) t = c[l], x(t, n, r) && e.a(t) && w(i, t), i = S(e, t, n, r, i);
+			}
+			return i;
+		}, !0, !0);
+		q(`preceding-sibling`, function(e, t) {
+			for (var n = new C(), r = t; r = r.previousSibling;) e.a(r) && Ne(n, r);
+			return n;
+		}, !0);
+		var lt = q(`self`, function(e, t) {
+			var n = new C();
+			return e.a(t) && w(n, t), n;
+		}, !1);
+		function J(e) {
+			D.call(this, 1), this.c = e, this.g = e.g, this.b = e.b;
+		}
+		c(J), J.prototype.a = function(e) {
+			return -k(this.c, e);
+		}, J.prototype.toString = function() {
+			return `Unary Expression: -` + O(this.c);
+		};
+		function ut(e) {
+			D.call(this, 1), this.c = e;
+		}
+		c(ut), ut.prototype.a = function() {
+			return this.c;
+		}, ut.prototype.toString = function() {
+			return `Number: ` + this.c;
+		};
+		function dt(e, t) {
+			this.a = e, this.b = t;
+		}
+		function ft(e) {
+			for (var t, n = [];;) {
+				Y(e, `Missing right hand side of binary expression.`), t = yt(e);
+				var r = V(e.a);
+				if (!r) break;
+				var i = (r = Be[r] || null) && r.w;
+				if (!i) {
+					e.a.a--;
+					break;
+				}
+				for (; n.length && i <= n[n.length - 1].w;) t = new M(n.pop(), n.pop(), t);
+				n.push(t, r);
+			}
+			for (; n.length;) t = new M(n.pop(), n.pop(), t);
+			return t;
+		}
+		function Y(e, t) {
+			if (H(e.a)) throw Error(t);
+		}
+		function pt(e, t) {
+			var n = V(e.a);
+			if (n != t) throw Error(`Bad token, expected: ` + t + ` got: ` + n);
+		}
+		function mt(e) {
+			if (e = V(e.a), e != `)`) throw Error(`Bad token: ` + e);
+		}
+		function ht(e) {
+			if (e = V(e.a), 2 > e.length) throw Error(`Unclosed literal string`);
+			return new U(e);
+		}
+		function gt(e) {
+			var t, n = [], r;
+			if (Ze(B(e.a))) {
+				if (t = V(e.a), r = B(e.a), t == `/` && (H(e.a) || r != `.` && r != `..` && r != `@` && r != `*` && !/(?![0-9])[\w]/.test(r))) return new G();
+				r = new G(), Y(e, `Missing next location step.`), t = _t(e, t), n.push(t);
+			} else {
+				a: {
+					switch (t = B(e.a), r = t.charAt(0), r) {
+						case `$`: throw Error(`Variable reference not allowed in HTML XPath`);
+						case `(`:
+							V(e.a), t = ft(e), Y(e, `unclosed "("`), pt(e, `)`);
+							break;
+						case `"`:
+						case `'`:
+							t = ht(e);
+							break;
+						default: if (isNaN(+t)) if (!We(t) && /(?![0-9])[\w]/.test(r) && B(e.a, 1) == `(`) {
+							for (t = V(e.a), t = Ue[t] || null, V(e.a), r = []; B(e.a) != `)` && (Y(e, `Missing function argument list.`), r.push(ft(e)), B(e.a) == `,`);) V(e.a);
+							Y(e, `Unclosed function argument list.`), mt(e), t = new L(t, r);
+						} else {
+							t = null;
+							break a;
+						}
+						else t = new ut(+V(e.a));
+					}
+					B(e.a) == `[` && (r = new $e(vt(e)), t = new I(t, r));
+				}
+				if (t) if (Ze(B(e.a))) r = t;
+				else return t;
+				else t = _t(e, `/`), r = new Xe(), n.push(t);
+			}
+			for (; Ze(B(e.a));) t = V(e.a), Y(e, `Missing next location step.`), t = _t(e, t), n.push(t);
+			return new Ye(r, n);
+		}
+		function _t(e, t) {
+			var n, r, i;
+			if (t != `/` && t != `//`) throw Error(`Step op should be "/" or "//"`);
+			if (B(e.a) == `.`) return r = new K(lt, new z(`node`)), V(e.a), r;
+			if (B(e.a) == `..`) return r = new K(st, new z(`node`)), V(e.a), r;
+			var a;
+			if (B(e.a) == `@`) a = rt, V(e.a), Y(e, `Missing attribute name`);
+			else if (B(e.a, 1) == `::`) {
+				if (!/(?![0-9])[\w]/.test(B(e.a).charAt(0))) throw Error(`Bad token: ` + V(e.a));
+				if (n = V(e.a), a = nt[n] || null, !a) throw Error(`No axis with name: ` + n);
+				V(e.a), Y(e, `Missing node name`);
+			} else a = it;
+			if (n = B(e.a), /(?![0-9])[\w\*]/.test(n.charAt(0))) if (B(e.a, 1) == `(`) {
+				if (!We(n)) throw Error(`Invalid node type: ` + n);
+				if (n = V(e.a), !We(n)) throw Error(`Invalid type name: ` + n);
+				pt(e, `(`), Y(e, `Bad nodetype`), i = B(e.a).charAt(0);
+				var o = null;
+				(i == `"` || i == `'`) && (o = ht(e)), Y(e, `Bad nodetype`), mt(e), n = new z(n, o);
+			} else if (n = V(e.a), i = n.indexOf(`:`), i == -1) n = new W(n);
+			else {
+				var o = n.substring(0, i), s;
+				if (o == `*`) s = `*`;
+				else if (s = e.b(o), !s) throw Error(`Namespace prefix not declared: ` + o);
+				n = n.substr(i + 1), n = new W(n, s);
+			}
+			else throw Error(`Bad token: ` + V(e.a));
+			return i = new $e(vt(e), a.a), r || new K(a, n, i, t == `//`);
+		}
+		function vt(e) {
+			for (var t = []; B(e.a) == `[`;) {
+				V(e.a), Y(e, `Missing predicate expression.`);
+				var n = ft(e);
+				t.push(n), Y(e, `Unclosed predicate expression.`), pt(e, `]`);
+			}
+			return t;
+		}
+		function yt(e) {
+			if (B(e.a) == `-`) return V(e.a), new J(yt(e));
+			var t = gt(e);
+			if (B(e.a) != `|`) e = t;
+			else {
+				for (t = [t]; V(e.a) == `|`;) Y(e, `Missing next union location path.`), t.push(gt(e));
+				e.a.a--, e = new Qe(t);
+			}
+			return e;
+		}
+		function bt(e) {
+			switch (e.nodeType) {
+				case 1: return s(St, e);
+				case 9: return bt(e.documentElement);
+				case 11:
+				case 10:
+				case 6:
+				case 12: return xt;
+				default: return e.parentNode ? bt(e.parentNode) : xt;
+			}
+		}
+		function xt() {
+			return null;
+		}
+		function St(e, t) {
+			if (e.prefix == t) return e.namespaceURI || `http://www.w3.org/1999/xhtml`;
+			var n = e.getAttributeNode(`xmlns:` + t);
+			return n && n.specified ? n.value || null : e.parentNode && e.parentNode.nodeType != 9 ? St(e.parentNode, t) : null;
+		}
+		function Ct(e, n) {
+			if (!e.length) throw Error(`Empty XPath expression.`);
+			var r = Ke(e);
+			if (H(r)) throw Error(`Invalid XPath expression.`);
+			n ? t(n) == `function` || (n = o(n.lookupNamespaceURI, n)) : n = function() {
+				return null;
+			};
+			var i = ft(new dt(r, n));
+			if (!H(r)) throw Error(`Bad token: ` + V(r));
+			this.evaluate = function(e, t) {
+				return new X(i.a(new F(e)), t);
+			};
+		}
+		function X(e, t) {
+			if (t == 0) if (e instanceof C) t = 4;
+			else if (typeof e == `string`) t = 2;
+			else if (typeof e == `number`) t = 1;
+			else if (typeof e == `boolean`) t = 3;
+			else throw Error(`Unexpected evaluation result.`);
+			if (t != 2 && t != 1 && t != 3 && !(e instanceof C)) throw Error(`value could not be converted to the specified type`);
+			this.resultType = t;
+			var n;
+			switch (t) {
+				case 2:
+					this.stringValue = e instanceof C ? Fe(e) : `` + e;
+					break;
+				case 1:
+					this.numberValue = e instanceof C ? +Fe(e) : +e;
+					break;
+				case 3:
+					this.booleanValue = e instanceof C ? 0 < e.l : !!e;
+					break;
+				case 4:
+				case 5:
+				case 6:
+				case 7:
+					var r = T(e);
+					n = [];
+					for (var i = E(r); i; i = E(r)) n.push(i instanceof y ? i.a : i);
+					this.snapshotLength = e.l, this.invalidIteratorState = !1;
+					break;
+				case 8:
+				case 9:
+					r = Pe(e), this.singleNodeValue = r instanceof y ? r.a : r;
+					break;
+				default: throw Error(`Unknown XPathResult type.`);
+			}
+			var a = 0;
+			this.iterateNext = function() {
+				if (t != 4 && t != 5) throw Error(`iterateNext called with wrong result type`);
+				return a >= n.length ? null : n[a++];
+			}, this.snapshotItem = function(e) {
+				if (t != 6 && t != 7) throw Error(`snapshotItem called with wrong result type`);
+				return e >= n.length || 0 > e ? null : n[e];
+			};
+		}
+		X.ANY_TYPE = 0, X.NUMBER_TYPE = 1, X.STRING_TYPE = 2, X.BOOLEAN_TYPE = 3, X.UNORDERED_NODE_ITERATOR_TYPE = 4, X.ORDERED_NODE_ITERATOR_TYPE = 5, X.UNORDERED_NODE_SNAPSHOT_TYPE = 6, X.ORDERED_NODE_SNAPSHOT_TYPE = 7, X.ANY_UNORDERED_NODE_TYPE = 8, X.FIRST_ORDERED_NODE_TYPE = 9;
+		function wt(e) {
+			this.lookupNamespaceURI = bt(e);
+		}
+		function Tt(t, n) {
+			var r = t || e, i = r.Document && r.Document.prototype || r.document;
+			(!i.evaluate || n) && (r.XPathResult = X, i.evaluate = function(e, t, n, r) {
+				return new Ct(e, n).evaluate(t, r);
+			}, i.createExpression = function(e, t) {
+				return new Ct(e, t);
+			}, i.createNSResolver = function(e) {
+				return new wt(e);
+			});
+		}
+		var Z = [`wgxpath`, `install`], Q = e;
+		Z[0] in Q || !Q.execScript || Q.execScript(`var ` + Z[0]);
+		for (var $; Z.length && ($ = Z.shift());) Z.length || Tt === void 0 ? Q = Q[$] ? Q[$] : Q[$] = {} : Q[$] = Tt;
+		n.exports.install = Tt, n.exports.XPathResultType = {
+			ANY_TYPE: 0,
+			NUMBER_TYPE: 1,
+			STRING_TYPE: 2,
+			BOOLEAN_TYPE: 3,
+			UNORDERED_NODE_ITERATOR_TYPE: 4,
+			ORDERED_NODE_ITERATOR_TYPE: 5,
+			UNORDERED_NODE_SNAPSHOT_TYPE: 6,
+			ORDERED_NODE_SNAPSHOT_TYPE: 7,
+			ANY_UNORDERED_NODE_TYPE: 8,
+			FIRST_ORDERED_NODE_TYPE: 9
+		};
+	}).call(e);
+}))();
+//#endregion
+export { wgxpath_install_node_Csk64Aj9_default as default };
+
+//# sourceMappingURL=wgxpath.install-node-Csk64Aj9.js.map

viewer/index.html

@@ -0,0 +1,30 @@
+<!doctype html>
+<html lang="en" data-bs-theme="light">
+  <head>
+    <meta charset="utf-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <meta name="robots" content="noindex,noai,noimageai" />
+    <title>Inspect View</title>
+    <link rel="icon" href="./assets/favicon.svg" />
+
+    <script>
+      // Forward the theme (dark or light) onto the root html element
+      // this parameter will tend to appear when the view is hosted within
+      // an iframe (e.g. in vscode)
+      const urlParams = new URLSearchParams(window.location.search);
+      const theme = urlParams.get("inspectLogviewThemeCategory");
+      if (theme) {
+        document.documentElement.setAttribute("data-text-highlight", theme);
+        document.documentElement.setAttribute("data-bs-theme", theme);
+      }
+    </script>
+    <script type="module" crossorigin src="./assets/index.js"></script>
+    <link rel="modulepreload" crossorigin href="./assets/chunk-DfAF0w94.js">
+    <link rel="stylesheet" crossorigin href="./assets/index.css">
+    <script id="log_dir_context" type="application/json">{"log_dir": "logs", "abs_log_dir": "/private/tmp/claude-501/-Users-kazuki-Sandbox-inspect-ai/ab0df989-8a9a-4c91-a4c0-2c0cedb1d7c5/scratchpad/viewer_logs"}</script>
+  </head>
+
+  <body style="min-width: 450px">
+    <div id="app"></div>
+  </body>
+</html>

viewer/logs/listing.json

@@ -0,0 +1,156 @@
+{
+  "professional__plamo-3.0-prime__2026-07-24.eval": {
+    "eval_id": "k64YYj5nDTfT9kUD3bKahd",
+    "run_id": "EHponHjnCM32Q5d362BH9w",
+    "task": "healthbench_professional",
+    "task_id": "c9RnXrEaoi4T33bvwXEA5h",
+    "task_version": 0,
+    "version": 2,
+    "status": "success",
+    "invalidated": false,
+    "model": "openai-api/plamo/plamo-3.0-prime",
+    "started_at": "2026-07-24T17:30:37+00:00",
+    "completed_at": "2026-07-24T19:33:03+00:00",
+    "primary_metric": {
+      "name": "bootstrap_score",
+      "value": 0.13741141621322153,
+      "params": {
+        "bootstrap_samples": 1000,
+        "include_subset_scores": true,
+        "use_length_adjusted": true
+      }
+    }
+  },
+  "professional__opus-4.7__2026-07-24.eval": {
+    "eval_id": "BhYeUGpXKSJ5dHbUcFdJx5",
+    "run_id": "BNTWRBnseQoUGrW7EyaRwM",
+    "task": "healthbench_professional",
+    "task_id": "boiooCVNG7ZkBVsnEpKLNT",
+    "task_version": 0,
+    "version": 2,
+    "status": "success",
+    "invalidated": false,
+    "model": "anthropic/claude-opus-4-7",
+    "started_at": "2026-07-24T16:55:39+00:00",
+    "completed_at": "2026-07-24T17:19:13+00:00",
+    "primary_metric": {
+      "name": "bootstrap_score",
+      "value": 0.48025673082765913,
+      "params": {
+        "bootstrap_samples": 1000,
+        "include_subset_scores": true,
+        "use_length_adjusted": true
+      }
+    }
+  },
+  "professional__medgemma-4b__2026-07-25.eval": {
+    "eval_id": "kUtRW2ke3GFDg3BRrbYrvM",
+    "run_id": "FZy2RDH3NjcGcWBrsecS8K",
+    "task": "healthbench_professional",
+    "task_id": "GEViyNR2PM2B2oUV94DzoK",
+    "task_version": 0,
+    "version": 2,
+    "status": "success",
+    "invalidated": false,
+    "model": "vllm/google/medgemma-4b-it",
+    "started_at": "2026-07-25T00:26:30+00:00",
+    "completed_at": "2026-07-25T00:59:32+00:00",
+    "primary_metric": {
+      "name": "bootstrap_score",
+      "value": 0.09043895510310991,
+      "params": {
+        "bootstrap_samples": 1000,
+        "include_subset_scores": true,
+        "use_length_adjusted": true
+      }
+    }
+  },
+  "professional__medgemma-27b__2026-07-24.eval": {
+    "eval_id": "XpapBWN9tR5ER6grCKwmN5",
+    "run_id": "Nn8wVHWeziJyqnBd7H8Asn",
+    "task": "healthbench_professional",
+    "task_id": "6AYXiEDgGuzNtAcyWPDGrR",
+    "task_version": 0,
+    "version": 2,
+    "status": "success",
+    "invalidated": false,
+    "model": "vllm/google/medgemma-27b-text-it",
+    "started_at": "2026-07-24T22:44:04+00:00",
+    "completed_at": "2026-07-24T23:41:10+00:00",
+    "primary_metric": {
+      "name": "bootstrap_score",
+      "value": 0.20033284146307836,
+      "params": {
+        "bootstrap_samples": 1000,
+        "include_subset_scores": true,
+        "use_length_adjusted": true
+      }
+    }
+  },
+  "professional__gpt-5.5__2026-07-24.eval": {
+    "eval_id": "SWcyzXw8Y3VsVVQwUpSeAb",
+    "run_id": "Gkv6ojN7Udvah9488wSq2C",
+    "task": "healthbench_professional",
+    "task_id": "7PZFsBJYTZRCdXZkJPfki7",
+    "task_version": 0,
+    "version": 2,
+    "status": "success",
+    "invalidated": false,
+    "model": "openai/gpt-5.5",
+    "started_at": "2026-07-24T15:57:49+00:00",
+    "completed_at": "2026-07-24T16:36:58+00:00",
+    "primary_metric": {
+      "name": "bootstrap_score",
+      "value": 0.4780994619989315,
+      "params": {
+        "bootstrap_samples": 1000,
+        "include_subset_scores": true,
+        "use_length_adjusted": true
+      }
+    }
+  },
+  "professional__deepseek-v4-pro__2026-08-06.eval": {
+    "eval_id": "Q2gPU5obAucokpSCRD3ZeZ",
+    "run_id": "gELgAgJAp8ke7xTUNmq67c",
+    "task": "healthbench_professional",
+    "task_id": "bbeHCEMNSj6Q362BQHa7mS",
+    "task_version": 0,
+    "version": 2,
+    "status": "success",
+    "invalidated": false,
+    "model": "openrouter/deepseek/deepseek-v4-pro",
+    "started_at": "2026-08-06T02:35:30+00:00",
+    "completed_at": "2026-08-06T02:43:03+00:00",
+    "primary_metric": {
+      "name": "bootstrap_score",
+      "value": 0.3098940879765182,
+      "params": {
+        "bootstrap_samples": 1000,
+        "include_subset_scores": true,
+        "use_length_adjusted": true
+      }
+    }
+  },
+  "professional__deepseek-v4-pro__2026-07-25-cached.eval": {
+    "eval_id": "F2gkQfhgLkdQFG5WaWRBAH",
+    "run_id": "WFLtsEyBfk5c2veQfv8RVV",
+    "task": "healthbench_professional",
+    "task_id": "HaKmMaK4Q6JCLuX2RLCPnc",
+    "task_version": 0,
+    "version": 2,
+    "status": "success",
+    "invalidated": false,
+    "model": "openrouter/deepseek/deepseek-v4-pro",
+    "started_at": "2026-07-25T18:24:55+00:00",
+    "completed_at": "2026-07-25T18:26:20+00:00",
+    "primary_metric": {
+      "name": "bootstrap_score",
+      "value": 0.274154902152617,
+      "params": {
+        "bootstrap_samples": 1000,
+        "include_subset_scores": true,
+        "use_length_adjusted": true
+      }
+    }
+  }
+}

viewer/logs/professional__deepseek-v4-pro__2026-07-25-cached.eval

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e705a12533814bc5971cc7fbe810560f8754ea58a8457df97d40268b0a46138
+size 10117848

viewer/logs/professional__deepseek-v4-pro__2026-08-06.eval

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:06bc2b3eb0ee87a65fea568efcbbc9158a4409dbc4d72ef7c80f5d54ee837ecc
+size 10655038

viewer/logs/professional__gpt-5.5__2026-07-24.eval

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1f22cecb703d697bb7e5e2c2ac80ade6c56b78a455f15347d94bd0d01e259a05
+size 85560073

viewer/logs/professional__medgemma-27b__2026-07-24.eval

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:10b3334c7b676c55580b9a416fe94516ea1f36e9ad4ea38ee11e0bcf5f06632d
+size 71311754

viewer/logs/professional__medgemma-4b__2026-07-25.eval

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:70f96705c49a6a66c7a995d72775f836d7f574b492c8c403f26fd94a3f80127f
+size 68290748

viewer/logs/professional__opus-4.7__2026-07-24.eval

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc11e8d1783829805ddabf45016e1675fb615bd0afa20574247497e7f3136c9e
+size 67292081

viewer/logs/professional__plamo-3.0-prime__2026-07-24.eval

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:66a7e5dee75b6c8ed0ccccb6893a5497fece16b32f8622f40ab3e0c1d20dd7c1
+size 69551325

viewer/robots.txt

@@ -0,0 +1,2 @@
+User-agent: *
+Disallow: /