Spaces:
Running
Consolidate hierarchy terminology + handle v2 hierarchy shape
Browse filesTwo threads in one change.
(1) v2 backend snapshot ships families[].composites[].benchmarks[]
instead of families[].leaves[]. The family-table only walked .leaves,
so every aggregator family rendered as a single non-expandable row
and clicking it navigated to one arbitrary child (e.g. clicking BFCL
landed on bfcl-live).
(2) The codebase mixed "Suite/Subtask" labels with "Composite/Slice"
data fields. Consolidate to one term per level so the UI and the
schema agree.
Hierarchy: 5 levels. Family -> Composite -> Benchmark -> Slice ->
Metric. "Suite" and "Subtask" are gone everywhere a human reads them.
Backend-shipped JSON keys (subtask_key, subtask_name,
metric.scope === "subtask", composite_benchmark_*) are kept since
they are wire contracts.
Family-table fix
- collectLeafEntries now treats nested benchmarks as the primary
source: walks family.standalone_benchmarks, family.benchmarks, and
family.composites[].benchmarks[] before falling back to the legacy
family.leaves[]. Each nested benchmark's summary_eval_ids[0] becomes
the navigation target, with `${family_key}_${benchmark_key}` as a
fallback when summary_eval_ids is missing.
- Verified end-to-end against the v2 snapshot
(j-chim/temp_evalcard_backend/warehouse/latest): bfcl now renders
"4 benchmarks" with the chevron, and clicking expands to BFCL Live /
BFCL Multi-turn / BFCL Non-live / BFCL Web Search.
Domain extraction in /evals
- The familyDomains and domainCounts memos walk the same nested
benchmarks before falling back to leaves, so the "Filter by
domain" pill list populates correctly under v2.
Hierarchy view
- Home page hierarchy strip collapses Split + Subtask into a single
"Slice" tier (5 levels instead of 6). Lede, kicker, OG image,
layout metadata description all updated to "five-level" to match.
Terminology rename (UI labels + local code identifiers)
- "Suite" / "Suites" -> "Composite" / "Composites"
- "Subtask" / "subtask" -> "Slice" / "slice"
- Renamed local identifiers in benchmark-detail.tsx and
eval-detail.tsx: SuiteGroup -> CompositeGroup,
getSuiteBadgeMeta -> getCompositeBadgeMeta,
normalizeSuiteKey -> normalizeCompositeKey,
groupBySuite -> groupByComposite,
getSuiteKey/Name/DisplayName -> getComposite*,
suiteGroups -> compositeGroups,
categorySuiteSections -> categoryCompositeSections,
suiteBenchmarkScoreRanges -> compositeBenchmarkScoreRanges,
activeSubtaskTab -> activeSliceTab,
subtaskMetricCounts -> sliceMetricCounts,
singleMetricSubtaskTabs -> singleMetricSliceTabs,
hasSubtaskTabs -> hasSliceTabs,
showSubtaskTopline -> showSliceTopline,
subtaskLabel/Count/DrivenBenchmarkCount -> slice* equivalents,
BenchmarkVariant discriminant strings "subtask" / "setup+subtask"
-> "slice" / "setup+slice",
subtasksOpen -> slicesOpen.
- Reworded prose in app/page.tsx, app/about/page.tsx,
app/evals/page.tsx, app/evals/[id]/page.tsx, app/layout.tsx,
app/opengraph-image.tsx, components/eval-card.tsx,
components/eval-detail.tsx, components/benchmark-evaluation-card.tsx,
components/policy-overview.tsx, components/model-compare-dialog.tsx.
Backend wire fields kept untouched (would require coordinated backend
change): metric.scope === "subtask", metric.subtask_key,
metric.subtask_name, summary.subtasks, slice.subtask_*,
composite_benchmark_name, composite_benchmark_key. A spec for the v2
maintainer covering the remaining family-grouping work
(gaia / gpqa / caparena variants) was sent separately.
- app/about/page.tsx +4 -4
- app/evals/[id]/page.tsx +4 -4
- app/evals/page.tsx +29 -1
- app/layout.tsx +1 -1
- app/opengraph-image.tsx +1 -1
- app/page.tsx +6 -11
- components/benchmark-detail.tsx +142 -142
- components/benchmark-evaluation-card.tsx +2 -2
- components/eval-card.tsx +1 -1
- components/eval-detail.tsx +62 -62
- components/family-table.tsx +50 -11
- components/model-compare-dialog.tsx +1 -1
- components/policy-overview.tsx +22 -19
|
@@ -65,8 +65,8 @@ export default function AboutPage() {
|
|
| 65 |
"Risk-domain annotations on benchmarks (capability, robustness, safety, agentic risk, fairness) so policy readers can locate which deployment-relevant property a number speaks to.",
|
| 66 |
],
|
| 67 |
[
|
| 68 |
-
"
|
| 69 |
-
"Family →
|
| 70 |
],
|
| 71 |
].map(([h, p]) => (
|
| 72 |
<div
|
|
@@ -146,7 +146,7 @@ export default function AboutPage() {
|
|
| 146 |
],
|
| 147 |
[
|
| 148 |
"Comparability",
|
| 149 |
-
"Can two scores under the same benchmark be put side-by-side? Tracks
|
| 150 |
],
|
| 151 |
].map(([h, p], i) => (
|
| 152 |
<li
|
|
@@ -183,7 +183,7 @@ export default function AboutPage() {
|
|
| 183 |
{[
|
| 184 |
[
|
| 185 |
"Canonicalisation",
|
| 186 |
-
"Heterogeneous score reports — papers, model cards, leaderboards, blog posts — are normalised to (model, benchmark,
|
| 187 |
],
|
| 188 |
[
|
| 189 |
"Source attribution",
|
|
|
|
| 65 |
"Risk-domain annotations on benchmarks (capability, robustness, safety, agentic risk, fairness) so policy readers can locate which deployment-relevant property a number speaks to.",
|
| 66 |
],
|
| 67 |
[
|
| 68 |
+
"Five-level hierarchy",
|
| 69 |
+
"Family → Composite → Single benchmark → Slice → Metric. Every score resolves to an explicit path, so aggregate claims drill down to the evidence supporting them.",
|
| 70 |
],
|
| 71 |
].map(([h, p]) => (
|
| 72 |
<div
|
|
|
|
| 146 |
],
|
| 147 |
[
|
| 148 |
"Comparability",
|
| 149 |
+
"Can two scores under the same benchmark be put side-by-side? Tracks slice, metric variant, and unit harmonisation; flags rows that cannot be ranked together.",
|
| 150 |
],
|
| 151 |
].map(([h, p], i) => (
|
| 152 |
<li
|
|
|
|
| 183 |
{[
|
| 184 |
[
|
| 185 |
"Canonicalisation",
|
| 186 |
+
"Heterogeneous score reports — papers, model cards, leaderboards, blog posts — are normalised to (model, benchmark, slice, metric, value, source) tuples. Model name aliases and benchmark version aliases are resolved against a curated mapping.",
|
| 187 |
],
|
| 188 |
[
|
| 189 |
"Source attribution",
|
|
@@ -135,7 +135,7 @@ export default function EvalDetailPage() {
|
|
| 135 |
}
|
| 136 |
|
| 137 |
// ---------------------------------------------------------------------------
|
| 138 |
-
// Composite
|
| 139 |
// Surfaces sub-benchmarks as a hairline grid and a per-model × per-metric
|
| 140 |
// matrix table. Both modes (research / policy) share the same chrome; the
|
| 141 |
// policy-note panel changes per benchmark, surfaced from the sub-summary card.
|
|
@@ -169,8 +169,8 @@ function CompositeEvalView({
|
|
| 169 |
const audience = card?.purpose_and_intended_users?.audience
|
| 170 |
const audienceText = Array.isArray(audience) ? audience.join("; ") : audience
|
| 171 |
const lede = isPolicy
|
| 172 |
-
? overview || goal || `
|
| 173 |
-
: goal || overview || `
|
| 174 |
|
| 175 |
return (
|
| 176 |
<div className="space-y-10">
|
|
@@ -281,7 +281,7 @@ function CompositeEvalView({
|
|
| 281 |
style={{ color: "var(--fg-muted)", maxWidth: 720 }}
|
| 282 |
>
|
| 283 |
{tab === "metrics"
|
| 284 |
-
? "Each card is one component benchmark inside this
|
| 285 |
: "Per-model scores across every component metric. Each column is a separately reported measure — distinct measures stay separate instead of collapsing into one number."}
|
| 286 |
</p>
|
| 287 |
|
|
|
|
| 135 |
}
|
| 136 |
|
| 137 |
// ---------------------------------------------------------------------------
|
| 138 |
+
// Composite view — paper §3.2 "composite reporting unit"
|
| 139 |
// Surfaces sub-benchmarks as a hairline grid and a per-model × per-metric
|
| 140 |
// matrix table. Both modes (research / policy) share the same chrome; the
|
| 141 |
// policy-note panel changes per benchmark, surfaced from the sub-summary card.
|
|
|
|
| 169 |
const audience = card?.purpose_and_intended_users?.audience
|
| 170 |
const audienceText = Array.isArray(audience) ? audience.join("; ") : audience
|
| 171 |
const lede = isPolicy
|
| 172 |
+
? overview || goal || `Composite aggregating ${subBenchmarkCount} component benchmarks across ${summary.models_count.toLocaleString()} models.`
|
| 173 |
+
: goal || overview || `Composite aggregating ${subBenchmarkCount} component benchmarks across ${summary.models_count.toLocaleString()} models.`
|
| 174 |
|
| 175 |
return (
|
| 176 |
<div className="space-y-10">
|
|
|
|
| 281 |
style={{ color: "var(--fg-muted)", maxWidth: 720 }}
|
| 282 |
>
|
| 283 |
{tab === "metrics"
|
| 284 |
+
? "Each card is one component benchmark inside this composite. Click a card to inspect its leaderboard, slices and benchmark card."
|
| 285 |
: "Per-model scores across every component metric. Each column is a separately reported measure — distinct measures stay separate instead of collapsing into one number."}
|
| 286 |
</p>
|
| 287 |
|
|
@@ -98,11 +98,31 @@ export default function EvalsPage() {
|
|
| 98 |
}
|
| 99 |
for (const fam of families) {
|
| 100 |
const seen = new Set<string>()
|
|
|
|
| 101 |
for (const d of lookupDomains(fam.key)) seen.add(d.trim().toLowerCase())
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 102 |
for (const leaf of fam.leaves ?? []) {
|
| 103 |
for (const d of leaf.tags?.domains ?? []) seen.add(d.trim().toLowerCase())
|
| 104 |
for (const d of lookupDomains(leaf.key)) seen.add(d.trim().toLowerCase())
|
| 105 |
}
|
|
|
|
|
|
|
| 106 |
for (const id of fam.eval_summary_ids ?? []) {
|
| 107 |
for (const d of lookupDomains(id)) seen.add(d.trim().toLowerCase())
|
| 108 |
}
|
|
@@ -126,6 +146,14 @@ export default function EvalsPage() {
|
|
| 126 |
for (const d of card?.benchmark_details?.domains ?? []) recordLabel(d)
|
| 127 |
}
|
| 128 |
for (const fam of families) {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 129 |
for (const leaf of fam.leaves ?? []) {
|
| 130 |
for (const d of leaf.tags?.domains ?? []) recordLabel(d)
|
| 131 |
}
|
|
@@ -244,7 +272,7 @@ export default function EvalsPage() {
|
|
| 244 |
</span>
|
| 245 |
</div>
|
| 246 |
<div className="ec-page-meta-item">
|
| 247 |
-
<span className="ec-page-meta-item-l">
|
| 248 |
<span className="ec-page-meta-item-v">
|
| 249 |
{stats.composite_count.toLocaleString()}
|
| 250 |
</span>
|
|
|
|
| 98 |
}
|
| 99 |
for (const fam of families) {
|
| 100 |
const seen = new Set<string>()
|
| 101 |
+
// Family-level fallback first (cards keyed by family slug).
|
| 102 |
for (const d of lookupDomains(fam.key)) seen.add(d.trim().toLowerCase())
|
| 103 |
+
|
| 104 |
+
// v2 primary path: walk every nested benchmark across composites,
|
| 105 |
+
// standalone benchmarks, and any family-level benchmarks array.
|
| 106 |
+
const nestedBenchmarks = [
|
| 107 |
+
...(fam.standalone_benchmarks ?? []),
|
| 108 |
+
...(fam.benchmarks ?? []),
|
| 109 |
+
...(fam.composites ?? []).flatMap((c) => c.benchmarks ?? []),
|
| 110 |
+
]
|
| 111 |
+
for (const benchmark of nestedBenchmarks) {
|
| 112 |
+
for (const d of benchmark.tags?.domains ?? []) seen.add(d.trim().toLowerCase())
|
| 113 |
+
for (const d of lookupDomains(benchmark.key)) seen.add(d.trim().toLowerCase())
|
| 114 |
+
for (const id of benchmark.summary_eval_ids ?? []) {
|
| 115 |
+
for (const d of lookupDomains(id)) seen.add(d.trim().toLowerCase())
|
| 116 |
+
}
|
| 117 |
+
}
|
| 118 |
+
|
| 119 |
+
// Legacy fallback: per-leaf tags + cards keyed by leaf slug.
|
| 120 |
for (const leaf of fam.leaves ?? []) {
|
| 121 |
for (const d of leaf.tags?.domains ?? []) seen.add(d.trim().toLowerCase())
|
| 122 |
for (const d of lookupDomains(leaf.key)) seen.add(d.trim().toLowerCase())
|
| 123 |
}
|
| 124 |
+
|
| 125 |
+
// Family-level eval_summary_ids cover both shapes.
|
| 126 |
for (const id of fam.eval_summary_ids ?? []) {
|
| 127 |
for (const d of lookupDomains(id)) seen.add(d.trim().toLowerCase())
|
| 128 |
}
|
|
|
|
| 146 |
for (const d of card?.benchmark_details?.domains ?? []) recordLabel(d)
|
| 147 |
}
|
| 148 |
for (const fam of families) {
|
| 149 |
+
const nestedBenchmarks = [
|
| 150 |
+
...(fam.standalone_benchmarks ?? []),
|
| 151 |
+
...(fam.benchmarks ?? []),
|
| 152 |
+
...(fam.composites ?? []).flatMap((c) => c.benchmarks ?? []),
|
| 153 |
+
]
|
| 154 |
+
for (const benchmark of nestedBenchmarks) {
|
| 155 |
+
for (const d of benchmark.tags?.domains ?? []) recordLabel(d)
|
| 156 |
+
}
|
| 157 |
for (const leaf of fam.leaves ?? []) {
|
| 158 |
for (const d of leaf.tags?.domains ?? []) recordLabel(d)
|
| 159 |
}
|
|
|
|
| 272 |
</span>
|
| 273 |
</div>
|
| 274 |
<div className="ec-page-meta-item">
|
| 275 |
+
<span className="ec-page-meta-item-l">Composites</span>
|
| 276 |
<span className="ec-page-meta-item-v">
|
| 277 |
{stats.composite_count.toLocaleString()}
|
| 278 |
</span>
|
|
@@ -9,7 +9,7 @@ const SITE_URL = "https://evalcards.evalevalai.com"
|
|
| 9 |
const SITE_NAME = "Eval Cards"
|
| 10 |
const SITE_TITLE = "Eval Cards — a reporting layer for AI evaluations"
|
| 11 |
const SITE_DESCRIPTION =
|
| 12 |
-
"A public registry of reported model–benchmark results, organised under a
|
| 13 |
|
| 14 |
export const metadata: Metadata = {
|
| 15 |
metadataBase: new URL(SITE_URL),
|
|
|
|
| 9 |
const SITE_NAME = "Eval Cards"
|
| 10 |
const SITE_TITLE = "Eval Cards — a reporting layer for AI evaluations"
|
| 11 |
const SITE_DESCRIPTION =
|
| 12 |
+
"A public registry of reported model–benchmark results, organised under a five-level rollout hierarchy and four interpretive signals: reproducibility, completeness, provenance, and comparability."
|
| 13 |
|
| 14 |
export const metadata: Metadata = {
|
| 15 |
metadataBase: new URL(SITE_URL),
|
|
@@ -110,7 +110,7 @@ export default function OpenGraphImage() {
|
|
| 110 |
maxWidth: "920px",
|
| 111 |
}}
|
| 112 |
>
|
| 113 |
-
A registry of reported model–benchmark results, organised under a
|
| 114 |
rollout hierarchy and four interpretive signals.
|
| 115 |
</div>
|
| 116 |
</div>
|
|
|
|
| 110 |
maxWidth: "920px",
|
| 111 |
}}
|
| 112 |
>
|
| 113 |
+
A registry of reported model–benchmark results, organised under a five-level
|
| 114 |
rollout hierarchy and four interpretive signals.
|
| 115 |
</div>
|
| 116 |
</div>
|
|
@@ -106,7 +106,7 @@ export default async function HomePage() {
|
|
| 106 |
</h1>
|
| 107 |
<p className="home-hero-lede">
|
| 108 |
<strong>Eval Cards</strong> is a registry of reported model–benchmark results,
|
| 109 |
-
organised under a
|
| 110 |
computed over the joined record.
|
| 111 |
</p>
|
| 112 |
<div className="flex flex-wrap items-center gap-3">
|
|
@@ -164,9 +164,9 @@ export default async function HomePage() {
|
|
| 164 |
</div>
|
| 165 |
</section>
|
| 166 |
|
| 167 |
-
{/*
|
| 168 |
<section className="hierarchy-strip">
|
| 169 |
-
<div className="kicker">
|
| 170 |
<div className="hierarchy-row">
|
| 171 |
{[
|
| 172 |
{
|
|
@@ -175,7 +175,7 @@ export default async function HomePage() {
|
|
| 175 |
ex: "SWE-bench family, MMLU family",
|
| 176 |
},
|
| 177 |
{
|
| 178 |
-
name: "
|
| 179 |
count: formatNumber(compositeCount),
|
| 180 |
ex: "Open LLM Leaderboard v2, HELM Instruct",
|
| 181 |
},
|
|
@@ -185,14 +185,9 @@ export default async function HomePage() {
|
|
| 185 |
ex: "GSM8K, IFEval, MMLU-Pro",
|
| 186 |
},
|
| 187 |
{
|
| 188 |
-
name: "
|
| 189 |
count: formatNumber(sliceCount),
|
| 190 |
-
ex: "algebra (within MATH),
|
| 191 |
-
},
|
| 192 |
-
{
|
| 193 |
-
name: "Subtask",
|
| 194 |
-
count: "—",
|
| 195 |
-
ex: "level-5, multi-turn",
|
| 196 |
},
|
| 197 |
{
|
| 198 |
name: "Metric",
|
|
|
|
| 106 |
</h1>
|
| 107 |
<p className="home-hero-lede">
|
| 108 |
<strong>Eval Cards</strong> is a registry of reported model–benchmark results,
|
| 109 |
+
organised under a five-level rollout hierarchy and four interpretive signals
|
| 110 |
computed over the joined record.
|
| 111 |
</p>
|
| 112 |
<div className="flex flex-wrap items-center gap-3">
|
|
|
|
| 164 |
</div>
|
| 165 |
</section>
|
| 166 |
|
| 167 |
+
{/* FIVE-LEVEL HIERARCHY STRIP -------------------------------------- */}
|
| 168 |
<section className="hierarchy-strip">
|
| 169 |
+
<div className="kicker">Five-level rollout hierarchy</div>
|
| 170 |
<div className="hierarchy-row">
|
| 171 |
{[
|
| 172 |
{
|
|
|
|
| 175 |
ex: "SWE-bench family, MMLU family",
|
| 176 |
},
|
| 177 |
{
|
| 178 |
+
name: "Composite",
|
| 179 |
count: formatNumber(compositeCount),
|
| 180 |
ex: "Open LLM Leaderboard v2, HELM Instruct",
|
| 181 |
},
|
|
|
|
| 185 |
ex: "GSM8K, IFEval, MMLU-Pro",
|
| 186 |
},
|
| 187 |
{
|
| 188 |
+
name: "Slice",
|
| 189 |
count: formatNumber(sliceCount),
|
| 190 |
+
ex: "algebra (within MATH), level-5, multi-turn",
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 191 |
},
|
| 192 |
{
|
| 193 |
name: "Metric",
|
|
@@ -62,10 +62,10 @@ interface BenchmarkVariant {
|
|
| 62 |
evaluation: BenchmarkEvaluation
|
| 63 |
result: EvaluationResult
|
| 64 |
label: string
|
| 65 |
-
variantType: "setup" | "
|
| 66 |
metricLabel: string
|
| 67 |
setupLabel: string | null
|
| 68 |
-
|
| 69 |
displayScore: string
|
| 70 |
normalizedScore: number
|
| 71 |
rankPosition: number | null
|
|
@@ -92,9 +92,9 @@ interface BenchmarkGroup {
|
|
| 92 |
variants: BenchmarkVariant[]
|
| 93 |
}
|
| 94 |
|
| 95 |
-
interface
|
| 96 |
-
|
| 97 |
-
|
| 98 |
benchmarks: BenchmarkGroup[]
|
| 99 |
avgRawScore: number
|
| 100 |
avgNormalizedScore: number
|
|
@@ -269,77 +269,77 @@ function getSourceTypeDisplayName(value: string | null | undefined) {
|
|
| 269 |
return normalizeDisplayLabel(value?.replace(/_/g, " ")) || "Unknown"
|
| 270 |
}
|
| 271 |
|
| 272 |
-
function
|
| 273 |
const k = key.toLowerCase().replace(/[-.\s]+/g, "_").replace(/^_+|_+$/g, "")
|
| 274 |
if (/^fibble\d*_arena$/.test(k)) return "fibble_arena"
|
| 275 |
if (/^arc_agi_v\d+/.test(k)) return "arc_agi"
|
| 276 |
return k
|
| 277 |
}
|
| 278 |
|
| 279 |
-
function doesLabelMatchSuiteKey(label: string | null | undefined,
|
| 280 |
if (!label) {
|
| 281 |
return false
|
| 282 |
}
|
| 283 |
|
| 284 |
-
return
|
| 285 |
}
|
| 286 |
|
| 287 |
-
function
|
| 288 |
const evaluation = group.variants[0]?.evaluation
|
| 289 |
const backendSuiteKey =
|
| 290 |
evaluation?.benchmark_parent_key ||
|
| 291 |
evaluation?.benchmark_family_key ||
|
| 292 |
evaluation?.benchmark
|
| 293 |
|
| 294 |
-
return
|
| 295 |
}
|
| 296 |
|
| 297 |
-
function
|
| 298 |
-
const normalizedKey =
|
| 299 |
return SUITE_DISPLAY_NAMES[normalizedKey] ?? normalizeDisplayLabel(key)
|
| 300 |
}
|
| 301 |
|
| 302 |
-
function
|
| 303 |
const evaluation = group.variants[0]?.evaluation
|
| 304 |
const benchmarkCardName = group.benchmarkCard?.benchmark_details?.name
|
| 305 |
const backendParentName = evaluation?.benchmark_parent_name
|
| 306 |
const backendFamilyName = evaluation?.benchmark_family_name
|
| 307 |
|
| 308 |
-
if (doesLabelMatchSuiteKey(backendParentName,
|
| 309 |
return normalizeDisplayLabel(backendParentName)
|
| 310 |
}
|
| 311 |
|
| 312 |
-
if (doesLabelMatchSuiteKey(backendFamilyName,
|
| 313 |
return normalizeDisplayLabel(backendFamilyName)
|
| 314 |
}
|
| 315 |
|
| 316 |
-
if (doesLabelMatchSuiteKey(benchmarkCardName,
|
| 317 |
return normalizeDisplayLabel(benchmarkCardName)
|
| 318 |
}
|
| 319 |
|
| 320 |
-
return
|
| 321 |
}
|
| 322 |
|
| 323 |
-
function
|
| 324 |
groups: BenchmarkGroup[],
|
| 325 |
modelIds: string[],
|
| 326 |
peerRanks: PeerRanksMap
|
| 327 |
-
):
|
| 328 |
-
const
|
| 329 |
for (const group of groups) {
|
| 330 |
-
const key =
|
| 331 |
-
const existing =
|
| 332 |
existing.push(group)
|
| 333 |
-
|
| 334 |
}
|
| 335 |
|
| 336 |
-
return Array.from(
|
| 337 |
const scores = benchmarks.map(b => b.avgNormalizedScore).filter(Number.isFinite)
|
| 338 |
const avgScore = scores.length > 0 ? scores.reduce((a, b) => a + b, 0) / scores.length : 0
|
| 339 |
const rawScores = benchmarks.map((benchmark) => benchmark.avgRawScore).filter(Number.isFinite)
|
| 340 |
const avgRawScore = rawScores.length > 0 ? rawScores.reduce((a, b) => a + b, 0) / rawScores.length : 0
|
| 341 |
|
| 342 |
-
// Find best rank across all benchmarks in
|
| 343 |
let bestRank: { position: number; total: number } | null = null
|
| 344 |
for (const b of benchmarks) {
|
| 345 |
const rank = getGroupPeerRank(b, modelIds, peerRanks)
|
|
@@ -350,8 +350,8 @@ function groupBySuite(
|
|
| 350 |
}
|
| 351 |
|
| 352 |
return {
|
| 353 |
-
|
| 354 |
-
|
| 355 |
benchmarks,
|
| 356 |
avgRawScore,
|
| 357 |
avgNormalizedScore: avgScore,
|
|
@@ -359,7 +359,7 @@ function groupBySuite(
|
|
| 359 |
bestRank,
|
| 360 |
}
|
| 361 |
}).sort((a, b) => {
|
| 362 |
-
// Sort by best peer rank ratio (lower = better); unranked
|
| 363 |
const aRatio = a.bestRank ? a.bestRank.position / (a.bestRank.total || a.bestRank.position) : Infinity
|
| 364 |
const bRatio = b.bestRank ? b.bestRank.position / (b.bestRank.total || b.bestRank.position) : Infinity
|
| 365 |
if (aRatio !== bRatio) return aRatio - bRatio
|
|
@@ -466,25 +466,25 @@ function getMetricDisplayLabel(result: EvaluationResult) {
|
|
| 466 |
function getVariantDescriptor(
|
| 467 |
evaluation: BenchmarkEvaluation,
|
| 468 |
result: EvaluationResult
|
| 469 |
-
): Pick<BenchmarkVariant, "label" | "variantType" | "metricLabel" | "setupLabel" | "
|
| 470 |
const evaluationVariantRaw = getEvaluationVariantLabel(evaluation)
|
| 471 |
const evaluationVariant = evaluationVariantRaw ? formatSetupDisplayLabel(evaluationVariantRaw) : null
|
| 472 |
const metricLabel = getMetricDisplayLabel(result)
|
| 473 |
const metricKey = normalizeDisplayKey(metricLabel)
|
| 474 |
const metricIsAmbiguous = AMBIGUOUS_GROUP_LABELS.has(metricKey)
|
| 475 |
-
const
|
| 476 |
const setupLabel = evaluationVariant ? formatSetupDisplayLabel(evaluationVariant) : null
|
| 477 |
-
const baseLabel =
|
| 478 |
-
? (metricIsAmbiguous ?
|
| 479 |
: metricLabel
|
| 480 |
|
| 481 |
-
if (setupLabel &&
|
| 482 |
return {
|
| 483 |
label: `${setupLabel} · ${baseLabel}`,
|
| 484 |
-
variantType: "setup+
|
| 485 |
metricLabel,
|
| 486 |
setupLabel,
|
| 487 |
-
|
| 488 |
}
|
| 489 |
}
|
| 490 |
|
|
@@ -494,17 +494,17 @@ function getVariantDescriptor(
|
|
| 494 |
variantType: "setup",
|
| 495 |
metricLabel,
|
| 496 |
setupLabel,
|
| 497 |
-
|
| 498 |
}
|
| 499 |
}
|
| 500 |
|
| 501 |
-
if (
|
| 502 |
return {
|
| 503 |
label: baseLabel,
|
| 504 |
-
variantType:
|
| 505 |
metricLabel,
|
| 506 |
setupLabel: null,
|
| 507 |
-
|
| 508 |
}
|
| 509 |
}
|
| 510 |
|
|
@@ -513,7 +513,7 @@ function getVariantDescriptor(
|
|
| 513 |
variantType: "default",
|
| 514 |
metricLabel,
|
| 515 |
setupLabel: null,
|
| 516 |
-
|
| 517 |
}
|
| 518 |
}
|
| 519 |
|
|
@@ -603,7 +603,7 @@ function getTableConfigLabel(row: VariantRowData) {
|
|
| 603 |
return row.variant.setupLabel
|
| 604 |
}
|
| 605 |
|
| 606 |
-
if (row.variant.variantType === "
|
| 607 |
return "Default setup"
|
| 608 |
}
|
| 609 |
|
|
@@ -865,9 +865,9 @@ function getVariantTypeTone(variantType: BenchmarkVariant["variantType"]) {
|
|
| 865 |
switch (variantType) {
|
| 866 |
case "setup":
|
| 867 |
return "bg-sky-100 text-sky-800 dark:bg-sky-950/50 dark:text-sky-300"
|
| 868 |
-
case "
|
| 869 |
return "bg-violet-100 text-violet-800 dark:bg-violet-950/50 dark:text-violet-300"
|
| 870 |
-
case "setup+
|
| 871 |
return "bg-amber-100 text-amber-800 dark:bg-amber-950/50 dark:text-amber-300"
|
| 872 |
default:
|
| 873 |
return "bg-muted text-muted-foreground"
|
|
@@ -878,10 +878,10 @@ function getVariantTypeLabel(variantType: BenchmarkVariant["variantType"]) {
|
|
| 878 |
switch (variantType) {
|
| 879 |
case "setup":
|
| 880 |
return "Setup change"
|
| 881 |
-
case "
|
| 882 |
-
return "Benchmark
|
| 883 |
-
case "setup+
|
| 884 |
-
return "Setup +
|
| 885 |
default:
|
| 886 |
return "Single run"
|
| 887 |
}
|
|
@@ -911,8 +911,8 @@ function formatSetupDisplayLabel(setupLabel: string | null) {
|
|
| 911 |
}
|
| 912 |
|
| 913 |
function getVariantPrimaryLabel(variant: BenchmarkVariant, groupTitle: string) {
|
| 914 |
-
if (variant.
|
| 915 |
-
return variant.
|
| 916 |
}
|
| 917 |
|
| 918 |
if (variant.metricLabel) {
|
|
@@ -929,41 +929,41 @@ function getVariantPrimaryLabel(variant: BenchmarkVariant, groupTitle: string) {
|
|
| 929 |
return variant.label
|
| 930 |
}
|
| 931 |
|
| 932 |
-
function
|
| 933 |
return Array.from(
|
| 934 |
new Set(
|
| 935 |
group.variants
|
| 936 |
-
.map((variant) => variant.
|
| 937 |
.filter((label): label is string => Boolean(label))
|
| 938 |
)
|
| 939 |
)
|
| 940 |
}
|
| 941 |
|
| 942 |
-
function
|
| 943 |
-
return
|
| 944 |
}
|
| 945 |
|
| 946 |
function getBenchmarkGroupHeading(group: BenchmarkGroup) {
|
| 947 |
return group.canonicalTitle
|
| 948 |
}
|
| 949 |
|
| 950 |
-
function
|
| 951 |
-
if (
|
| 952 |
return {
|
| 953 |
-
count:
|
| 954 |
-
label: `sub-benchmark${
|
| 955 |
className:
|
| 956 |
"border-sky-200/80 bg-sky-50/70 text-sky-700 dark:border-sky-900/60 dark:bg-sky-950/30 dark:text-sky-300",
|
| 957 |
}
|
| 958 |
}
|
| 959 |
|
| 960 |
-
const singleGroup =
|
| 961 |
if (!singleGroup) {
|
| 962 |
return null
|
| 963 |
}
|
| 964 |
|
| 965 |
-
const
|
| 966 |
-
if (!
|
| 967 |
return {
|
| 968 |
count: 1,
|
| 969 |
label: "sub-benchmark",
|
|
@@ -972,11 +972,11 @@ function getSuiteBadgeMeta(suite: SuiteGroup) {
|
|
| 972 |
}
|
| 973 |
}
|
| 974 |
|
| 975 |
-
const
|
| 976 |
-
if (
|
| 977 |
return {
|
| 978 |
-
count:
|
| 979 |
-
label: `
|
| 980 |
className:
|
| 981 |
"border-emerald-200/70 bg-emerald-50 text-emerald-700 dark:border-emerald-900/60 dark:bg-emerald-950/30 dark:text-emerald-300",
|
| 982 |
}
|
|
@@ -1405,7 +1405,7 @@ function getVariantDedupKey(variant: BenchmarkVariant) {
|
|
| 1405 |
metricLabel: variant.metricLabel,
|
| 1406 |
variantType: variant.variantType,
|
| 1407 |
setupLabel: variant.setupLabel,
|
| 1408 |
-
|
| 1409 |
displayScore: variant.displayScore,
|
| 1410 |
sourceOrganization: getOrganizationDisplayName(variant.evaluation.source_metadata.source_organization_name),
|
| 1411 |
sourceName: normalizeDisplayLabel(variant.evaluation.source_metadata.source_name ?? ""),
|
|
@@ -1459,7 +1459,7 @@ function buildBenchmarkGroups(
|
|
| 1459 |
variantType: descriptor.variantType,
|
| 1460 |
metricLabel: descriptor.metricLabel,
|
| 1461 |
setupLabel: descriptor.setupLabel,
|
| 1462 |
-
|
| 1463 |
displayScore,
|
| 1464 |
normalizedScore,
|
| 1465 |
rankPosition,
|
|
@@ -1570,10 +1570,10 @@ function buildBenchmarkGroups(
|
|
| 1570 |
|
| 1571 |
group.variants = Array.from(dedupedVariants.values())
|
| 1572 |
group.variants.sort((a, b) => {
|
| 1573 |
-
const
|
| 1574 |
-
const
|
| 1575 |
-
if (
|
| 1576 |
-
return
|
| 1577 |
}
|
| 1578 |
|
| 1579 |
const aPrimaryLabel = getVariantPrimaryLabel(a, group.title)
|
|
@@ -2043,9 +2043,9 @@ export function BenchmarkDetail({
|
|
| 2043 |
}))
|
| 2044 |
}, [filteredBenchmarkGroups, comparisonIndex, summary.categories_covered])
|
| 2045 |
|
| 2046 |
-
const
|
| 2047 |
-
const groups =
|
| 2048 |
-
// Re-sort
|
| 2049 |
return groups.sort((a, b) => {
|
| 2050 |
const aMax = Math.max(...a.benchmarks.map(getRelevanceScore))
|
| 2051 |
const bMax = Math.max(...b.benchmarks.map(getRelevanceScore))
|
|
@@ -2053,48 +2053,48 @@ export function BenchmarkDetail({
|
|
| 2053 |
})
|
| 2054 |
}, [filteredBenchmarkGroups, modelIds, peerRanks, getRelevanceScore])
|
| 2055 |
|
| 2056 |
-
const
|
| 2057 |
() =>
|
| 2058 |
groupedFilteredBenchmarkGroups
|
| 2059 |
.map(({ category, groups }) => ({
|
| 2060 |
category,
|
| 2061 |
-
|
| 2062 |
const aMax = Math.max(...a.benchmarks.map(getRelevanceScore))
|
| 2063 |
const bMax = Math.max(...b.benchmarks.map(getRelevanceScore))
|
| 2064 |
return bMax - aMax
|
| 2065 |
}),
|
| 2066 |
}))
|
| 2067 |
-
.filter((section) => section.
|
| 2068 |
[groupedFilteredBenchmarkGroups, modelIds, peerRanks, getRelevanceScore]
|
| 2069 |
)
|
| 2070 |
|
| 2071 |
const categoryScoreRanges = useMemo(() => {
|
| 2072 |
const ranges = new Map<CategoryType, ScoreRange>()
|
| 2073 |
|
| 2074 |
-
for (const section of
|
| 2075 |
ranges.set(
|
| 2076 |
section.category,
|
| 2077 |
-
getScoreRange(section.
|
| 2078 |
)
|
| 2079 |
}
|
| 2080 |
|
| 2081 |
return ranges
|
| 2082 |
-
}, [
|
| 2083 |
|
| 2084 |
-
const
|
| 2085 |
const ranges = new Map<string, ScoreRange>()
|
| 2086 |
|
| 2087 |
-
for (const section of
|
| 2088 |
-
for (const
|
| 2089 |
ranges.set(
|
| 2090 |
-
|
| 2091 |
-
getScoreRange(
|
| 2092 |
)
|
| 2093 |
}
|
| 2094 |
}
|
| 2095 |
|
| 2096 |
return ranges
|
| 2097 |
-
}, [
|
| 2098 |
|
| 2099 |
const benchmarkGroupLookup = useMemo(
|
| 2100 |
() => new Map(benchmarkGroups.map((group) => [group.key, group] as const)),
|
|
@@ -2104,11 +2104,11 @@ export function BenchmarkDetail({
|
|
| 2104 |
? benchmarkGroupLookup.get(activeBenchmarkGroupKey) ?? null
|
| 2105 |
: null
|
| 2106 |
|
| 2107 |
-
const toggleSuite = (
|
| 2108 |
setExpandedSuites((prev) => {
|
| 2109 |
const next = new Set(prev)
|
| 2110 |
-
if (next.has(
|
| 2111 |
-
else next.add(
|
| 2112 |
return next
|
| 2113 |
})
|
| 2114 |
}
|
|
@@ -2150,10 +2150,10 @@ export function BenchmarkDetail({
|
|
| 2150 |
)
|
| 2151 |
const repeatedBenchmarkCount = overviewBenchmarkGroups.filter((group) => group.variants.length > 1).length
|
| 2152 |
const setupDrivenBenchmarkCount = overviewBenchmarkGroups.filter((group) =>
|
| 2153 |
-
group.variants.some((variant) => variant.variantType === "setup" || variant.variantType === "setup+
|
| 2154 |
).length
|
| 2155 |
-
const
|
| 2156 |
-
group.variants.some((variant) => variant.variantType === "
|
| 2157 |
).length
|
| 2158 |
|
| 2159 |
useEffect(() => {
|
|
@@ -2414,7 +2414,7 @@ export function BenchmarkDetail({
|
|
| 2414 |
summary.model_info.name,
|
| 2415 |
])
|
| 2416 |
|
| 2417 |
-
// A plotbox can expose a top-level "view" selector (
|
| 2418 |
// benchmarks, components) and an optional metric tab rail beneath the chart.
|
| 2419 |
// Plotbox grouping is driven entirely by comparison-index's own
|
| 2420 |
// benchmark_family_key so it stays in sync with the backend.
|
|
@@ -2448,7 +2448,7 @@ export function BenchmarkDetail({
|
|
| 2448 |
familyName: string
|
| 2449 |
category: CategoryType
|
| 2450 |
kind: "single-eval" | "multi-eval"
|
| 2451 |
-
childKindLabel: "metric" | "benchmark" | "component" | "
|
| 2452 |
views: PlotboxView[]
|
| 2453 |
primaryGroup: BenchmarkGroup
|
| 2454 |
}
|
|
@@ -2534,7 +2534,7 @@ export function BenchmarkDetail({
|
|
| 2534 |
const { familyName, category, resolved } = bucket
|
| 2535 |
|
| 2536 |
if (resolved.length === 1) {
|
| 2537 |
-
// One eval in scope —
|
| 2538 |
// metrics move to a compact tab rail beneath the chart.
|
| 2539 |
const { group, evalEntry } = resolved[0]
|
| 2540 |
const evalDisplay =
|
|
@@ -2545,10 +2545,10 @@ export function BenchmarkDetail({
|
|
| 2545 |
>()
|
| 2546 |
|
| 2547 |
for (const variant of group.variants) {
|
| 2548 |
-
const viewKey = variant.
|
| 2549 |
-
? `
|
| 2550 |
: "default"
|
| 2551 |
-
const label = variant.
|
| 2552 |
const bucketForView = singleEvalViewBuckets.get(viewKey) ?? {
|
| 2553 |
viewKey,
|
| 2554 |
label,
|
|
@@ -2600,7 +2600,7 @@ export function BenchmarkDetail({
|
|
| 2600 |
familyName: evalDisplay,
|
| 2601 |
category,
|
| 2602 |
kind: "single-eval",
|
| 2603 |
-
childKindLabel: views.length > 1 ? "
|
| 2604 |
views,
|
| 2605 |
primaryGroup: group,
|
| 2606 |
})
|
|
@@ -2656,7 +2656,7 @@ export function BenchmarkDetail({
|
|
| 2656 |
if (views.length === 0) continue
|
| 2657 |
|
| 2658 |
let hasComponent = false
|
| 2659 |
-
let
|
| 2660 |
let hasDistinctLeaves = false
|
| 2661 |
for (const r of children) {
|
| 2662 |
const leafKey = r.evalEntry.benchmark_leaf_key
|
|
@@ -2666,17 +2666,17 @@ export function BenchmarkDetail({
|
|
| 2666 |
if (r.group.variants[0]?.evaluation.benchmark_component_key ?? null) {
|
| 2667 |
hasComponent = true
|
| 2668 |
} else {
|
| 2669 |
-
|
| 2670 |
}
|
| 2671 |
}
|
| 2672 |
const childKindLabel: PlotboxUnit["childKindLabel"] =
|
| 2673 |
-
hasComponent &&
|
| 2674 |
? "component"
|
| 2675 |
: hasComponent
|
| 2676 |
? "metric"
|
| 2677 |
: hasDistinctLeaves
|
| 2678 |
? "benchmark"
|
| 2679 |
-
: "
|
| 2680 |
|
| 2681 |
units.push({
|
| 2682 |
unitKey: `family:${famKey}`,
|
|
@@ -2909,8 +2909,8 @@ export function BenchmarkDetail({
|
|
| 2909 |
const childKindPlural =
|
| 2910 |
unit.childKindLabel === "metric"
|
| 2911 |
? childKindCount === 1 ? "metric" : "metrics"
|
| 2912 |
-
: unit.childKindLabel === "
|
| 2913 |
-
? childKindCount === 1 ? "
|
| 2914 |
: unit.childKindLabel === "benchmark"
|
| 2915 |
? childKindCount === 1 ? "benchmark" : "benchmarks"
|
| 2916 |
: childKindCount === 1 ? "component" : "components"
|
|
@@ -3395,10 +3395,10 @@ export function BenchmarkDetail({
|
|
| 3395 |
? `${reportingStats.missingGenerationConfigs} entries are missing generation config, limiting cross-slice comparability.`
|
| 3396 |
: "Generation configuration is present across the result set."}
|
| 3397 |
</p>
|
| 3398 |
-
{(setupDrivenBenchmarkCount > 0 ||
|
| 3399 |
<p className="text-[13px] leading-[1.7] text-[color:var(--fg-muted)]">
|
| 3400 |
Decomposition: <span className="text-[color:var(--fg)]">{setupDrivenBenchmarkCount}</span> setup-aware ·{" "}
|
| 3401 |
-
<span className="text-[color:var(--fg)]">{
|
| 3402 |
{reportingStats.libraryList.length > 0 && (
|
| 3403 |
<>
|
| 3404 |
{" "}Eval libraries: <span className="text-[color:var(--fg)]">{reportingStats.libraryList.join(", ")}</span>.
|
|
@@ -3531,7 +3531,7 @@ export function BenchmarkDetail({
|
|
| 3531 |
|
| 3532 |
<p className="text-[14px] leading-[1.7] text-[color:var(--fg-muted)] max-w-[64rem] mb-6">
|
| 3533 |
{isResearchView
|
| 3534 |
-
? "Benchmark-first view of this model's reported results, grouped by category. Setup spread and
|
| 3535 |
: "The public evidence behind this model, grouped by category. The strongest and most variable signals are listed first."}
|
| 3536 |
{policyHighlights.length > 0 && !isResearchView && (
|
| 3537 |
<>
|
|
@@ -3606,9 +3606,9 @@ export function BenchmarkDetail({
|
|
| 3606 |
)}
|
| 3607 |
{repeatedBenchmarkCount > 0 && (
|
| 3608 |
<>
|
| 3609 |
-
<dt>
|
| 3610 |
<dd>
|
| 3611 |
-
{repeatedBenchmarkCount} benchmark{repeatedBenchmarkCount === 1 ? "" : "s"} include multiple
|
| 3612 |
</dd>
|
| 3613 |
</>
|
| 3614 |
)}
|
|
@@ -3701,7 +3701,7 @@ export function BenchmarkDetail({
|
|
| 3701 |
<option value="rank">Sort · Best rank</option>
|
| 3702 |
<option value="score">Sort · Highest score</option>
|
| 3703 |
<option value="name">Sort · Name (A–Z)</option>
|
| 3704 |
-
<option value="variants">Sort · Most
|
| 3705 |
<option value="spread">Sort · Largest spread</option>
|
| 3706 |
</select>
|
| 3707 |
</div>
|
|
@@ -3856,7 +3856,7 @@ export function BenchmarkDetail({
|
|
| 3856 |
<span>{row.variant.result.generation_config.num_few_shot}-shot</span>
|
| 3857 |
)}
|
| 3858 |
{row.variant.setupLabel && <span>· {row.variant.setupLabel}</span>}
|
| 3859 |
-
{row.variant.
|
| 3860 |
</div>
|
| 3861 |
)}
|
| 3862 |
</div>
|
|
@@ -4527,22 +4527,22 @@ function BenchmarkResultCard({
|
|
| 4527 |
<Separator className="my-4" />
|
| 4528 |
<div className="mb-2">
|
| 4529 |
<div className="text-xs font-semibold text-muted-foreground uppercase tracking-wider">Detailed Breakdown</div>
|
| 4530 |
-
<div className="text-xs text-muted-foreground mt-1">Scores and structured metadata for individual
|
| 4531 |
</div>
|
| 4532 |
|
| 4533 |
{numericBreakdown.length > 0 && (
|
| 4534 |
<div className="grid grid-cols-1 sm:grid-cols-2 xl:grid-cols-3 gap-3">
|
| 4535 |
{numericBreakdown.map(([key, value]) => {
|
| 4536 |
let valDisplay = typeof value === 'number' ? value.toFixed(2) : value;
|
| 4537 |
-
let
|
| 4538 |
|
| 4539 |
if (typeof value === 'number') {
|
| 4540 |
if (unit === 'accuracy' || !unit || unit === 'pass@1') {
|
| 4541 |
valDisplay = formatRawScoreValue(value);
|
| 4542 |
-
|
| 4543 |
} else {
|
| 4544 |
valDisplay = value.toFixed(2);
|
| 4545 |
-
|
| 4546 |
}
|
| 4547 |
}
|
| 4548 |
|
|
@@ -4556,7 +4556,7 @@ function BenchmarkResultCard({
|
|
| 4556 |
{valDisplay}
|
| 4557 |
</div>
|
| 4558 |
{typeof value === 'number' && (
|
| 4559 |
-
<Progress value={
|
| 4560 |
)}
|
| 4561 |
</div>
|
| 4562 |
)})}
|
|
@@ -4800,7 +4800,7 @@ function AggregatedBenchmarkCard({
|
|
| 4800 |
Number.isFinite(latestTimestamp) ? formatCompactDate(String(latestTimestamp)) : formatCompactDate(group.variants[0]?.evaluation.retrieved_timestamp ?? "")
|
| 4801 |
const compactDomains = group.domains.slice(0, 2)
|
| 4802 |
const progressWidth = Math.max(4, Math.min(100, group.avgNormalizedScore * 100))
|
| 4803 |
-
const
|
| 4804 |
|
| 4805 |
const toggleRow = (rowKey: string) => {
|
| 4806 |
setExpandedRows((current) => ({
|
|
@@ -4851,9 +4851,9 @@ function AggregatedBenchmarkCard({
|
|
| 4851 |
card
|
| 4852 |
</span>
|
| 4853 |
)}
|
| 4854 |
-
{
|
| 4855 |
<span className="shrink-0 rounded-full border border-emerald-200/70 bg-emerald-50 px-1.5 py-0.5 text-[9px] font-semibold uppercase tracking-[0.12em] text-emerald-700 dark:border-emerald-900/60 dark:bg-emerald-950/30 dark:text-emerald-300">
|
| 4856 |
-
{
|
| 4857 |
</span>
|
| 4858 |
)}
|
| 4859 |
{compactDomains.map((domain) => (
|
|
@@ -4881,7 +4881,7 @@ function AggregatedBenchmarkCard({
|
|
| 4881 |
)}
|
| 4882 |
</div>
|
| 4883 |
|
| 4884 |
-
{/*
|
| 4885 |
<span className="shrink-0 text-[11px] text-muted-foreground w-16 text-right hidden sm:block">
|
| 4886 |
{group.variants.length} {group.variants.length === 1 ? "row" : "rows"}
|
| 4887 |
</span>
|
|
@@ -5077,9 +5077,9 @@ function AggregatedBenchmarkCard({
|
|
| 5077 |
</div>
|
| 5078 |
<div className="text-xs text-muted-foreground">
|
| 5079 |
{variant.setupLabel && <span>Setup: {variant.setupLabel}</span>}
|
| 5080 |
-
{variant.setupLabel && variant.
|
| 5081 |
-
{variant.
|
| 5082 |
-
{!variant.setupLabel && !variant.
|
| 5083 |
</div>
|
| 5084 |
</div>
|
| 5085 |
</div>
|
|
@@ -5193,8 +5193,8 @@ function BenchmarkDeepDiveDialogPanel({
|
|
| 5193 |
const [resolvedRanks, setResolvedRanks] = useState<Record<string, { position: number; total: number | null }>>({})
|
| 5194 |
const [isResolvingRanks, setIsResolvingRanks] = useState(false)
|
| 5195 |
const compactDomains = group.domains.slice(0, 2)
|
| 5196 |
-
const
|
| 5197 |
-
const
|
| 5198 |
const sourceOrganizations = useMemo(
|
| 5199 |
() => new Set(group.variants.map((variant) => getOrganizationDisplayName(variant.evaluation.source_metadata.source_organization_name))),
|
| 5200 |
[group.variants]
|
|
@@ -5335,18 +5335,18 @@ function BenchmarkDeepDiveDialogPanel({
|
|
| 5335 |
// Kept only to drive the single-setup overview: when every reported row is
|
| 5336 |
// reported under the same setup, the detail table collapses into a compact
|
| 5337 |
// view that drops the redundant "Reporting setup" column.
|
| 5338 |
-
const
|
| 5339 |
-
if (!
|
| 5340 |
const setupOrder: string[] = []
|
| 5341 |
for (const row of variantRows) {
|
| 5342 |
const setupDisplayLabel = formatSetupDisplayLabel(row.variant.setupLabel)
|
| 5343 |
if (!setupOrder.includes(setupDisplayLabel)) setupOrder.push(setupDisplayLabel)
|
| 5344 |
}
|
| 5345 |
return { setupOrder }
|
| 5346 |
-
}, [
|
| 5347 |
|
| 5348 |
-
const useSingleSetupOverview = Boolean(
|
| 5349 |
-
const singleSetupDisplayLabel = useSingleSetupOverview ?
|
| 5350 |
|
| 5351 |
useEffect(() => {
|
| 5352 |
const pendingRows = variantRows.filter(
|
|
@@ -5411,7 +5411,7 @@ function BenchmarkDeepDiveDialogPanel({
|
|
| 5411 |
</DialogTitle>
|
| 5412 |
<DialogDescription className="mt-1.5 text-[13px] leading-[1.5] text-[color:var(--fg-muted)]">
|
| 5413 |
{isResearchView
|
| 5414 |
-
? "Inspect setup
|
| 5415 |
: "Inspect reporting setup and evidence details before interpreting benchmark position."}
|
| 5416 |
</DialogDescription>
|
| 5417 |
{(compactDomains.length > 0 || group.benchmarkCard) && (
|
|
@@ -5462,9 +5462,9 @@ function BenchmarkDeepDiveDialogPanel({
|
|
| 5462 |
<div className="kicker">Sources</div>
|
| 5463 |
<div className="mt-1 text-[18px] font-semibold tabular-nums">
|
| 5464 |
{sourceOrganizations.size}
|
| 5465 |
-
{
|
| 5466 |
<span className="ml-2 font-mono text-[10px] tracking-[0.12em] uppercase text-[color:var(--fg-subtle)]">
|
| 5467 |
-
· {
|
| 5468 |
</span>
|
| 5469 |
)}
|
| 5470 |
</div>
|
|
@@ -5482,7 +5482,7 @@ function BenchmarkDeepDiveDialogPanel({
|
|
| 5482 |
|
| 5483 |
{/* Sources — distinct reporting orgs and dataset links for this group.
|
| 5484 |
Pulled up to the top of the deep-dive so the per-row table can
|
| 5485 |
-
stay focused on
|
| 5486 |
{(() => {
|
| 5487 |
type SourceEntry = {
|
| 5488 |
key: string
|
|
@@ -5581,7 +5581,7 @@ function BenchmarkDeepDiveDialogPanel({
|
|
| 5581 |
{useSingleSetupOverview ? (
|
| 5582 |
<section>
|
| 5583 |
<div className="section-head">
|
| 5584 |
-
<h2>{hasAmbiguousPrimaryLabels ? "Reported runs" : "
|
| 5585 |
<span className="font-mono text-[10px] uppercase tracking-[0.15em] text-[color:var(--fg-subtle)]">
|
| 5586 |
{singleSetupDisplayLabel ? `${singleSetupDisplayLabel} · ` : ""}
|
| 5587 |
{variantRows.length} row{variantRows.length === 1 ? "" : "s"}
|
|
@@ -5594,8 +5594,8 @@ function BenchmarkDeepDiveDialogPanel({
|
|
| 5594 |
? "These rows share the same benchmark label, so run names or differing config fields are surfaced to show what changed across reports."
|
| 5595 |
: "These rows describe the same benchmark view, so the table surfaces the reported run name or setup differences that separate them."
|
| 5596 |
: isResearchView
|
| 5597 |
-
? "This benchmark reports one setup, so
|
| 5598 |
-
: "This benchmark only reports one setup, so the
|
| 5599 |
</p>
|
| 5600 |
|
| 5601 |
<div className="min-h-0 overflow-auto">
|
|
@@ -5603,7 +5603,7 @@ function BenchmarkDeepDiveDialogPanel({
|
|
| 5603 |
<thead>
|
| 5604 |
<tr>
|
| 5605 |
<th className="w-[60%]">
|
| 5606 |
-
{hasAmbiguousPrimaryLabels ? "Reported row" : "
|
| 5607 |
</th>
|
| 5608 |
<th className="w-[20%]">Setup detail</th>
|
| 5609 |
<th className="num w-[10%]">Score</th>
|
|
@@ -5678,14 +5678,14 @@ function BenchmarkDeepDiveDialogPanel({
|
|
| 5678 |
</span>
|
| 5679 |
</div>
|
| 5680 |
<p className="mb-4 max-w-[60rem] text-[13px] leading-[1.65] text-[color:var(--fg-muted)]">
|
| 5681 |
-
Primary row labels show the benchmark slice or
|
| 5682 |
</p>
|
| 5683 |
|
| 5684 |
<div className="min-h-0 overflow-auto">
|
| 5685 |
<table className="ec-htable table-fixed">
|
| 5686 |
<thead>
|
| 5687 |
<tr>
|
| 5688 |
-
<th className="w-[46%]">
|
| 5689 |
<th className="w-[36%]">Reporting setup</th>
|
| 5690 |
<th className="num w-[9%]">Score</th>
|
| 5691 |
<th className="num w-[9%]">Rank</th>
|
|
@@ -5895,7 +5895,7 @@ function VariantExpandedDetail({
|
|
| 5895 |
)}
|
| 5896 |
{sourceData?.dataset_version && <InlineMeta label="Dataset Version" value={sourceData.dataset_version} />}
|
| 5897 |
{sourceData?.hf_split && <InlineMeta label="Split" value={sourceData.hf_split} />}
|
| 5898 |
-
{variant.
|
| 5899 |
{variant.setupLabel && <InlineMeta label="Setup" value={formatSetupDisplayLabel(variant.setupLabel)} />}
|
| 5900 |
{inferencePlatform && <InlineMeta label="Inference Platform" value={inferencePlatform} />}
|
| 5901 |
{variant.evaluation.source_metadata.source_name && (
|
|
@@ -5952,7 +5952,7 @@ function VariantExpandedDetail({
|
|
| 5952 |
{numericBreakdown.length > 0 && (
|
| 5953 |
<div className="space-y-3">
|
| 5954 |
<div className="text-xs font-semibold uppercase tracking-[0.2em] text-muted-foreground">
|
| 5955 |
-
{isResearchView ? "
|
| 5956 |
</div>
|
| 5957 |
<div className="grid gap-3 md:grid-cols-2 2xl:grid-cols-3">
|
| 5958 |
{numericBreakdown.map(([key, value]) => {
|
|
|
|
| 62 |
evaluation: BenchmarkEvaluation
|
| 63 |
result: EvaluationResult
|
| 64 |
label: string
|
| 65 |
+
variantType: "setup" | "slice" | "setup+slice" | "default"
|
| 66 |
metricLabel: string
|
| 67 |
setupLabel: string | null
|
| 68 |
+
sliceLabel: string | null
|
| 69 |
displayScore: string
|
| 70 |
normalizedScore: number
|
| 71 |
rankPosition: number | null
|
|
|
|
| 92 |
variants: BenchmarkVariant[]
|
| 93 |
}
|
| 94 |
|
| 95 |
+
interface CompositeGroup {
|
| 96 |
+
compositeKey: string
|
| 97 |
+
compositeName: string
|
| 98 |
benchmarks: BenchmarkGroup[]
|
| 99 |
avgRawScore: number
|
| 100 |
avgNormalizedScore: number
|
|
|
|
| 269 |
return normalizeDisplayLabel(value?.replace(/_/g, " ")) || "Unknown"
|
| 270 |
}
|
| 271 |
|
| 272 |
+
function normalizeCompositeKey(key: string): string {
|
| 273 |
const k = key.toLowerCase().replace(/[-.\s]+/g, "_").replace(/^_+|_+$/g, "")
|
| 274 |
if (/^fibble\d*_arena$/.test(k)) return "fibble_arena"
|
| 275 |
if (/^arc_agi_v\d+/.test(k)) return "arc_agi"
|
| 276 |
return k
|
| 277 |
}
|
| 278 |
|
| 279 |
+
function doesLabelMatchSuiteKey(label: string | null | undefined, compositeKey: string) {
|
| 280 |
if (!label) {
|
| 281 |
return false
|
| 282 |
}
|
| 283 |
|
| 284 |
+
return normalizeCompositeKey(normalizeDisplayKey(label)) === normalizeCompositeKey(compositeKey)
|
| 285 |
}
|
| 286 |
|
| 287 |
+
function getCompositeKey(group: BenchmarkGroup): string {
|
| 288 |
const evaluation = group.variants[0]?.evaluation
|
| 289 |
const backendSuiteKey =
|
| 290 |
evaluation?.benchmark_parent_key ||
|
| 291 |
evaluation?.benchmark_family_key ||
|
| 292 |
evaluation?.benchmark
|
| 293 |
|
| 294 |
+
return normalizeCompositeKey(backendSuiteKey ?? group.key)
|
| 295 |
}
|
| 296 |
|
| 297 |
+
function getCompositeDisplayName(key: string): string {
|
| 298 |
+
const normalizedKey = normalizeCompositeKey(key)
|
| 299 |
return SUITE_DISPLAY_NAMES[normalizedKey] ?? normalizeDisplayLabel(key)
|
| 300 |
}
|
| 301 |
|
| 302 |
+
function getCompositeName(group: BenchmarkGroup, compositeKey: string): string {
|
| 303 |
const evaluation = group.variants[0]?.evaluation
|
| 304 |
const benchmarkCardName = group.benchmarkCard?.benchmark_details?.name
|
| 305 |
const backendParentName = evaluation?.benchmark_parent_name
|
| 306 |
const backendFamilyName = evaluation?.benchmark_family_name
|
| 307 |
|
| 308 |
+
if (doesLabelMatchSuiteKey(backendParentName, compositeKey)) {
|
| 309 |
return normalizeDisplayLabel(backendParentName)
|
| 310 |
}
|
| 311 |
|
| 312 |
+
if (doesLabelMatchSuiteKey(backendFamilyName, compositeKey)) {
|
| 313 |
return normalizeDisplayLabel(backendFamilyName)
|
| 314 |
}
|
| 315 |
|
| 316 |
+
if (doesLabelMatchSuiteKey(benchmarkCardName, compositeKey)) {
|
| 317 |
return normalizeDisplayLabel(benchmarkCardName)
|
| 318 |
}
|
| 319 |
|
| 320 |
+
return getCompositeDisplayName(compositeKey)
|
| 321 |
}
|
| 322 |
|
| 323 |
+
function groupByComposite(
|
| 324 |
groups: BenchmarkGroup[],
|
| 325 |
modelIds: string[],
|
| 326 |
peerRanks: PeerRanksMap
|
| 327 |
+
): CompositeGroup[] {
|
| 328 |
+
const composites = new Map<string, BenchmarkGroup[]>()
|
| 329 |
for (const group of groups) {
|
| 330 |
+
const key = getCompositeKey(group)
|
| 331 |
+
const existing = composites.get(key) ?? []
|
| 332 |
existing.push(group)
|
| 333 |
+
composites.set(key, existing)
|
| 334 |
}
|
| 335 |
|
| 336 |
+
return Array.from(composites.entries()).map(([compositeKey, benchmarks]) => {
|
| 337 |
const scores = benchmarks.map(b => b.avgNormalizedScore).filter(Number.isFinite)
|
| 338 |
const avgScore = scores.length > 0 ? scores.reduce((a, b) => a + b, 0) / scores.length : 0
|
| 339 |
const rawScores = benchmarks.map((benchmark) => benchmark.avgRawScore).filter(Number.isFinite)
|
| 340 |
const avgRawScore = rawScores.length > 0 ? rawScores.reduce((a, b) => a + b, 0) / rawScores.length : 0
|
| 341 |
|
| 342 |
+
// Find best rank across all benchmarks in the composite
|
| 343 |
let bestRank: { position: number; total: number } | null = null
|
| 344 |
for (const b of benchmarks) {
|
| 345 |
const rank = getGroupPeerRank(b, modelIds, peerRanks)
|
|
|
|
| 350 |
}
|
| 351 |
|
| 352 |
return {
|
| 353 |
+
compositeKey,
|
| 354 |
+
compositeName: benchmarks[0] ? getCompositeName(benchmarks[0], compositeKey) : getCompositeDisplayName(compositeKey),
|
| 355 |
benchmarks,
|
| 356 |
avgRawScore,
|
| 357 |
avgNormalizedScore: avgScore,
|
|
|
|
| 359 |
bestRank,
|
| 360 |
}
|
| 361 |
}).sort((a, b) => {
|
| 362 |
+
// Sort by best peer rank ratio (lower = better); unranked composites go to the bottom
|
| 363 |
const aRatio = a.bestRank ? a.bestRank.position / (a.bestRank.total || a.bestRank.position) : Infinity
|
| 364 |
const bRatio = b.bestRank ? b.bestRank.position / (b.bestRank.total || b.bestRank.position) : Infinity
|
| 365 |
if (aRatio !== bRatio) return aRatio - bRatio
|
|
|
|
| 466 |
function getVariantDescriptor(
|
| 467 |
evaluation: BenchmarkEvaluation,
|
| 468 |
result: EvaluationResult
|
| 469 |
+
): Pick<BenchmarkVariant, "label" | "variantType" | "metricLabel" | "setupLabel" | "sliceLabel"> {
|
| 470 |
const evaluationVariantRaw = getEvaluationVariantLabel(evaluation)
|
| 471 |
const evaluationVariant = evaluationVariantRaw ? formatSetupDisplayLabel(evaluationVariantRaw) : null
|
| 472 |
const metricLabel = getMetricDisplayLabel(result)
|
| 473 |
const metricKey = normalizeDisplayKey(metricLabel)
|
| 474 |
const metricIsAmbiguous = AMBIGUOUS_GROUP_LABELS.has(metricKey)
|
| 475 |
+
const sliceLabel = evaluation.slice_name ? normalizeDisplayLabel(evaluation.slice_name) : null
|
| 476 |
const setupLabel = evaluationVariant ? formatSetupDisplayLabel(evaluationVariant) : null
|
| 477 |
+
const baseLabel = sliceLabel
|
| 478 |
+
? (metricIsAmbiguous ? sliceLabel : `${sliceLabel} · ${metricLabel}`)
|
| 479 |
: metricLabel
|
| 480 |
|
| 481 |
+
if (setupLabel && sliceLabel) {
|
| 482 |
return {
|
| 483 |
label: `${setupLabel} · ${baseLabel}`,
|
| 484 |
+
variantType: "setup+slice",
|
| 485 |
metricLabel,
|
| 486 |
setupLabel,
|
| 487 |
+
sliceLabel,
|
| 488 |
}
|
| 489 |
}
|
| 490 |
|
|
|
|
| 494 |
variantType: "setup",
|
| 495 |
metricLabel,
|
| 496 |
setupLabel,
|
| 497 |
+
sliceLabel: null,
|
| 498 |
}
|
| 499 |
}
|
| 500 |
|
| 501 |
+
if (sliceLabel || !metricIsAmbiguous) {
|
| 502 |
return {
|
| 503 |
label: baseLabel,
|
| 504 |
+
variantType: sliceLabel ? "slice" : "default",
|
| 505 |
metricLabel,
|
| 506 |
setupLabel: null,
|
| 507 |
+
sliceLabel: sliceLabel ?? null,
|
| 508 |
}
|
| 509 |
}
|
| 510 |
|
|
|
|
| 513 |
variantType: "default",
|
| 514 |
metricLabel,
|
| 515 |
setupLabel: null,
|
| 516 |
+
sliceLabel: null,
|
| 517 |
}
|
| 518 |
}
|
| 519 |
|
|
|
|
| 603 |
return row.variant.setupLabel
|
| 604 |
}
|
| 605 |
|
| 606 |
+
if (row.variant.variantType === "slice") {
|
| 607 |
return "Default setup"
|
| 608 |
}
|
| 609 |
|
|
|
|
| 865 |
switch (variantType) {
|
| 866 |
case "setup":
|
| 867 |
return "bg-sky-100 text-sky-800 dark:bg-sky-950/50 dark:text-sky-300"
|
| 868 |
+
case "slice":
|
| 869 |
return "bg-violet-100 text-violet-800 dark:bg-violet-950/50 dark:text-violet-300"
|
| 870 |
+
case "setup+slice":
|
| 871 |
return "bg-amber-100 text-amber-800 dark:bg-amber-950/50 dark:text-amber-300"
|
| 872 |
default:
|
| 873 |
return "bg-muted text-muted-foreground"
|
|
|
|
| 878 |
switch (variantType) {
|
| 879 |
case "setup":
|
| 880 |
return "Setup change"
|
| 881 |
+
case "slice":
|
| 882 |
+
return "Benchmark slice"
|
| 883 |
+
case "setup+slice":
|
| 884 |
+
return "Setup + slice"
|
| 885 |
default:
|
| 886 |
return "Single run"
|
| 887 |
}
|
|
|
|
| 911 |
}
|
| 912 |
|
| 913 |
function getVariantPrimaryLabel(variant: BenchmarkVariant, groupTitle: string) {
|
| 914 |
+
if (variant.sliceLabel) {
|
| 915 |
+
return variant.sliceLabel
|
| 916 |
}
|
| 917 |
|
| 918 |
if (variant.metricLabel) {
|
|
|
|
| 929 |
return variant.label
|
| 930 |
}
|
| 931 |
|
| 932 |
+
function getGroupSliceLabels(group: BenchmarkGroup) {
|
| 933 |
return Array.from(
|
| 934 |
new Set(
|
| 935 |
group.variants
|
| 936 |
+
.map((variant) => variant.sliceLabel?.trim())
|
| 937 |
.filter((label): label is string => Boolean(label))
|
| 938 |
)
|
| 939 |
)
|
| 940 |
}
|
| 941 |
|
| 942 |
+
function getGroupSliceCount(group: BenchmarkGroup) {
|
| 943 |
+
return getGroupSliceLabels(group).length
|
| 944 |
}
|
| 945 |
|
| 946 |
function getBenchmarkGroupHeading(group: BenchmarkGroup) {
|
| 947 |
return group.canonicalTitle
|
| 948 |
}
|
| 949 |
|
| 950 |
+
function getCompositeBadgeMeta(composite: CompositeGroup) {
|
| 951 |
+
if (composite.benchmarks.length > 1) {
|
| 952 |
return {
|
| 953 |
+
count: composite.benchmarks.length,
|
| 954 |
+
label: `sub-benchmark${composite.benchmarks.length === 1 ? "" : "s"}`,
|
| 955 |
className:
|
| 956 |
"border-sky-200/80 bg-sky-50/70 text-sky-700 dark:border-sky-900/60 dark:bg-sky-950/30 dark:text-sky-300",
|
| 957 |
}
|
| 958 |
}
|
| 959 |
|
| 960 |
+
const singleGroup = composite.benchmarks[0]
|
| 961 |
if (!singleGroup) {
|
| 962 |
return null
|
| 963 |
}
|
| 964 |
|
| 965 |
+
const compositeMatchesBenchmark = normalizeCompositeKey(composite.compositeName) === normalizeCompositeKey(singleGroup.title)
|
| 966 |
+
if (!compositeMatchesBenchmark) {
|
| 967 |
return {
|
| 968 |
count: 1,
|
| 969 |
label: "sub-benchmark",
|
|
|
|
| 972 |
}
|
| 973 |
}
|
| 974 |
|
| 975 |
+
const sliceCount = getGroupSliceCount(singleGroup)
|
| 976 |
+
if (sliceCount > 0) {
|
| 977 |
return {
|
| 978 |
+
count: sliceCount,
|
| 979 |
+
label: `slice${sliceCount === 1 ? "" : "s"}`,
|
| 980 |
className:
|
| 981 |
"border-emerald-200/70 bg-emerald-50 text-emerald-700 dark:border-emerald-900/60 dark:bg-emerald-950/30 dark:text-emerald-300",
|
| 982 |
}
|
|
|
|
| 1405 |
metricLabel: variant.metricLabel,
|
| 1406 |
variantType: variant.variantType,
|
| 1407 |
setupLabel: variant.setupLabel,
|
| 1408 |
+
sliceLabel: variant.sliceLabel,
|
| 1409 |
displayScore: variant.displayScore,
|
| 1410 |
sourceOrganization: getOrganizationDisplayName(variant.evaluation.source_metadata.source_organization_name),
|
| 1411 |
sourceName: normalizeDisplayLabel(variant.evaluation.source_metadata.source_name ?? ""),
|
|
|
|
| 1459 |
variantType: descriptor.variantType,
|
| 1460 |
metricLabel: descriptor.metricLabel,
|
| 1461 |
setupLabel: descriptor.setupLabel,
|
| 1462 |
+
sliceLabel: descriptor.sliceLabel,
|
| 1463 |
displayScore,
|
| 1464 |
normalizedScore,
|
| 1465 |
rankPosition,
|
|
|
|
| 1570 |
|
| 1571 |
group.variants = Array.from(dedupedVariants.values())
|
| 1572 |
group.variants.sort((a, b) => {
|
| 1573 |
+
const aIsSlice = Boolean(a.evaluation.slice_key)
|
| 1574 |
+
const bIsSlice = Boolean(b.evaluation.slice_key)
|
| 1575 |
+
if (aIsSlice !== bIsSlice) {
|
| 1576 |
+
return aIsSlice ? 1 : -1
|
| 1577 |
}
|
| 1578 |
|
| 1579 |
const aPrimaryLabel = getVariantPrimaryLabel(a, group.title)
|
|
|
|
| 2043 |
}))
|
| 2044 |
}, [filteredBenchmarkGroups, comparisonIndex, summary.categories_covered])
|
| 2045 |
|
| 2046 |
+
const compositeGroups = useMemo(() => {
|
| 2047 |
+
const groups = groupByComposite(filteredBenchmarkGroups, modelIds, peerRanks)
|
| 2048 |
+
// Re-sort composites by max relevance of their benchmarks
|
| 2049 |
return groups.sort((a, b) => {
|
| 2050 |
const aMax = Math.max(...a.benchmarks.map(getRelevanceScore))
|
| 2051 |
const bMax = Math.max(...b.benchmarks.map(getRelevanceScore))
|
|
|
|
| 2053 |
})
|
| 2054 |
}, [filteredBenchmarkGroups, modelIds, peerRanks, getRelevanceScore])
|
| 2055 |
|
| 2056 |
+
const categoryCompositeSections = useMemo(
|
| 2057 |
() =>
|
| 2058 |
groupedFilteredBenchmarkGroups
|
| 2059 |
.map(({ category, groups }) => ({
|
| 2060 |
category,
|
| 2061 |
+
composites: groupByComposite(groups, modelIds, peerRanks).sort((a, b) => {
|
| 2062 |
const aMax = Math.max(...a.benchmarks.map(getRelevanceScore))
|
| 2063 |
const bMax = Math.max(...b.benchmarks.map(getRelevanceScore))
|
| 2064 |
return bMax - aMax
|
| 2065 |
}),
|
| 2066 |
}))
|
| 2067 |
+
.filter((section) => section.composites.length > 0),
|
| 2068 |
[groupedFilteredBenchmarkGroups, modelIds, peerRanks, getRelevanceScore]
|
| 2069 |
)
|
| 2070 |
|
| 2071 |
const categoryScoreRanges = useMemo(() => {
|
| 2072 |
const ranges = new Map<CategoryType, ScoreRange>()
|
| 2073 |
|
| 2074 |
+
for (const section of categoryCompositeSections) {
|
| 2075 |
ranges.set(
|
| 2076 |
section.category,
|
| 2077 |
+
getScoreRange(section.composites.map((composite) => composite.avgNormalizedScore))
|
| 2078 |
)
|
| 2079 |
}
|
| 2080 |
|
| 2081 |
return ranges
|
| 2082 |
+
}, [categoryCompositeSections])
|
| 2083 |
|
| 2084 |
+
const compositeBenchmarkScoreRanges = useMemo(() => {
|
| 2085 |
const ranges = new Map<string, ScoreRange>()
|
| 2086 |
|
| 2087 |
+
for (const section of categoryCompositeSections) {
|
| 2088 |
+
for (const composite of section.composites) {
|
| 2089 |
ranges.set(
|
| 2090 |
+
composite.compositeKey,
|
| 2091 |
+
getScoreRange(composite.benchmarks.map((group) => group.avgNormalizedScore))
|
| 2092 |
)
|
| 2093 |
}
|
| 2094 |
}
|
| 2095 |
|
| 2096 |
return ranges
|
| 2097 |
+
}, [categoryCompositeSections])
|
| 2098 |
|
| 2099 |
const benchmarkGroupLookup = useMemo(
|
| 2100 |
() => new Map(benchmarkGroups.map((group) => [group.key, group] as const)),
|
|
|
|
| 2104 |
? benchmarkGroupLookup.get(activeBenchmarkGroupKey) ?? null
|
| 2105 |
: null
|
| 2106 |
|
| 2107 |
+
const toggleSuite = (compositeKey: string) => {
|
| 2108 |
setExpandedSuites((prev) => {
|
| 2109 |
const next = new Set(prev)
|
| 2110 |
+
if (next.has(compositeKey)) next.delete(compositeKey)
|
| 2111 |
+
else next.add(compositeKey)
|
| 2112 |
return next
|
| 2113 |
})
|
| 2114 |
}
|
|
|
|
| 2150 |
)
|
| 2151 |
const repeatedBenchmarkCount = overviewBenchmarkGroups.filter((group) => group.variants.length > 1).length
|
| 2152 |
const setupDrivenBenchmarkCount = overviewBenchmarkGroups.filter((group) =>
|
| 2153 |
+
group.variants.some((variant) => variant.variantType === "setup" || variant.variantType === "setup+slice")
|
| 2154 |
).length
|
| 2155 |
+
const sliceDrivenBenchmarkCount = overviewBenchmarkGroups.filter((group) =>
|
| 2156 |
+
group.variants.some((variant) => variant.variantType === "slice" || variant.variantType === "setup+slice")
|
| 2157 |
).length
|
| 2158 |
|
| 2159 |
useEffect(() => {
|
|
|
|
| 2414 |
summary.model_info.name,
|
| 2415 |
])
|
| 2416 |
|
| 2417 |
+
// A plotbox can expose a top-level "view" selector (slices, child
|
| 2418 |
// benchmarks, components) and an optional metric tab rail beneath the chart.
|
| 2419 |
// Plotbox grouping is driven entirely by comparison-index's own
|
| 2420 |
// benchmark_family_key so it stays in sync with the backend.
|
|
|
|
| 2448 |
familyName: string
|
| 2449 |
category: CategoryType
|
| 2450 |
kind: "single-eval" | "multi-eval"
|
| 2451 |
+
childKindLabel: "metric" | "benchmark" | "component" | "slice" | null
|
| 2452 |
views: PlotboxView[]
|
| 2453 |
primaryGroup: BenchmarkGroup
|
| 2454 |
}
|
|
|
|
| 2534 |
const { familyName, category, resolved } = bucket
|
| 2535 |
|
| 2536 |
if (resolved.length === 1) {
|
| 2537 |
+
// One eval in scope — slices/splits become the view selector while
|
| 2538 |
// metrics move to a compact tab rail beneath the chart.
|
| 2539 |
const { group, evalEntry } = resolved[0]
|
| 2540 |
const evalDisplay =
|
|
|
|
| 2545 |
>()
|
| 2546 |
|
| 2547 |
for (const variant of group.variants) {
|
| 2548 |
+
const viewKey = variant.sliceLabel
|
| 2549 |
+
? `slice:${normalizeDisplayKey(variant.sliceLabel)}`
|
| 2550 |
: "default"
|
| 2551 |
+
const label = variant.sliceLabel || "Overall"
|
| 2552 |
const bucketForView = singleEvalViewBuckets.get(viewKey) ?? {
|
| 2553 |
viewKey,
|
| 2554 |
label,
|
|
|
|
| 2600 |
familyName: evalDisplay,
|
| 2601 |
category,
|
| 2602 |
kind: "single-eval",
|
| 2603 |
+
childKindLabel: views.length > 1 ? "slice" : null,
|
| 2604 |
views,
|
| 2605 |
primaryGroup: group,
|
| 2606 |
})
|
|
|
|
| 2656 |
if (views.length === 0) continue
|
| 2657 |
|
| 2658 |
let hasComponent = false
|
| 2659 |
+
let hasSlice = false
|
| 2660 |
let hasDistinctLeaves = false
|
| 2661 |
for (const r of children) {
|
| 2662 |
const leafKey = r.evalEntry.benchmark_leaf_key
|
|
|
|
| 2666 |
if (r.group.variants[0]?.evaluation.benchmark_component_key ?? null) {
|
| 2667 |
hasComponent = true
|
| 2668 |
} else {
|
| 2669 |
+
hasSlice = true
|
| 2670 |
}
|
| 2671 |
}
|
| 2672 |
const childKindLabel: PlotboxUnit["childKindLabel"] =
|
| 2673 |
+
hasComponent && hasSlice
|
| 2674 |
? "component"
|
| 2675 |
: hasComponent
|
| 2676 |
? "metric"
|
| 2677 |
: hasDistinctLeaves
|
| 2678 |
? "benchmark"
|
| 2679 |
+
: "slice"
|
| 2680 |
|
| 2681 |
units.push({
|
| 2682 |
unitKey: `family:${famKey}`,
|
|
|
|
| 2909 |
const childKindPlural =
|
| 2910 |
unit.childKindLabel === "metric"
|
| 2911 |
? childKindCount === 1 ? "metric" : "metrics"
|
| 2912 |
+
: unit.childKindLabel === "slice"
|
| 2913 |
+
? childKindCount === 1 ? "slice" : "slices"
|
| 2914 |
: unit.childKindLabel === "benchmark"
|
| 2915 |
? childKindCount === 1 ? "benchmark" : "benchmarks"
|
| 2916 |
: childKindCount === 1 ? "component" : "components"
|
|
|
|
| 3395 |
? `${reportingStats.missingGenerationConfigs} entries are missing generation config, limiting cross-slice comparability.`
|
| 3396 |
: "Generation configuration is present across the result set."}
|
| 3397 |
</p>
|
| 3398 |
+
{(setupDrivenBenchmarkCount > 0 || sliceDrivenBenchmarkCount > 0) && (
|
| 3399 |
<p className="text-[13px] leading-[1.7] text-[color:var(--fg-muted)]">
|
| 3400 |
Decomposition: <span className="text-[color:var(--fg)]">{setupDrivenBenchmarkCount}</span> setup-aware ·{" "}
|
| 3401 |
+
<span className="text-[color:var(--fg)]">{sliceDrivenBenchmarkCount}</span> slice-aware.
|
| 3402 |
{reportingStats.libraryList.length > 0 && (
|
| 3403 |
<>
|
| 3404 |
{" "}Eval libraries: <span className="text-[color:var(--fg)]">{reportingStats.libraryList.join(", ")}</span>.
|
|
|
|
| 3531 |
|
| 3532 |
<p className="text-[14px] leading-[1.7] text-[color:var(--fg-muted)] max-w-[64rem] mb-6">
|
| 3533 |
{isResearchView
|
| 3534 |
+
? "Benchmark-first view of this model's reported results, grouped by category. Setup spread and slice-vs-setup differences surface up-front."
|
| 3535 |
: "The public evidence behind this model, grouped by category. The strongest and most variable signals are listed first."}
|
| 3536 |
{policyHighlights.length > 0 && !isResearchView && (
|
| 3537 |
<>
|
|
|
|
| 3606 |
)}
|
| 3607 |
{repeatedBenchmarkCount > 0 && (
|
| 3608 |
<>
|
| 3609 |
+
<dt>Slice spread</dt>
|
| 3610 |
<dd>
|
| 3611 |
+
{repeatedBenchmarkCount} benchmark{repeatedBenchmarkCount === 1 ? "" : "s"} include multiple slices or setups.
|
| 3612 |
</dd>
|
| 3613 |
</>
|
| 3614 |
)}
|
|
|
|
| 3701 |
<option value="rank">Sort · Best rank</option>
|
| 3702 |
<option value="score">Sort · Highest score</option>
|
| 3703 |
<option value="name">Sort · Name (A–Z)</option>
|
| 3704 |
+
<option value="variants">Sort · Most slices</option>
|
| 3705 |
<option value="spread">Sort · Largest spread</option>
|
| 3706 |
</select>
|
| 3707 |
</div>
|
|
|
|
| 3856 |
<span>{row.variant.result.generation_config.num_few_shot}-shot</span>
|
| 3857 |
)}
|
| 3858 |
{row.variant.setupLabel && <span>· {row.variant.setupLabel}</span>}
|
| 3859 |
+
{row.variant.sliceLabel && <span>· {row.variant.sliceLabel}</span>}
|
| 3860 |
</div>
|
| 3861 |
)}
|
| 3862 |
</div>
|
|
|
|
| 4527 |
<Separator className="my-4" />
|
| 4528 |
<div className="mb-2">
|
| 4529 |
<div className="text-xs font-semibold text-muted-foreground uppercase tracking-wider">Detailed Breakdown</div>
|
| 4530 |
+
<div className="text-xs text-muted-foreground mt-1">Scores and structured metadata for individual slices or metrics</div>
|
| 4531 |
</div>
|
| 4532 |
|
| 4533 |
{numericBreakdown.length > 0 && (
|
| 4534 |
<div className="grid grid-cols-1 sm:grid-cols-2 xl:grid-cols-3 gap-3">
|
| 4535 |
{numericBreakdown.map(([key, value]) => {
|
| 4536 |
let valDisplay = typeof value === 'number' ? value.toFixed(2) : value;
|
| 4537 |
+
let normalized_slice = 0;
|
| 4538 |
|
| 4539 |
if (typeof value === 'number') {
|
| 4540 |
if (unit === 'accuracy' || !unit || unit === 'pass@1') {
|
| 4541 |
valDisplay = formatRawScoreValue(value);
|
| 4542 |
+
normalized_slice = value;
|
| 4543 |
} else {
|
| 4544 |
valDisplay = value.toFixed(2);
|
| 4545 |
+
normalized_slice = (value - min_score) / (max_score - min_score);
|
| 4546 |
}
|
| 4547 |
}
|
| 4548 |
|
|
|
|
| 4556 |
{valDisplay}
|
| 4557 |
</div>
|
| 4558 |
{typeof value === 'number' && (
|
| 4559 |
+
<Progress value={normalized_slice * 100} className="h-1 mt-2" />
|
| 4560 |
)}
|
| 4561 |
</div>
|
| 4562 |
)})}
|
|
|
|
| 4800 |
Number.isFinite(latestTimestamp) ? formatCompactDate(String(latestTimestamp)) : formatCompactDate(group.variants[0]?.evaluation.retrieved_timestamp ?? "")
|
| 4801 |
const compactDomains = group.domains.slice(0, 2)
|
| 4802 |
const progressWidth = Math.max(4, Math.min(100, group.avgNormalizedScore * 100))
|
| 4803 |
+
const sliceCount = getGroupSliceCount(group)
|
| 4804 |
|
| 4805 |
const toggleRow = (rowKey: string) => {
|
| 4806 |
setExpandedRows((current) => ({
|
|
|
|
| 4851 |
card
|
| 4852 |
</span>
|
| 4853 |
)}
|
| 4854 |
+
{sliceCount > 0 && (
|
| 4855 |
<span className="shrink-0 rounded-full border border-emerald-200/70 bg-emerald-50 px-1.5 py-0.5 text-[9px] font-semibold uppercase tracking-[0.12em] text-emerald-700 dark:border-emerald-900/60 dark:bg-emerald-950/30 dark:text-emerald-300">
|
| 4856 |
+
{sliceCount} slice{sliceCount === 1 ? "" : "s"}
|
| 4857 |
</span>
|
| 4858 |
)}
|
| 4859 |
{compactDomains.map((domain) => (
|
|
|
|
| 4881 |
)}
|
| 4882 |
</div>
|
| 4883 |
|
| 4884 |
+
{/* Slice count */}
|
| 4885 |
<span className="shrink-0 text-[11px] text-muted-foreground w-16 text-right hidden sm:block">
|
| 4886 |
{group.variants.length} {group.variants.length === 1 ? "row" : "rows"}
|
| 4887 |
</span>
|
|
|
|
| 5077 |
</div>
|
| 5078 |
<div className="text-xs text-muted-foreground">
|
| 5079 |
{variant.setupLabel && <span>Setup: {variant.setupLabel}</span>}
|
| 5080 |
+
{variant.setupLabel && variant.sliceLabel && <span> • </span>}
|
| 5081 |
+
{variant.sliceLabel && <span>Slice: {variant.sliceLabel}</span>}
|
| 5082 |
+
{!variant.setupLabel && !variant.sliceLabel && <span>{group.title}</span>}
|
| 5083 |
</div>
|
| 5084 |
</div>
|
| 5085 |
</div>
|
|
|
|
| 5193 |
const [resolvedRanks, setResolvedRanks] = useState<Record<string, { position: number; total: number | null }>>({})
|
| 5194 |
const [isResolvingRanks, setIsResolvingRanks] = useState(false)
|
| 5195 |
const compactDomains = group.domains.slice(0, 2)
|
| 5196 |
+
const sliceCount = getGroupSliceCount(group)
|
| 5197 |
+
const hasSliceMatrix = sliceCount > 0
|
| 5198 |
const sourceOrganizations = useMemo(
|
| 5199 |
() => new Set(group.variants.map((variant) => getOrganizationDisplayName(variant.evaluation.source_metadata.source_organization_name))),
|
| 5200 |
[group.variants]
|
|
|
|
| 5335 |
// Kept only to drive the single-setup overview: when every reported row is
|
| 5336 |
// reported under the same setup, the detail table collapses into a compact
|
| 5337 |
// view that drops the redundant "Reporting setup" column.
|
| 5338 |
+
const sliceSetups = useMemo(() => {
|
| 5339 |
+
if (!hasSliceMatrix) return null
|
| 5340 |
const setupOrder: string[] = []
|
| 5341 |
for (const row of variantRows) {
|
| 5342 |
const setupDisplayLabel = formatSetupDisplayLabel(row.variant.setupLabel)
|
| 5343 |
if (!setupOrder.includes(setupDisplayLabel)) setupOrder.push(setupDisplayLabel)
|
| 5344 |
}
|
| 5345 |
return { setupOrder }
|
| 5346 |
+
}, [hasSliceMatrix, variantRows])
|
| 5347 |
|
| 5348 |
+
const useSingleSetupOverview = Boolean(sliceSetups && sliceSetups.setupOrder.length === 1)
|
| 5349 |
+
const singleSetupDisplayLabel = useSingleSetupOverview ? sliceSetups?.setupOrder[0] ?? null : null
|
| 5350 |
|
| 5351 |
useEffect(() => {
|
| 5352 |
const pendingRows = variantRows.filter(
|
|
|
|
| 5411 |
</DialogTitle>
|
| 5412 |
<DialogDescription className="mt-1.5 text-[13px] leading-[1.5] text-[color:var(--fg-muted)]">
|
| 5413 |
{isResearchView
|
| 5414 |
+
? "Inspect setup slices, score details, and source provenance in one focused view."
|
| 5415 |
: "Inspect reporting setup and evidence details before interpreting benchmark position."}
|
| 5416 |
</DialogDescription>
|
| 5417 |
{(compactDomains.length > 0 || group.benchmarkCard) && (
|
|
|
|
| 5462 |
<div className="kicker">Sources</div>
|
| 5463 |
<div className="mt-1 text-[18px] font-semibold tabular-nums">
|
| 5464 |
{sourceOrganizations.size}
|
| 5465 |
+
{hasSliceMatrix && (
|
| 5466 |
<span className="ml-2 font-mono text-[10px] tracking-[0.12em] uppercase text-[color:var(--fg-subtle)]">
|
| 5467 |
+
· {sliceCount} slice{sliceCount === 1 ? "" : "s"}
|
| 5468 |
</span>
|
| 5469 |
)}
|
| 5470 |
</div>
|
|
|
|
| 5482 |
|
| 5483 |
{/* Sources — distinct reporting orgs and dataset links for this group.
|
| 5484 |
Pulled up to the top of the deep-dive so the per-row table can
|
| 5485 |
+
stay focused on slice / setup / score. */}
|
| 5486 |
{(() => {
|
| 5487 |
type SourceEntry = {
|
| 5488 |
key: string
|
|
|
|
| 5581 |
{useSingleSetupOverview ? (
|
| 5582 |
<section>
|
| 5583 |
<div className="section-head">
|
| 5584 |
+
<h2>{hasAmbiguousPrimaryLabels ? "Reported runs" : "Slice overview"}</h2>
|
| 5585 |
<span className="font-mono text-[10px] uppercase tracking-[0.15em] text-[color:var(--fg-subtle)]">
|
| 5586 |
{singleSetupDisplayLabel ? `${singleSetupDisplayLabel} · ` : ""}
|
| 5587 |
{variantRows.length} row{variantRows.length === 1 ? "" : "s"}
|
|
|
|
| 5594 |
? "These rows share the same benchmark label, so run names or differing config fields are surfaced to show what changed across reports."
|
| 5595 |
: "These rows describe the same benchmark view, so the table surfaces the reported run name or setup differences that separate them."
|
| 5596 |
: isResearchView
|
| 5597 |
+
? "This benchmark reports one setup, so slices, scores, and provenance are merged into one comparison view."
|
| 5598 |
+
: "This benchmark only reports one setup, so the slice evidence is consolidated into a single reader-friendly view."}
|
| 5599 |
</p>
|
| 5600 |
|
| 5601 |
<div className="min-h-0 overflow-auto">
|
|
|
|
| 5603 |
<thead>
|
| 5604 |
<tr>
|
| 5605 |
<th className="w-[60%]">
|
| 5606 |
+
{hasAmbiguousPrimaryLabels ? "Reported row" : "Slice"}
|
| 5607 |
</th>
|
| 5608 |
<th className="w-[20%]">Setup detail</th>
|
| 5609 |
<th className="num w-[10%]">Score</th>
|
|
|
|
| 5678 |
</span>
|
| 5679 |
</div>
|
| 5680 |
<p className="mb-4 max-w-[60rem] text-[13px] leading-[1.65] text-[color:var(--fg-muted)]">
|
| 5681 |
+
Primary row labels show the benchmark slice or slice. Setup and source details sit alongside each row.
|
| 5682 |
</p>
|
| 5683 |
|
| 5684 |
<div className="min-h-0 overflow-auto">
|
| 5685 |
<table className="ec-htable table-fixed">
|
| 5686 |
<thead>
|
| 5687 |
<tr>
|
| 5688 |
+
<th className="w-[46%]">Slice</th>
|
| 5689 |
<th className="w-[36%]">Reporting setup</th>
|
| 5690 |
<th className="num w-[9%]">Score</th>
|
| 5691 |
<th className="num w-[9%]">Rank</th>
|
|
|
|
| 5895 |
)}
|
| 5896 |
{sourceData?.dataset_version && <InlineMeta label="Dataset Version" value={sourceData.dataset_version} />}
|
| 5897 |
{sourceData?.hf_split && <InlineMeta label="Split" value={sourceData.hf_split} />}
|
| 5898 |
+
{variant.sliceLabel && <InlineMeta label="Slice" value={normalizeDisplayLabel(variant.sliceLabel)} />}
|
| 5899 |
{variant.setupLabel && <InlineMeta label="Setup" value={formatSetupDisplayLabel(variant.setupLabel)} />}
|
| 5900 |
{inferencePlatform && <InlineMeta label="Inference Platform" value={inferencePlatform} />}
|
| 5901 |
{variant.evaluation.source_metadata.source_name && (
|
|
|
|
| 5952 |
{numericBreakdown.length > 0 && (
|
| 5953 |
<div className="space-y-3">
|
| 5954 |
<div className="text-xs font-semibold uppercase tracking-[0.2em] text-muted-foreground">
|
| 5955 |
+
{isResearchView ? "Slice Scores" : "Reported Metrics"}
|
| 5956 |
</div>
|
| 5957 |
<div className="grid gap-3 md:grid-cols-2 2xl:grid-cols-3">
|
| 5958 |
{numericBreakdown.map(([key, value]) => {
|
|
@@ -135,7 +135,7 @@ function formatScoreValue(value: number | null | undefined) {
|
|
| 135 |
|
| 136 |
function getCoverageSummaryLabel(data: BenchmarkEvaluationCardData) {
|
| 137 |
if (data.benchmarks_count > 0) {
|
| 138 |
-
return `${data.benchmarks_count} benchmark
|
| 139 |
}
|
| 140 |
|
| 141 |
if (data.latest_source_name) {
|
|
@@ -302,7 +302,7 @@ export function BenchmarkEvaluationCard({
|
|
| 302 |
<Badge variant="secondary">{data.variant_count} versions</Badge>
|
| 303 |
)}
|
| 304 |
{paramsBillions && <Badge variant="secondary">{paramsBillions} parameters</Badge>}
|
| 305 |
-
<Badge variant="outline">{data.benchmarks_count} benchmark
|
| 306 |
<Badge variant="outline">{data.evaluations_count} reported results</Badge>
|
| 307 |
{reproducibilityGapCount > 0 && (
|
| 308 |
<Badge
|
|
|
|
| 135 |
|
| 136 |
function getCoverageSummaryLabel(data: BenchmarkEvaluationCardData) {
|
| 137 |
if (data.benchmarks_count > 0) {
|
| 138 |
+
return `${data.benchmarks_count} benchmark composite${data.benchmarks_count === 1 ? "" : "s"} surfaced`
|
| 139 |
}
|
| 140 |
|
| 141 |
if (data.latest_source_name) {
|
|
|
|
| 302 |
<Badge variant="secondary">{data.variant_count} versions</Badge>
|
| 303 |
)}
|
| 304 |
{paramsBillions && <Badge variant="secondary">{paramsBillions} parameters</Badge>}
|
| 305 |
+
<Badge variant="outline">{data.benchmarks_count} benchmark composites</Badge>
|
| 306 |
<Badge variant="outline">{data.evaluations_count} reported results</Badge>
|
| 307 |
{reproducibilityGapCount > 0 && (
|
| 308 |
<Badge
|
|
@@ -120,7 +120,7 @@ export function EvalCard({ summary, delayMs = 0 }: EvalCardProps) {
|
|
| 120 |
<div className="min-w-0">
|
| 121 |
<div className="text-xl font-bold">{summary.evaluation_name}</div>
|
| 122 |
<div className="mt-1 text-xs font-medium uppercase tracking-[0.18em] text-muted-foreground">
|
| 123 |
-
|
| 124 |
</div>
|
| 125 |
<div className="mt-1 text-sm text-muted-foreground line-clamp-2">
|
| 126 |
{/*
|
|
|
|
| 120 |
<div className="min-w-0">
|
| 121 |
<div className="text-xl font-bold">{summary.evaluation_name}</div>
|
| 122 |
<div className="mt-1 text-xs font-medium uppercase tracking-[0.18em] text-muted-foreground">
|
| 123 |
+
Composite: {summary.composite_benchmark_name}
|
| 124 |
</div>
|
| 125 |
<div className="mt-1 text-sm text-muted-foreground line-clamp-2">
|
| 126 |
{/*
|
|
@@ -113,14 +113,14 @@ interface SliceTab {
|
|
| 113 |
* - <= SLICE_PILL_THRESHOLD: render every slice as a pill (current familiar UX).
|
| 114 |
* - > SLICE_PILL_THRESHOLD: render "All slices" + currently-selected pill +
|
| 115 |
* a "Browse N slices" button that opens a searchable dialog. Hundreds of
|
| 116 |
-
*
|
| 117 |
*/
|
| 118 |
function SliceSelector({
|
| 119 |
-
|
| 120 |
onChange,
|
| 121 |
tabs,
|
| 122 |
}: {
|
| 123 |
-
|
| 124 |
onChange: (key: string) => void
|
| 125 |
tabs: SliceTab[]
|
| 126 |
}) {
|
|
@@ -128,7 +128,7 @@ function SliceSelector({
|
|
| 128 |
const [search, setSearch] = useState("")
|
| 129 |
|
| 130 |
const useBrowser = tabs.length > SLICE_PILL_THRESHOLD
|
| 131 |
-
const activeTab = tabs.find((tab) => tab.key ===
|
| 132 |
|
| 133 |
const filteredTabs = useMemo(() => {
|
| 134 |
const query = search.trim().toLowerCase()
|
|
@@ -143,7 +143,7 @@ function SliceSelector({
|
|
| 143 |
<div className="flex flex-wrap gap-2">
|
| 144 |
<button
|
| 145 |
type="button"
|
| 146 |
-
className={`ec-pill${
|
| 147 |
onClick={() => onChange("all")}
|
| 148 |
>
|
| 149 |
All slices
|
|
@@ -152,7 +152,7 @@ function SliceSelector({
|
|
| 152 |
<button
|
| 153 |
key={tab.key}
|
| 154 |
type="button"
|
| 155 |
-
className={`ec-pill${
|
| 156 |
onClick={() => onChange(tab.key)}
|
| 157 |
>
|
| 158 |
{tab.label}
|
|
@@ -172,7 +172,7 @@ function SliceSelector({
|
|
| 172 |
<div className="flex flex-wrap items-center gap-2">
|
| 173 |
<button
|
| 174 |
type="button"
|
| 175 |
-
className={`ec-pill${
|
| 176 |
onClick={() => onChange("all")}
|
| 177 |
>
|
| 178 |
All slices
|
|
@@ -230,11 +230,11 @@ function SliceSelector({
|
|
| 230 |
}}
|
| 231 |
className={cn(
|
| 232 |
"flex w-full items-center justify-between border-b px-4 py-2.5 text-left text-sm transition-colors hover:bg-muted/40",
|
| 233 |
-
|
| 234 |
)}
|
| 235 |
>
|
| 236 |
<span>All slices (no filter)</span>
|
| 237 |
-
{
|
| 238 |
</button>
|
| 239 |
{filteredTabs.length === 0 ? (
|
| 240 |
<div className="px-4 py-6 text-center text-sm text-muted-foreground">
|
|
@@ -251,11 +251,11 @@ function SliceSelector({
|
|
| 251 |
}}
|
| 252 |
className={cn(
|
| 253 |
"flex w-full items-center justify-between border-b px-4 py-2 text-left text-sm transition-colors hover:bg-muted/40 last:border-b-0",
|
| 254 |
-
|
| 255 |
)}
|
| 256 |
>
|
| 257 |
<span className="min-w-0 truncate pr-2">{tab.label}</span>
|
| 258 |
-
{
|
| 259 |
<span className="shrink-0 text-xs text-muted-foreground">selected</span>
|
| 260 |
)}
|
| 261 |
</button>
|
|
@@ -543,10 +543,10 @@ export function EvalDetail({ summary }: EvalDetailProps) {
|
|
| 543 |
: "Not linked"
|
| 544 |
const leaderboardDescription = isResearchView
|
| 545 |
? summary.is_aggregated
|
| 546 |
-
? "Models ranked by average raw score across the
|
| 547 |
: "Models ranked by raw score for this benchmark."
|
| 548 |
: summary.is_aggregated
|
| 549 |
-
? "Averaged model results across the
|
| 550 |
: "Model results with benchmark context, source dataset detail, and optional instance-data links."
|
| 551 |
const reportingCompleteness = summary.evalcards?.annotations?.reporting_completeness
|
| 552 |
const documentationPopulatedCount = reportingCompleteness
|
|
@@ -656,7 +656,7 @@ export function EvalDetail({ summary }: EvalDetailProps) {
|
|
| 656 |
className="font-mono text-[10px] uppercase tracking-[0.12em]"
|
| 657 |
style={{ color: "var(--fg-subtle)" }}
|
| 658 |
>
|
| 659 |
-
metric spec · completeness · comparability{summary.subtasks?.length ? " ·
|
| 660 |
</span>
|
| 661 |
</div>
|
| 662 |
{overviewOpen ? (
|
|
@@ -678,10 +678,10 @@ export function EvalDetail({ summary }: EvalDetailProps) {
|
|
| 678 |
{isResearchView ? "Metric specification" : "Reading context"}
|
| 679 |
</div>
|
| 680 |
<dl className="ec-datalist">
|
| 681 |
-
<dt>
|
| 682 |
<dd>
|
| 683 |
{summary.is_aggregated
|
| 684 |
-
? summary.aggregate_sources?.map((source) => source.composite_benchmark_name).join(", ") || "Multiple
|
| 685 |
: summary.composite_benchmark_name}
|
| 686 |
</dd>
|
| 687 |
<dt>{isResearchView ? "Benchmark ID" : "What this covers"}</dt>
|
|
@@ -744,7 +744,7 @@ export function EvalDetail({ summary }: EvalDetailProps) {
|
|
| 744 |
>
|
| 745 |
<div className="kicker mb-2">Benchmark structure</div>
|
| 746 |
<p className="text-[13px] mb-4" style={{ color: "var(--fg-muted)", maxWidth: 640 }}>
|
| 747 |
-
Benchmark-level summary metrics and
|
| 748 |
</p>
|
| 749 |
|
| 750 |
{summary.root_metrics && summary.root_metrics.length > 0 && (
|
|
@@ -776,15 +776,15 @@ export function EvalDetail({ summary }: EvalDetailProps) {
|
|
| 776 |
className="font-mono uppercase mb-1"
|
| 777 |
style={{ fontSize: 10, letterSpacing: "0.14em", color: "var(--fg-subtle)" }}
|
| 778 |
>
|
| 779 |
-
|
| 780 |
</div>
|
| 781 |
<ul
|
| 782 |
className="flex flex-col"
|
| 783 |
style={{ borderTop: "1px solid var(--border-soft)" }}
|
| 784 |
>
|
| 785 |
-
{summary.subtasks.map((
|
| 786 |
<li
|
| 787 |
-
key={
|
| 788 |
className="grid gap-x-4 py-3"
|
| 789 |
style={{
|
| 790 |
gridTemplateColumns: "minmax(160px, 280px) 1fr",
|
|
@@ -793,20 +793,20 @@ export function EvalDetail({ summary }: EvalDetailProps) {
|
|
| 793 |
>
|
| 794 |
<div className="min-w-0">
|
| 795 |
<div className="font-semibold text-[13px] truncate">
|
| 796 |
-
{
|
| 797 |
</div>
|
| 798 |
-
{
|
| 799 |
<div
|
| 800 |
className="mt-0.5 font-mono text-[10px] uppercase tracking-[0.08em] truncate"
|
| 801 |
style={{ color: "var(--fg-subtle)" }}
|
| 802 |
-
title={
|
| 803 |
>
|
| 804 |
-
{
|
| 805 |
</div>
|
| 806 |
)}
|
| 807 |
</div>
|
| 808 |
<div className="flex flex-wrap items-center gap-1.5">
|
| 809 |
-
{
|
| 810 |
<span
|
| 811 |
key={metric.metric_summary_id}
|
| 812 |
className="ec-tag"
|
|
@@ -919,13 +919,13 @@ export function EvalDetail({ summary }: EvalDetailProps) {
|
|
| 919 |
<tbody>
|
| 920 |
{pagedLeaderboardRows.map(({ key, rank, modelResult, normalizedScore }) => {
|
| 921 |
const isExpanded = expandedRows[key] ?? false
|
| 922 |
-
const
|
| 923 |
? Object.entries(modelResult.score_details.details).filter(([, value]) => typeof value === "number")
|
| 924 |
: []
|
| 925 |
const hasExpandableDetails =
|
| 926 |
isResearchView ||
|
| 927 |
(modelResult.aggregate_components && modelResult.aggregate_components.length > 1) ||
|
| 928 |
-
|
| 929 |
|
| 930 |
const datasetName = Array.isArray(modelResult.source_data)
|
| 931 |
? undefined
|
|
@@ -1256,28 +1256,28 @@ export function EvalDetail({ summary }: EvalDetailProps) {
|
|
| 1256 |
</div>
|
| 1257 |
)}
|
| 1258 |
|
| 1259 |
-
{
|
| 1260 |
<div className="space-y-2">
|
| 1261 |
<div
|
| 1262 |
className="font-mono uppercase"
|
| 1263 |
style={{ fontSize: 10, letterSpacing: "0.14em", color: "var(--fg-subtle)" }}
|
| 1264 |
>
|
| 1265 |
-
|
| 1266 |
</div>
|
| 1267 |
<div className="overflow-x-auto" style={{ border: "1px solid var(--border-soft)" }}>
|
| 1268 |
<table className="ec-htable">
|
| 1269 |
<thead>
|
| 1270 |
<tr>
|
| 1271 |
-
<th>
|
| 1272 |
<th className="num">Raw</th>
|
| 1273 |
</tr>
|
| 1274 |
</thead>
|
| 1275 |
<tbody>
|
| 1276 |
-
{
|
| 1277 |
const numericValue = value as number
|
| 1278 |
return (
|
| 1279 |
-
<tr key={
|
| 1280 |
-
<td className="font-medium text-[13px] capitalize">{
|
| 1281 |
<td className="num font-mono tabular-nums text-[13px]" style={{ color: "var(--fg-muted)" }}>
|
| 1282 |
{formatRawScore(numericValue, summary.metric_config.unit)}
|
| 1283 |
</td>
|
|
@@ -1453,7 +1453,7 @@ function MultiMetricLeaderboard({
|
|
| 1453 |
return root?.column_key ?? metrics[0]?.column_key ?? "model"
|
| 1454 |
})
|
| 1455 |
const [sortDirection, setSortDirection] = useState<"asc" | "desc">("desc")
|
| 1456 |
-
const [
|
| 1457 |
const [minParamStep, setMinParamStep] = useState(0)
|
| 1458 |
const [maxParamStep, setMaxParamStep] = useState(PARAM_RANGE_MAX_INDEX)
|
| 1459 |
const [expandedRows, setExpandedRows] = useState<Record<string, boolean>>({})
|
|
@@ -1486,7 +1486,7 @@ function MultiMetricLeaderboard({
|
|
| 1486 |
const leaderboardRows = summary.leaderboard_rows ?? []
|
| 1487 |
const allMetricKeys = useMemo(() => leaderboardMetrics.map((metric) => metric.column_key), [leaderboardMetrics])
|
| 1488 |
// Cap default visible columns to avoid hangs on benchmarks with hundreds of metrics
|
| 1489 |
-
// (e.g. helm_air_bench has 374
|
| 1490 |
const DEFAULT_VISIBLE_METRIC_CAP = 24
|
| 1491 |
const defaultVisibleMetricKeys = useMemo(
|
| 1492 |
() => allMetricKeys.slice(0, DEFAULT_VISIBLE_METRIC_CAP),
|
|
@@ -1498,7 +1498,7 @@ function MultiMetricLeaderboard({
|
|
| 1498 |
[leaderboardMetrics]
|
| 1499 |
)
|
| 1500 |
const visibleMetricKeySet = useMemo(() => new Set(visibleMetricKeys), [visibleMetricKeys])
|
| 1501 |
-
const
|
| 1502 |
const counts = new Map<string, number>()
|
| 1503 |
for (const metric of leaderboardMetrics) {
|
| 1504 |
if (metric.scope === "subtask" && metric.subtask_key) {
|
|
@@ -1508,16 +1508,16 @@ function MultiMetricLeaderboard({
|
|
| 1508 |
return counts
|
| 1509 |
}, [leaderboardMetrics])
|
| 1510 |
|
| 1511 |
-
const
|
| 1512 |
return leaderboardMetrics
|
| 1513 |
-
.filter((metric) => metric.scope === "subtask" && metric.subtask_key &&
|
| 1514 |
.map((metric) => ({
|
| 1515 |
key: metric.subtask_key as string,
|
| 1516 |
label: metric.subtask_name ?? getCompactMetricLabel(metric.display_name),
|
| 1517 |
}))
|
| 1518 |
-
}, [leaderboardMetrics,
|
| 1519 |
|
| 1520 |
-
const
|
| 1521 |
|
| 1522 |
const visibleMetrics = useMemo(
|
| 1523 |
() =>
|
|
@@ -1526,13 +1526,13 @@ function MultiMetricLeaderboard({
|
|
| 1526 |
return false
|
| 1527 |
}
|
| 1528 |
|
| 1529 |
-
if (!
|
| 1530 |
return true
|
| 1531 |
}
|
| 1532 |
|
| 1533 |
-
return metric.scope === "subtask" && metric.subtask_key ===
|
| 1534 |
}),
|
| 1535 |
-
[
|
| 1536 |
)
|
| 1537 |
const visibleMetricColumnKeySet = useMemo(
|
| 1538 |
() => new Set(visibleMetrics.map((metric) => metric.column_key)),
|
|
@@ -1628,7 +1628,7 @@ function MultiMetricLeaderboard({
|
|
| 1628 |
}, [defaultVisibleMetricKeys, summary.evaluation_id])
|
| 1629 |
|
| 1630 |
useEffect(() => {
|
| 1631 |
-
|
| 1632 |
}, [summary.evaluation_id])
|
| 1633 |
|
| 1634 |
useEffect(() => {
|
|
@@ -1648,21 +1648,21 @@ function MultiMetricLeaderboard({
|
|
| 1648 |
}, [leaderboardMetricMap, leaderboardMetrics, sortKey, visibleMetricColumnKeySet])
|
| 1649 |
|
| 1650 |
useEffect(() => {
|
| 1651 |
-
if (!
|
| 1652 |
-
if (
|
| 1653 |
-
|
| 1654 |
}
|
| 1655 |
return
|
| 1656 |
}
|
| 1657 |
|
| 1658 |
-
if (
|
| 1659 |
return
|
| 1660 |
}
|
| 1661 |
|
| 1662 |
-
if (!
|
| 1663 |
-
|
| 1664 |
}
|
| 1665 |
-
}, [
|
| 1666 |
|
| 1667 |
const pagedRows = useMemo(
|
| 1668 |
() => sortedRows.slice(0, page * 50),
|
|
@@ -1761,7 +1761,7 @@ function MultiMetricLeaderboard({
|
|
| 1761 |
{leaderboardMetrics.map((metric) => {
|
| 1762 |
const isVisible = visibleMetricKeySet.has(metric.column_key)
|
| 1763 |
const isLastVisible = isVisible && visibleMetrics.length === 1
|
| 1764 |
-
const visibleLabel = metric.scope === "subtask" && metric.subtask_key &&
|
| 1765 |
? metric.subtask_name
|
| 1766 |
: getCompactMetricLabel(metric.display_name)
|
| 1767 |
|
|
@@ -1793,7 +1793,7 @@ function MultiMetricLeaderboard({
|
|
| 1793 |
.filter((v): v is number => isNumericScore(v))
|
| 1794 |
if (values.length < 3) return null
|
| 1795 |
const label =
|
| 1796 |
-
metric.scope === "subtask" && metric.subtask_key &&
|
| 1797 |
? metric.subtask_name
|
| 1798 |
: getCompactMetricLabel(metric.display_name)
|
| 1799 |
return {
|
|
@@ -1816,12 +1816,12 @@ function MultiMetricLeaderboard({
|
|
| 1816 |
})()}
|
| 1817 |
|
| 1818 |
<div className="ec-card" style={{ padding: 0, overflow: "hidden" }}>
|
| 1819 |
-
{
|
| 1820 |
<div className="border-b bg-background px-5 py-3 sm:px-6">
|
| 1821 |
<SliceSelector
|
| 1822 |
-
|
| 1823 |
-
onChange={
|
| 1824 |
-
tabs={
|
| 1825 |
/>
|
| 1826 |
</div>
|
| 1827 |
)}
|
|
@@ -1868,13 +1868,13 @@ function MultiMetricLeaderboard({
|
|
| 1868 |
{getSortIndicator("developer")}
|
| 1869 |
</th>
|
| 1870 |
{visibleMetrics.map((metric) => {
|
| 1871 |
-
const
|
| 1872 |
-
!
|
| 1873 |
-
!(metric.scope === "subtask" && metric.subtask_key &&
|
| 1874 |
metric.scope === "subtask" &&
|
| 1875 |
metric.subtask_name
|
| 1876 |
const mainLabel =
|
| 1877 |
-
metric.scope === "subtask" && metric.subtask_key &&
|
| 1878 |
? metric.subtask_name
|
| 1879 |
: getCompactMetricLabel(metric.display_name)
|
| 1880 |
return (
|
|
@@ -1885,7 +1885,7 @@ function MultiMetricLeaderboard({
|
|
| 1885 |
onClick={() => handleSort(metric.column_key)}
|
| 1886 |
title={describeLeaderboardMetric(metric)}
|
| 1887 |
>
|
| 1888 |
-
{
|
| 1889 |
<div
|
| 1890 |
className="font-mono normal-case"
|
| 1891 |
style={{
|
|
|
|
| 113 |
* - <= SLICE_PILL_THRESHOLD: render every slice as a pill (current familiar UX).
|
| 114 |
* - > SLICE_PILL_THRESHOLD: render "All slices" + currently-selected pill +
|
| 115 |
* a "Browse N slices" button that opens a searchable dialog. Hundreds of
|
| 116 |
+
* slices (e.g. AIRBench's 374) fit cleanly.
|
| 117 |
*/
|
| 118 |
function SliceSelector({
|
| 119 |
+
activeSliceTab,
|
| 120 |
onChange,
|
| 121 |
tabs,
|
| 122 |
}: {
|
| 123 |
+
activeSliceTab: string
|
| 124 |
onChange: (key: string) => void
|
| 125 |
tabs: SliceTab[]
|
| 126 |
}) {
|
|
|
|
| 128 |
const [search, setSearch] = useState("")
|
| 129 |
|
| 130 |
const useBrowser = tabs.length > SLICE_PILL_THRESHOLD
|
| 131 |
+
const activeTab = tabs.find((tab) => tab.key === activeSliceTab)
|
| 132 |
|
| 133 |
const filteredTabs = useMemo(() => {
|
| 134 |
const query = search.trim().toLowerCase()
|
|
|
|
| 143 |
<div className="flex flex-wrap gap-2">
|
| 144 |
<button
|
| 145 |
type="button"
|
| 146 |
+
className={`ec-pill${activeSliceTab === "all" ? " on" : ""}`}
|
| 147 |
onClick={() => onChange("all")}
|
| 148 |
>
|
| 149 |
All slices
|
|
|
|
| 152 |
<button
|
| 153 |
key={tab.key}
|
| 154 |
type="button"
|
| 155 |
+
className={`ec-pill${activeSliceTab === tab.key ? " on" : ""}`}
|
| 156 |
onClick={() => onChange(tab.key)}
|
| 157 |
>
|
| 158 |
{tab.label}
|
|
|
|
| 172 |
<div className="flex flex-wrap items-center gap-2">
|
| 173 |
<button
|
| 174 |
type="button"
|
| 175 |
+
className={`ec-pill${activeSliceTab === "all" ? " on" : ""}`}
|
| 176 |
onClick={() => onChange("all")}
|
| 177 |
>
|
| 178 |
All slices
|
|
|
|
| 230 |
}}
|
| 231 |
className={cn(
|
| 232 |
"flex w-full items-center justify-between border-b px-4 py-2.5 text-left text-sm transition-colors hover:bg-muted/40",
|
| 233 |
+
activeSliceTab === "all" && "bg-muted/40 font-semibold"
|
| 234 |
)}
|
| 235 |
>
|
| 236 |
<span>All slices (no filter)</span>
|
| 237 |
+
{activeSliceTab === "all" && <span className="text-xs text-muted-foreground">selected</span>}
|
| 238 |
</button>
|
| 239 |
{filteredTabs.length === 0 ? (
|
| 240 |
<div className="px-4 py-6 text-center text-sm text-muted-foreground">
|
|
|
|
| 251 |
}}
|
| 252 |
className={cn(
|
| 253 |
"flex w-full items-center justify-between border-b px-4 py-2 text-left text-sm transition-colors hover:bg-muted/40 last:border-b-0",
|
| 254 |
+
activeSliceTab === tab.key && "bg-muted/40 font-semibold"
|
| 255 |
)}
|
| 256 |
>
|
| 257 |
<span className="min-w-0 truncate pr-2">{tab.label}</span>
|
| 258 |
+
{activeSliceTab === tab.key && (
|
| 259 |
<span className="shrink-0 text-xs text-muted-foreground">selected</span>
|
| 260 |
)}
|
| 261 |
</button>
|
|
|
|
| 543 |
: "Not linked"
|
| 544 |
const leaderboardDescription = isResearchView
|
| 545 |
? summary.is_aggregated
|
| 546 |
+
? "Models ranked by average raw score across the composite's component benchmarks."
|
| 547 |
: "Models ranked by raw score for this benchmark."
|
| 548 |
: summary.is_aggregated
|
| 549 |
+
? "Averaged model results across the composite's component benchmarks, with drill-down to each component score."
|
| 550 |
: "Model results with benchmark context, source dataset detail, and optional instance-data links."
|
| 551 |
const reportingCompleteness = summary.evalcards?.annotations?.reporting_completeness
|
| 552 |
const documentationPopulatedCount = reportingCompleteness
|
|
|
|
| 656 |
className="font-mono text-[10px] uppercase tracking-[0.12em]"
|
| 657 |
style={{ color: "var(--fg-subtle)" }}
|
| 658 |
>
|
| 659 |
+
metric spec · completeness · comparability{summary.subtasks?.length ? " · slices" : ""}
|
| 660 |
</span>
|
| 661 |
</div>
|
| 662 |
{overviewOpen ? (
|
|
|
|
| 678 |
{isResearchView ? "Metric specification" : "Reading context"}
|
| 679 |
</div>
|
| 680 |
<dl className="ec-datalist">
|
| 681 |
+
<dt>Composite</dt>
|
| 682 |
<dd>
|
| 683 |
{summary.is_aggregated
|
| 684 |
+
? summary.aggregate_sources?.map((source) => source.composite_benchmark_name).join(", ") || "Multiple composites"
|
| 685 |
: summary.composite_benchmark_name}
|
| 686 |
</dd>
|
| 687 |
<dt>{isResearchView ? "Benchmark ID" : "What this covers"}</dt>
|
|
|
|
| 744 |
>
|
| 745 |
<div className="kicker mb-2">Benchmark structure</div>
|
| 746 |
<p className="text-[13px] mb-4" style={{ color: "var(--fg-muted)", maxWidth: 640 }}>
|
| 747 |
+
Benchmark-level summary metrics and slices grouped in one compact section.
|
| 748 |
</p>
|
| 749 |
|
| 750 |
{summary.root_metrics && summary.root_metrics.length > 0 && (
|
|
|
|
| 776 |
className="font-mono uppercase mb-1"
|
| 777 |
style={{ fontSize: 10, letterSpacing: "0.14em", color: "var(--fg-subtle)" }}
|
| 778 |
>
|
| 779 |
+
Slice breakdown · {summary.subtasks.length}
|
| 780 |
</div>
|
| 781 |
<ul
|
| 782 |
className="flex flex-col"
|
| 783 |
style={{ borderTop: "1px solid var(--border-soft)" }}
|
| 784 |
>
|
| 785 |
+
{summary.subtasks.map((slice) => (
|
| 786 |
<li
|
| 787 |
+
key={slice.subtask_key}
|
| 788 |
className="grid gap-x-4 py-3"
|
| 789 |
style={{
|
| 790 |
gridTemplateColumns: "minmax(160px, 280px) 1fr",
|
|
|
|
| 793 |
>
|
| 794 |
<div className="min-w-0">
|
| 795 |
<div className="font-semibold text-[13px] truncate">
|
| 796 |
+
{slice.display_name || slice.subtask_name}
|
| 797 |
</div>
|
| 798 |
+
{slice.canonical_display_name && slice.canonical_display_name !== (slice.display_name || slice.subtask_name) && (
|
| 799 |
<div
|
| 800 |
className="mt-0.5 font-mono text-[10px] uppercase tracking-[0.08em] truncate"
|
| 801 |
style={{ color: "var(--fg-subtle)" }}
|
| 802 |
+
title={slice.canonical_display_name}
|
| 803 |
>
|
| 804 |
+
{slice.canonical_display_name}
|
| 805 |
</div>
|
| 806 |
)}
|
| 807 |
</div>
|
| 808 |
<div className="flex flex-wrap items-center gap-1.5">
|
| 809 |
+
{slice.metrics.map((metric) => (
|
| 810 |
<span
|
| 811 |
key={metric.metric_summary_id}
|
| 812 |
className="ec-tag"
|
|
|
|
| 919 |
<tbody>
|
| 920 |
{pagedLeaderboardRows.map(({ key, rank, modelResult, normalizedScore }) => {
|
| 921 |
const isExpanded = expandedRows[key] ?? false
|
| 922 |
+
const slices = modelResult.score_details.details
|
| 923 |
? Object.entries(modelResult.score_details.details).filter(([, value]) => typeof value === "number")
|
| 924 |
: []
|
| 925 |
const hasExpandableDetails =
|
| 926 |
isResearchView ||
|
| 927 |
(modelResult.aggregate_components && modelResult.aggregate_components.length > 1) ||
|
| 928 |
+
slices.length > 1
|
| 929 |
|
| 930 |
const datasetName = Array.isArray(modelResult.source_data)
|
| 931 |
? undefined
|
|
|
|
| 1256 |
</div>
|
| 1257 |
)}
|
| 1258 |
|
| 1259 |
+
{slices.length > 1 && (
|
| 1260 |
<div className="space-y-2">
|
| 1261 |
<div
|
| 1262 |
className="font-mono uppercase"
|
| 1263 |
style={{ fontSize: 10, letterSpacing: "0.14em", color: "var(--fg-subtle)" }}
|
| 1264 |
>
|
| 1265 |
+
Slice breakdown
|
| 1266 |
</div>
|
| 1267 |
<div className="overflow-x-auto" style={{ border: "1px solid var(--border-soft)" }}>
|
| 1268 |
<table className="ec-htable">
|
| 1269 |
<thead>
|
| 1270 |
<tr>
|
| 1271 |
+
<th>Slice</th>
|
| 1272 |
<th className="num">Raw</th>
|
| 1273 |
</tr>
|
| 1274 |
</thead>
|
| 1275 |
<tbody>
|
| 1276 |
+
{slices.map(([sliceName, value]) => {
|
| 1277 |
const numericValue = value as number
|
| 1278 |
return (
|
| 1279 |
+
<tr key={sliceName}>
|
| 1280 |
+
<td className="font-medium text-[13px] capitalize">{sliceName.replace(/_/g, " ")}</td>
|
| 1281 |
<td className="num font-mono tabular-nums text-[13px]" style={{ color: "var(--fg-muted)" }}>
|
| 1282 |
{formatRawScore(numericValue, summary.metric_config.unit)}
|
| 1283 |
</td>
|
|
|
|
| 1453 |
return root?.column_key ?? metrics[0]?.column_key ?? "model"
|
| 1454 |
})
|
| 1455 |
const [sortDirection, setSortDirection] = useState<"asc" | "desc">("desc")
|
| 1456 |
+
const [activeSliceTab, setActiveSliceTab] = useState<string>("all")
|
| 1457 |
const [minParamStep, setMinParamStep] = useState(0)
|
| 1458 |
const [maxParamStep, setMaxParamStep] = useState(PARAM_RANGE_MAX_INDEX)
|
| 1459 |
const [expandedRows, setExpandedRows] = useState<Record<string, boolean>>({})
|
|
|
|
| 1486 |
const leaderboardRows = summary.leaderboard_rows ?? []
|
| 1487 |
const allMetricKeys = useMemo(() => leaderboardMetrics.map((metric) => metric.column_key), [leaderboardMetrics])
|
| 1488 |
// Cap default visible columns to avoid hangs on benchmarks with hundreds of metrics
|
| 1489 |
+
// (e.g. helm_air_bench has 374 slice×metric pairs). Users can opt in to more.
|
| 1490 |
const DEFAULT_VISIBLE_METRIC_CAP = 24
|
| 1491 |
const defaultVisibleMetricKeys = useMemo(
|
| 1492 |
() => allMetricKeys.slice(0, DEFAULT_VISIBLE_METRIC_CAP),
|
|
|
|
| 1498 |
[leaderboardMetrics]
|
| 1499 |
)
|
| 1500 |
const visibleMetricKeySet = useMemo(() => new Set(visibleMetricKeys), [visibleMetricKeys])
|
| 1501 |
+
const sliceMetricCounts = useMemo(() => {
|
| 1502 |
const counts = new Map<string, number>()
|
| 1503 |
for (const metric of leaderboardMetrics) {
|
| 1504 |
if (metric.scope === "subtask" && metric.subtask_key) {
|
|
|
|
| 1508 |
return counts
|
| 1509 |
}, [leaderboardMetrics])
|
| 1510 |
|
| 1511 |
+
const singleMetricSliceTabs = useMemo(() => {
|
| 1512 |
return leaderboardMetrics
|
| 1513 |
+
.filter((metric) => metric.scope === "subtask" && metric.subtask_key && sliceMetricCounts.get(metric.subtask_key) === 1)
|
| 1514 |
.map((metric) => ({
|
| 1515 |
key: metric.subtask_key as string,
|
| 1516 |
label: metric.subtask_name ?? getCompactMetricLabel(metric.display_name),
|
| 1517 |
}))
|
| 1518 |
+
}, [leaderboardMetrics, sliceMetricCounts])
|
| 1519 |
|
| 1520 |
+
const hasSliceTabs = singleMetricSliceTabs.length > 1
|
| 1521 |
|
| 1522 |
const visibleMetrics = useMemo(
|
| 1523 |
() =>
|
|
|
|
| 1526 |
return false
|
| 1527 |
}
|
| 1528 |
|
| 1529 |
+
if (!hasSliceTabs || activeSliceTab === "all") {
|
| 1530 |
return true
|
| 1531 |
}
|
| 1532 |
|
| 1533 |
+
return metric.scope === "subtask" && metric.subtask_key === activeSliceTab
|
| 1534 |
}),
|
| 1535 |
+
[activeSliceTab, hasSliceTabs, leaderboardMetrics, visibleMetricKeySet]
|
| 1536 |
)
|
| 1537 |
const visibleMetricColumnKeySet = useMemo(
|
| 1538 |
() => new Set(visibleMetrics.map((metric) => metric.column_key)),
|
|
|
|
| 1628 |
}, [defaultVisibleMetricKeys, summary.evaluation_id])
|
| 1629 |
|
| 1630 |
useEffect(() => {
|
| 1631 |
+
setActiveSliceTab("all")
|
| 1632 |
}, [summary.evaluation_id])
|
| 1633 |
|
| 1634 |
useEffect(() => {
|
|
|
|
| 1648 |
}, [leaderboardMetricMap, leaderboardMetrics, sortKey, visibleMetricColumnKeySet])
|
| 1649 |
|
| 1650 |
useEffect(() => {
|
| 1651 |
+
if (!hasSliceTabs) {
|
| 1652 |
+
if (activeSliceTab !== "all") {
|
| 1653 |
+
setActiveSliceTab("all")
|
| 1654 |
}
|
| 1655 |
return
|
| 1656 |
}
|
| 1657 |
|
| 1658 |
+
if (activeSliceTab === "all") {
|
| 1659 |
return
|
| 1660 |
}
|
| 1661 |
|
| 1662 |
+
if (!singleMetricSliceTabs.some((tab) => tab.key === activeSliceTab)) {
|
| 1663 |
+
setActiveSliceTab("all")
|
| 1664 |
}
|
| 1665 |
+
}, [activeSliceTab, hasSliceTabs, singleMetricSliceTabs])
|
| 1666 |
|
| 1667 |
const pagedRows = useMemo(
|
| 1668 |
() => sortedRows.slice(0, page * 50),
|
|
|
|
| 1761 |
{leaderboardMetrics.map((metric) => {
|
| 1762 |
const isVisible = visibleMetricKeySet.has(metric.column_key)
|
| 1763 |
const isLastVisible = isVisible && visibleMetrics.length === 1
|
| 1764 |
+
const visibleLabel = metric.scope === "subtask" && metric.subtask_key && sliceMetricCounts.get(metric.subtask_key) === 1 && metric.subtask_name
|
| 1765 |
? metric.subtask_name
|
| 1766 |
: getCompactMetricLabel(metric.display_name)
|
| 1767 |
|
|
|
|
| 1793 |
.filter((v): v is number => isNumericScore(v))
|
| 1794 |
if (values.length < 3) return null
|
| 1795 |
const label =
|
| 1796 |
+
metric.scope === "subtask" && metric.subtask_key && sliceMetricCounts.get(metric.subtask_key) === 1 && metric.subtask_name
|
| 1797 |
? metric.subtask_name
|
| 1798 |
: getCompactMetricLabel(metric.display_name)
|
| 1799 |
return {
|
|
|
|
| 1816 |
})()}
|
| 1817 |
|
| 1818 |
<div className="ec-card" style={{ padding: 0, overflow: "hidden" }}>
|
| 1819 |
+
{hasSliceTabs && (
|
| 1820 |
<div className="border-b bg-background px-5 py-3 sm:px-6">
|
| 1821 |
<SliceSelector
|
| 1822 |
+
activeSliceTab={activeSliceTab}
|
| 1823 |
+
onChange={setActiveSliceTab}
|
| 1824 |
+
tabs={singleMetricSliceTabs}
|
| 1825 |
/>
|
| 1826 |
</div>
|
| 1827 |
)}
|
|
|
|
| 1868 |
{getSortIndicator("developer")}
|
| 1869 |
</th>
|
| 1870 |
{visibleMetrics.map((metric) => {
|
| 1871 |
+
const showSliceTopline =
|
| 1872 |
+
!hasSliceTabs &&
|
| 1873 |
+
!(metric.scope === "subtask" && metric.subtask_key && sliceMetricCounts.get(metric.subtask_key) === 1) &&
|
| 1874 |
metric.scope === "subtask" &&
|
| 1875 |
metric.subtask_name
|
| 1876 |
const mainLabel =
|
| 1877 |
+
metric.scope === "subtask" && metric.subtask_key && sliceMetricCounts.get(metric.subtask_key) === 1 && metric.subtask_name
|
| 1878 |
? metric.subtask_name
|
| 1879 |
: getCompactMetricLabel(metric.display_name)
|
| 1880 |
return (
|
|
|
|
| 1885 |
onClick={() => handleSort(metric.column_key)}
|
| 1886 |
title={describeLeaderboardMetric(metric)}
|
| 1887 |
>
|
| 1888 |
+
{showSliceTopline && (
|
| 1889 |
<div
|
| 1890 |
className="font-mono normal-case"
|
| 1891 |
style={{
|
|
@@ -4,7 +4,7 @@ import { Fragment, useMemo, useState } from "react"
|
|
| 4 |
import { useRouter } from "next/navigation"
|
| 5 |
import { ArrowUpRight, ChevronDown, ChevronRight } from "lucide-react"
|
| 6 |
|
| 7 |
-
import type { HierarchyFamily, HierarchyLeaf } from "@/lib/backend-artifacts"
|
| 8 |
import type { BenchmarkCard, CategoryType } from "@/lib/benchmark-schema"
|
| 9 |
import type { BenchmarkEvalListItem } from "@/lib/eval-processing"
|
| 10 |
|
|
@@ -67,18 +67,60 @@ interface LeafEntry {
|
|
| 67 |
domains: string[]
|
| 68 |
}
|
| 69 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 70 |
function collectLeafEntries(
|
| 71 |
fam: HierarchyFamily,
|
| 72 |
benchmarkCards?: Record<string, BenchmarkCard>,
|
| 73 |
): LeafEntry[] {
|
| 74 |
const out: LeafEntry[] = []
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 75 |
for (const leaf of fam.leaves ?? []) {
|
| 76 |
-
// Backends differ in whether leaves carry an explicit
|
| 77 |
-
// `eval_summary_ids` array. When absent, fall back to the
|
| 78 |
-
// pipeline's standard `${fam.key}_${leaf.key}` naming, then to the
|
| 79 |
-
// bare leaf key — both are stable enough for the detail page to
|
| 80 |
-
// resolve. This stops the inline benchmarks grid from disappearing
|
| 81 |
-
// on a backend that ships hierarchy.json without leaf eval ids.
|
| 82 |
const explicit = leaf.eval_summary_ids ?? []
|
| 83 |
const ids =
|
| 84 |
explicit.length > 0
|
|
@@ -87,10 +129,6 @@ function collectLeafEntries(
|
|
| 87 |
? [`${fam.key}_${leaf.key}`, leaf.key]
|
| 88 |
: []
|
| 89 |
if (ids.length === 0) continue
|
| 90 |
-
// Domain sources, in order of trust:
|
| 91 |
-
// (1) hierarchy `leaf.tags.domains` — sometimes absent
|
| 92 |
-
// (2) benchmark-metadata keyed by leaf.key
|
| 93 |
-
// (3) benchmark-metadata keyed by the leaf's eval_summary_id
|
| 94 |
const collected = new Set<string>()
|
| 95 |
for (const d of leaf.tags?.domains ?? []) collected.add(d.toLowerCase())
|
| 96 |
const cardByLeaf = benchmarkCards?.[leaf.key]
|
|
@@ -107,6 +145,7 @@ function collectLeafEntries(
|
|
| 107 |
domains: Array.from(collected),
|
| 108 |
})
|
| 109 |
}
|
|
|
|
| 110 |
return out
|
| 111 |
}
|
| 112 |
|
|
|
|
| 4 |
import { useRouter } from "next/navigation"
|
| 5 |
import { ArrowUpRight, ChevronDown, ChevronRight } from "lucide-react"
|
| 6 |
|
| 7 |
+
import type { HierarchyBenchmark, HierarchyFamily, HierarchyLeaf } from "@/lib/backend-artifacts"
|
| 8 |
import type { BenchmarkCard, CategoryType } from "@/lib/benchmark-schema"
|
| 9 |
import type { BenchmarkEvalListItem } from "@/lib/eval-processing"
|
| 10 |
|
|
|
|
| 67 |
domains: string[]
|
| 68 |
}
|
| 69 |
|
| 70 |
+
/**
|
| 71 |
+
* Build the per-row list of expandable benchmarks under a family.
|
| 72 |
+
*
|
| 73 |
+
* Primary path: the v2 production shape, where the hierarchy nests
|
| 74 |
+
* benchmarks under `families[].composites[].benchmarks[]` (plus
|
| 75 |
+
* `standalone_benchmarks[]` and the family-level `benchmarks[]`). Each
|
| 76 |
+
* benchmark carries its own `summary_eval_ids` for navigation.
|
| 77 |
+
*
|
| 78 |
+
* Fallback path: the legacy `families[].leaves[]` shape used by older
|
| 79 |
+
* snapshots. Kept for compatibility while older caches are still in
|
| 80 |
+
* circulation.
|
| 81 |
+
*/
|
| 82 |
function collectLeafEntries(
|
| 83 |
fam: HierarchyFamily,
|
| 84 |
benchmarkCards?: Record<string, BenchmarkCard>,
|
| 85 |
): LeafEntry[] {
|
| 86 |
const out: LeafEntry[] = []
|
| 87 |
+
|
| 88 |
+
// ── Primary: nested benchmarks (v2) ────────────────────────────────
|
| 89 |
+
const nested: HierarchyBenchmark[] = [
|
| 90 |
+
...(fam.standalone_benchmarks ?? []),
|
| 91 |
+
...(fam.benchmarks ?? []),
|
| 92 |
+
...(fam.composites ?? []).flatMap((c) => c.benchmarks ?? []),
|
| 93 |
+
]
|
| 94 |
+
for (const benchmark of nested) {
|
| 95 |
+
const summaryIds = benchmark.summary_eval_ids ?? []
|
| 96 |
+
const ids =
|
| 97 |
+
summaryIds.length > 0
|
| 98 |
+
? summaryIds
|
| 99 |
+
: benchmark.key
|
| 100 |
+
? [`${fam.key}_${benchmark.key}`, benchmark.key]
|
| 101 |
+
: []
|
| 102 |
+
if (ids.length === 0) continue
|
| 103 |
+
const collected = new Set<string>()
|
| 104 |
+
for (const d of benchmark.tags?.domains ?? []) collected.add(d.toLowerCase())
|
| 105 |
+
const cardByKey = benchmarkCards?.[benchmark.key]
|
| 106 |
+
for (const d of cardByKey?.benchmark_details?.domains ?? []) collected.add(d.toLowerCase())
|
| 107 |
+
for (const id of ids) {
|
| 108 |
+
const cardById = benchmarkCards?.[id]
|
| 109 |
+
for (const d of cardById?.benchmark_details?.domains ?? []) collected.add(d.toLowerCase())
|
| 110 |
+
}
|
| 111 |
+
out.push({
|
| 112 |
+
id: ids[0],
|
| 113 |
+
leafKey: benchmark.key,
|
| 114 |
+
leafName: benchmark.display_name || benchmark.key,
|
| 115 |
+
evalsCount: ids.length,
|
| 116 |
+
domains: Array.from(collected),
|
| 117 |
+
})
|
| 118 |
+
}
|
| 119 |
+
|
| 120 |
+
if (out.length > 0) return out
|
| 121 |
+
|
| 122 |
+
// ── Fallback: legacy `leaves` shape ────────────────────────────────
|
| 123 |
for (const leaf of fam.leaves ?? []) {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 124 |
const explicit = leaf.eval_summary_ids ?? []
|
| 125 |
const ids =
|
| 126 |
explicit.length > 0
|
|
|
|
| 129 |
? [`${fam.key}_${leaf.key}`, leaf.key]
|
| 130 |
: []
|
| 131 |
if (ids.length === 0) continue
|
|
|
|
|
|
|
|
|
|
|
|
|
| 132 |
const collected = new Set<string>()
|
| 133 |
for (const d of leaf.tags?.domains ?? []) collected.add(d.toLowerCase())
|
| 134 |
const cardByLeaf = benchmarkCards?.[leaf.key]
|
|
|
|
| 145 |
domains: Array.from(collected),
|
| 146 |
})
|
| 147 |
}
|
| 148 |
+
|
| 149 |
return out
|
| 150 |
}
|
| 151 |
|
|
@@ -428,7 +428,7 @@ export function ModelCompareDialog({
|
|
| 428 |
) : null}
|
| 429 |
{row.key === "latest" ? (
|
| 430 |
<div className="flex items-center gap-2">
|
| 431 |
-
<span>{model.latest_source_name || `${model.benchmarks_count} benchmark
|
| 432 |
{model.source_urls[0] ? (
|
| 433 |
<a
|
| 434 |
href={model.source_urls[0]}
|
|
|
|
| 428 |
) : null}
|
| 429 |
{row.key === "latest" ? (
|
| 430 |
<div className="flex items-center gap-2">
|
| 431 |
+
<span>{model.latest_source_name || `${model.benchmarks_count} benchmark composites summarized`}</span>
|
| 432 |
{model.source_urls[0] ? (
|
| 433 |
<a
|
| 434 |
href={model.source_urls[0]}
|
|
@@ -51,7 +51,7 @@ export function PolicyOverview({ summary }: PolicyOverviewProps) {
|
|
| 51 |
const card = summary.benchmark_card
|
| 52 |
|
| 53 |
// Defensive check: pipelines older than the "ancestor card leak" fix
|
| 54 |
-
// sometimes attach the parent
|
| 55 |
// helm_classic's card embedded under XSUM). Detect when the card's own
|
| 56 |
// name is clearly not this benchmark and ignore its narrative text — the
|
| 57 |
// synthesized fallback below produces something accurate instead.
|
|
@@ -81,17 +81,20 @@ export function PolicyOverview({ summary }: PolicyOverviewProps) {
|
|
| 81 |
const goal = cardMatchesEval ? card?.purpose_and_intended_users?.goal?.trim() || "" : ""
|
| 82 |
|
| 83 |
// Detect "parent" benchmark pages — either an aggregated composite or a
|
| 84 |
-
// multi-metric matrix where each column is a
|
| 85 |
// per-evaluation `metric_config.evaluation_description` belongs to whichever
|
| 86 |
// component was processed first (e.g. just the "airline" subset of Tau
|
| 87 |
// Bench 2) and would mislead a policy reader. Synthesize parent framing
|
| 88 |
-
// instead and surface the
|
|
|
|
|
|
|
|
|
|
| 89 |
const isAggregated = summary.is_aggregated === true
|
| 90 |
const aggregateNames = (summary.aggregate_sources ?? [])
|
| 91 |
.map((s) => s.composite_benchmark_name)
|
| 92 |
.filter((s): s is string => typeof s === "string" && s.length > 0)
|
| 93 |
|
| 94 |
-
const
|
| 95 |
const seen = new Set<string>()
|
| 96 |
const labels: string[] = []
|
| 97 |
const add = (raw: string | undefined | null) => {
|
|
@@ -103,8 +106,8 @@ export function PolicyOverview({ summary }: PolicyOverviewProps) {
|
|
| 103 |
seen.add(key)
|
| 104 |
labels.push(trimmed)
|
| 105 |
}
|
| 106 |
-
for (const
|
| 107 |
-
add(
|
| 108 |
}
|
| 109 |
for (const metric of summary.leaderboard_metrics ?? []) {
|
| 110 |
if (metric.scope === "subtask") {
|
|
@@ -116,14 +119,14 @@ export function PolicyOverview({ summary }: PolicyOverviewProps) {
|
|
| 116 |
}, [summary.subtasks, summary.leaderboard_metrics, aggregateNames])
|
| 117 |
|
| 118 |
const isMatrix = (summary.leaderboard_metrics?.length ?? 0) > 1
|
| 119 |
-
const isParentPage = isAggregated || (isMatrix &&
|
| 120 |
const useComponentDescription = !isParentPage
|
| 121 |
|
| 122 |
-
const parentFallback = isParentPage &&
|
| 123 |
-
? `${summary.evaluation_name} reports results across ${
|
| 124 |
-
isAggregated ? "component benchmarks" : "
|
| 125 |
}. Each is evaluated separately; the score shown is the ${
|
| 126 |
-
isAggregated ? "average" : "per-
|
| 127 |
}.`
|
| 128 |
: null
|
| 129 |
|
|
@@ -134,7 +137,7 @@ export function PolicyOverview({ summary }: PolicyOverviewProps) {
|
|
| 134 |
(useComponentDescription ? summary.metric_config.evaluation_description : summary.evaluation_name)
|
| 135 |
|
| 136 |
const [expanded, setExpanded] = useState(false)
|
| 137 |
-
const [
|
| 138 |
const isLong = summaryText.length > SUMMARY_PREVIEW_CHARS
|
| 139 |
const visibleText = expanded || !isLong
|
| 140 |
? summaryText
|
|
@@ -299,15 +302,15 @@ export function PolicyOverview({ summary }: PolicyOverviewProps) {
|
|
| 299 |
</dd>
|
| 300 |
</dl>
|
| 301 |
|
| 302 |
-
{isParentPage &&
|
| 303 |
<div
|
| 304 |
className="mt-4"
|
| 305 |
style={{ border: "1px solid var(--border-soft)", background: "var(--bg)" }}
|
| 306 |
>
|
| 307 |
<button
|
| 308 |
type="button"
|
| 309 |
-
onClick={() =>
|
| 310 |
-
aria-expanded={
|
| 311 |
className="flex w-full items-center justify-between gap-3 px-3.5 py-2.5 text-left transition-colors hover:bg-[color:var(--bg-warm)]"
|
| 312 |
>
|
| 313 |
<span
|
|
@@ -315,20 +318,20 @@ export function PolicyOverview({ summary }: PolicyOverviewProps) {
|
|
| 315 |
style={{ fontSize: 10, letterSpacing: "0.14em", color: "var(--fg-muted)" }}
|
| 316 |
>
|
| 317 |
<Layers className="h-3.5 w-3.5" />
|
| 318 |
-
{isAggregated ? `Component benchmarks (${
|
| 319 |
</span>
|
| 320 |
-
{
|
| 321 |
<ChevronUp className="h-4 w-4" style={{ color: "var(--fg-muted)" }} />
|
| 322 |
) : (
|
| 323 |
<ChevronDown className="h-4 w-4" style={{ color: "var(--fg-muted)" }} />
|
| 324 |
)}
|
| 325 |
</button>
|
| 326 |
-
{
|
| 327 |
<ul
|
| 328 |
className="grid list-disc gap-x-6 gap-y-1 px-3.5 pb-3.5 pl-9 text-[13px] sm:grid-cols-2 lg:grid-cols-3"
|
| 329 |
style={{ color: "var(--fg)" }}
|
| 330 |
>
|
| 331 |
-
{
|
| 332 |
<li key={name} className="capitalize">
|
| 333 |
{name}
|
| 334 |
</li>
|
|
|
|
| 51 |
const card = summary.benchmark_card
|
| 52 |
|
| 53 |
// Defensive check: pipelines older than the "ancestor card leak" fix
|
| 54 |
+
// sometimes attach the parent composite's card to a leaf benchmark (e.g.
|
| 55 |
// helm_classic's card embedded under XSUM). Detect when the card's own
|
| 56 |
// name is clearly not this benchmark and ignore its narrative text — the
|
| 57 |
// synthesized fallback below produces something accurate instead.
|
|
|
|
| 81 |
const goal = cardMatchesEval ? card?.purpose_and_intended_users?.goal?.trim() || "" : ""
|
| 82 |
|
| 83 |
// Detect "parent" benchmark pages — either an aggregated composite or a
|
| 84 |
+
// multi-metric matrix where each column is a slice. In both cases the
|
| 85 |
// per-evaluation `metric_config.evaluation_description` belongs to whichever
|
| 86 |
// component was processed first (e.g. just the "airline" subset of Tau
|
| 87 |
// Bench 2) and would mislead a policy reader. Synthesize parent framing
|
| 88 |
+
// instead and surface the slices separately.
|
| 89 |
+
// Note: at the data layer the backend still ships these as `subtasks` /
|
| 90 |
+
// `metric.scope === "subtask"`; we read those fields directly but label
|
| 91 |
+
// them "slice" in the UI.
|
| 92 |
const isAggregated = summary.is_aggregated === true
|
| 93 |
const aggregateNames = (summary.aggregate_sources ?? [])
|
| 94 |
.map((s) => s.composite_benchmark_name)
|
| 95 |
.filter((s): s is string => typeof s === "string" && s.length > 0)
|
| 96 |
|
| 97 |
+
const sliceLabels = useMemo(() => {
|
| 98 |
const seen = new Set<string>()
|
| 99 |
const labels: string[] = []
|
| 100 |
const add = (raw: string | undefined | null) => {
|
|
|
|
| 106 |
seen.add(key)
|
| 107 |
labels.push(trimmed)
|
| 108 |
}
|
| 109 |
+
for (const slice of summary.subtasks ?? []) {
|
| 110 |
+
add(slice.display_name || slice.subtask_name)
|
| 111 |
}
|
| 112 |
for (const metric of summary.leaderboard_metrics ?? []) {
|
| 113 |
if (metric.scope === "subtask") {
|
|
|
|
| 119 |
}, [summary.subtasks, summary.leaderboard_metrics, aggregateNames])
|
| 120 |
|
| 121 |
const isMatrix = (summary.leaderboard_metrics?.length ?? 0) > 1
|
| 122 |
+
const isParentPage = isAggregated || (isMatrix && sliceLabels.length > 1)
|
| 123 |
const useComponentDescription = !isParentPage
|
| 124 |
|
| 125 |
+
const parentFallback = isParentPage && sliceLabels.length > 1
|
| 126 |
+
? `${summary.evaluation_name} reports results across ${sliceLabels.length} ${
|
| 127 |
+
isAggregated ? "component benchmarks" : "slices"
|
| 128 |
}. Each is evaluated separately; the score shown is the ${
|
| 129 |
+
isAggregated ? "average" : "per-slice result"
|
| 130 |
}.`
|
| 131 |
: null
|
| 132 |
|
|
|
|
| 137 |
(useComponentDescription ? summary.metric_config.evaluation_description : summary.evaluation_name)
|
| 138 |
|
| 139 |
const [expanded, setExpanded] = useState(false)
|
| 140 |
+
const [slicesOpen, setSlicesOpen] = useState(false)
|
| 141 |
const isLong = summaryText.length > SUMMARY_PREVIEW_CHARS
|
| 142 |
const visibleText = expanded || !isLong
|
| 143 |
? summaryText
|
|
|
|
| 302 |
</dd>
|
| 303 |
</dl>
|
| 304 |
|
| 305 |
+
{isParentPage && sliceLabels.length > 1 && (
|
| 306 |
<div
|
| 307 |
className="mt-4"
|
| 308 |
style={{ border: "1px solid var(--border-soft)", background: "var(--bg)" }}
|
| 309 |
>
|
| 310 |
<button
|
| 311 |
type="button"
|
| 312 |
+
onClick={() => setSlicesOpen((v) => !v)}
|
| 313 |
+
aria-expanded={slicesOpen}
|
| 314 |
className="flex w-full items-center justify-between gap-3 px-3.5 py-2.5 text-left transition-colors hover:bg-[color:var(--bg-warm)]"
|
| 315 |
>
|
| 316 |
<span
|
|
|
|
| 318 |
style={{ fontSize: 10, letterSpacing: "0.14em", color: "var(--fg-muted)" }}
|
| 319 |
>
|
| 320 |
<Layers className="h-3.5 w-3.5" />
|
| 321 |
+
{isAggregated ? `Component benchmarks (${sliceLabels.length})` : `Slices (${sliceLabels.length})`}
|
| 322 |
</span>
|
| 323 |
+
{slicesOpen ? (
|
| 324 |
<ChevronUp className="h-4 w-4" style={{ color: "var(--fg-muted)" }} />
|
| 325 |
) : (
|
| 326 |
<ChevronDown className="h-4 w-4" style={{ color: "var(--fg-muted)" }} />
|
| 327 |
)}
|
| 328 |
</button>
|
| 329 |
+
{slicesOpen && (
|
| 330 |
<ul
|
| 331 |
className="grid list-disc gap-x-6 gap-y-1 px-3.5 pb-3.5 pl-9 text-[13px] sm:grid-cols-2 lg:grid-cols-3"
|
| 332 |
style={{ color: "var(--fg)" }}
|
| 333 |
>
|
| 334 |
+
{sliceLabels.map((name) => (
|
| 335 |
<li key={name} className="capitalize">
|
| 336 |
{name}
|
| 337 |
</li>
|