Overview table gets a row per variant, and its cost columns follow the row

The cross-subset overview picked each column's value independently, taking the
extremum per model. For the two cost columns that meant the minimum, which is
always the default variant, while the model cell was labelled with whichever
variant led on the selected metric. A row reading "RealMLP (tuned + ensembled)"
quoted the default's fit time: 10.06 against 2950.72, roughly 200x low, because
tuning fits 200 configurations where the default fits one. 20 of 39 model rows
were affected, by 6.8x to 902x.

Rows are now keyed by (model, variant), so default, tuned and tuned + ensembled
each get their own line and a row's cost is the cost of the run it scores. That
removes the mismatch by construction, and puts the price of tuning on the page.

"One per model" collapses each method to its best variant, decided by the
leading column. It sits in the overview's own row and feeds only that table,
and is named after the win-rate matrix's button so the two read as one control.

The upstream artifacts were correct throughout: the full leaderboard table
(72,992 time values) and pareto_front_points.csv (67,232) both agree with
website_leaderboard.csv per (model, variant), and api.py works per record.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

main.py

@@ -45,6 +45,11 @@ def _choice_hints() -> dict[str, str]:
                 hints[label] = notes[key]
 
     hints["🤖 Models"] = "Always on: TabArena is a model benchmark first."
+    hints["One per model"] = (
+        "Keep only each model's best-performing variant, one row per model — as the win-rate "
+        "matrix does. With it off, every variant gets its own row, so you can read what "
+        "tuning bought."
+    )
     hints["Include imputed models"] = PROTOCOL_NOTES["imputed"]
     hints["TabArena-Lite (single split)"] = PROTOCOL_NOTES["lite"]
     # A category with no entrant yet says so, matching the disabled state the page renders.
@@ -332,7 +337,8 @@ CSS = """
    the chips themselves. `main.taStampTitles` moves each one into a native `title`, so it shows
    on hover instead, and the ? badge is what says the text is there at all. */
 .ta-controls span[data-testid="block-info"] { display: none !important; }
-.ta-controlrow .ta-catchip label::after {
+.ta-controlrow .ta-catchip label::after,
+.metric-select .ta-catchip label::after {
     content: "?";
     flex: 0 0 auto;
     width: 14px;
@@ -346,7 +352,8 @@ CSS = """
     line-height: 12px;
     text-align: center;
 }
-.ta-controlrow .ta-catchip label:hover::after { opacity: 0.85; }
+.ta-controlrow .ta-catchip label:hover::after,
+.metric-select .ta-catchip label:hover::after { opacity: 0.85; }
 /* Chip-bar buttons carry a tooltip too; a dotted underline hints at it without a badge on
    every chip, which at five per row would be louder than the chips. */
 .ta-controls .tab-buttons button[title] { text-decoration: underline dotted #ffffff40 1px; text-underline-offset: 3px; }
@@ -1818,6 +1825,8 @@ function taStampTitles() {
     };
     document.querySelectorAll(".ta-controls .ta-catchip label").forEach(stamp);
     document.querySelectorAll(".ta-controls .tab-buttons button").forEach(stamp);
+    // The overview's own toggle sits outside the control card but explains itself the same way.
+    document.querySelectorAll(".metric-select .ta-catchip label").forEach(stamp);
 
     // The row caption is a CSS ::before, which cannot hold a title, so it goes on the element
     // the caption is drawn on. A chip inside has its own title, and the nearest one wins, so

pages.py

@@ -359,9 +359,10 @@ def render_internal_page(page: LeaderboardPage) -> None:
     gr.HTML(
         '<div class="ta-section-head" id="ta-overview-section">'
         "<h2>🔭 Performance across leaderboards</h2>"
-        '<p class="ta-section-sub">Best result per method across every task and dataset-size subset. '
+        '<p class="ta-section-sub">Every method variant across every task and dataset-size subset. '
         "The quickest way to spot the ones that are strong everywhere against those that shine on "
-        "one subset.</p>"
+        "one subset, and to see what tuning buys. <i>One per model</i> collapses each method to its "
+        "best variant.</p>"
         "</div>"
     )
     with gr.Row(elem_classes="metric-select"):
@@ -375,16 +376,43 @@ def render_internal_page(page: LeaderboardPage) -> None:
             min_width=200,
         )
         metric_tldr = gr.Markdown(OVERVIEW_METRIC_TLDR["Elo"], elem_classes="metric-tldr")
+        # Lives here rather than in the control card because it only decides which rows this one
+        # table has. Same collapse the win-rate matrix offers and named the same, so the two read
+        # as one control. Off by default: how much tuning buys a model is one of the things this
+        # table is for.
+        overview_one_per_model = gr.Checkbox(
+            value=False,
+            label="One per model",
+            container=False,
+            scale=0,
+            min_width=180,
+            elem_classes=["ta-catchip"],
+        )
     overview_metric.change(
         lambda m: OVERVIEW_METRIC_TLDR.get(m, ""), overview_metric, metric_tldr, api_visibility="private"
     )
 
     # The overview follows the selectors too: the column each one points at leads its group and
     # is tinted, so the reader's own view is where their eye already is.
-    @gr.render(inputs=[overview_metric, entrants_state, tasks_state, datasets_state, care_state])
-    def _render_overview(metric, entrants, tasks, datasets, care):
+    @gr.render(
+        inputs=[
+            overview_metric,
+            entrants_state,
+            tasks_state,
+            datasets_state,
+            care_state,
+            overview_one_per_model,
+        ]
+    )
+    def _render_overview(metric, entrants, tasks, datasets, care, one_per_model):
         make_cross_subset_overview(
-            page.data_root, metric, entrants=entrants, tasks=tasks, datasets=datasets, care=care
+            page.data_root,
+            metric,
+            entrants=entrants,
+            tasks=tasks,
+            datasets=datasets,
+            care=care,
+            one_per_model=one_per_model,
         )
 
     # The "I care about" metric drives the overview's own selector, so the two cannot disagree

views.py

@@ -997,8 +997,15 @@ _GROUP_TITLE = {"cost": "What it costs", "task": "By Task", "size": "By Dataset
 # Size definitions are reused from DATASET_SIZE_NOTE so they can't drift from the subset blurbs.
 _COLUMN_TOOLTIPS: dict[str, str] = {
     "Overall": "All tasks across every dataset size. This is the headline ranking.",
-    "Fit (s/1K)": "Median seconds to train per 1000 rows, across all datasets. Lower is better.",
-    "Infer (s/1K)": "Median seconds to predict per 1000 rows, across all datasets. Lower is better.",
+    "Fit (s/1K)": (
+        "Median seconds to train per 1000 rows, across all datasets, for the variant shown. "
+        "Tuning fits 200 configurations, so a tuned variant costs far more than its default. "
+        "Lower is better."
+    ),
+    "Infer (s/1K)": (
+        "Median seconds to predict per 1000 rows, across all datasets, for the variant shown. "
+        "Lower is better."
+    ),
     "Class.": "Classification tasks only (binary + multiclass).",
     "Regr.": "Regression tasks only.",
     "Binary": "Binary classification tasks only.",
@@ -1046,26 +1053,30 @@ def _selected_overview_columns(tasks: str, datasets: str, care: str) -> list[str
     return [c for c in wanted if c]
 
 
-def _subset_best(df: pd.DataFrame, column: str, higher_is_better: bool) -> pd.DataFrame:
-    """Best variant per model in a subset for `column`.
+_SUBSET_FIELDS = ["base", "TypeName", "Type", "variant", "url", "verified", "imputed"]
+
+
+def _subset_rows(df: pd.DataFrame, column: str) -> pd.DataFrame:
+    """A subset's rows, `Model` split into base/variant/url and `column` renamed to `val`.
 
     Systems stay in. Which entrants compete is decided by the pool the reader selected, and
     the overview reports the numbers computed for that pool, so dropping a competitor here
     would show a field the numbers were not computed against.
     """
-    df = df.copy()
-    df = df.dropna(subset=[column])
+    df = df.dropna(subset=[column]).copy()
     parsed = df["Model"].map(parse_model)
     df["base"] = [p[0] for p in parsed]
     df["variant"] = [p[1] for p in parsed]
     df["url"] = [p[2] for p in parsed]
     df["imputed"] = df["Imputed"].astype(bool) if "Imputed" in df.columns else False
     df["verified"] = df["Verified"] if "Verified" in df.columns else ""
-    grouped = df.groupby("base")[column]
-    best = df.loc[grouped.idxmax() if higher_is_better else grouped.idxmin()]
-    return best[
-        ["base", "TypeName", "Type", "variant", "url", column, "verified", "imputed"]
-    ].rename(columns={column: "val"})
+    return df.rename(columns={column: "val"})[[*_SUBSET_FIELDS, "val"]]
+
+
+def _subset_best(rows: pd.DataFrame, higher_is_better: bool) -> pd.DataFrame:
+    """Best-performing variant per model, one row each."""
+    grouped = rows.groupby("base")["val"]
+    return rows.loc[grouped.idxmax() if higher_is_better else grouped.idxmin()]
 
 
 def _overview_th(label: str, *, rowspan: int | None = None, selected: bool = False) -> str:
@@ -1227,8 +1238,14 @@ def make_cross_subset_overview(
     tasks: str = "all",
     datasets: str = "all",
     care: str = "quality",
+    one_per_model: bool = False,
 ) -> gr.HTML:
-    """Heatmap of the best `metric` per method (rows) and subset (columns), within one pool.
+    """Heatmap of `metric` per entrant (rows) and subset (columns), within one pool.
+
+    A row is one model *variant* — default, tuned, or tuned + ensembled — so its quality and its
+    cost are always the same run's. With `one_per_model` only each model's best-performing
+    variant is kept, decided by the leading column, as the win-rate matrix's button of the same
+    name does.
 
     `entrants` selects the pool. Every column is read from that pool's artifacts, so the whole
     grid is one consistent field of competitors.
@@ -1243,29 +1260,50 @@ def make_cross_subset_overview(
         metric = "Elo"
     column, higher_is_better, fmt = _OVERVIEW_METRIC_SPECS[metric]
 
-    val_by_col: dict[str, dict[str, float]] = {}
-    imp_by_col: dict[str, dict[str, bool]] = {}
-    meta: dict[str, dict] = {}
+    val_by_col: dict[str, dict[tuple[str, str], float]] = {}
+    imp_by_col: dict[str, dict[tuple[str, str], bool]] = {}
+    meta: dict[tuple[str, str], dict] = {}
     present: list[tuple[str, str]] = []  # (label, group)
 
+    # Read every column first. A cost column reports its own metric, the rest the selected one.
+    loaded: list[tuple[str, str, pd.DataFrame]] = []
     for label, group, subset, value_column in _OVERVIEW_COLUMNS:
         subset = replace(subset, entrants=entrants)
         path = Path(data_root) / subset.rel_path / "website_leaderboard.csv"
         if not path.exists():
             continue
-        # A cost column reports its own metric; a subset column reports the selected one.
-        col, col_higher_better = (column, higher_is_better)
-        if value_column is not None:
-            col, col_higher_better, _ = _FIXED_COLUMN_SPECS[label]
-            if col not in load_leaderboard_csv(str(path.resolve())).columns:
-                continue
-        best = _subset_best(load_leaderboard_csv(str(path.resolve())), col, col_higher_better)
-        val_by_col[label] = dict(zip(best["base"], best["val"]))
-        imp_by_col[label] = dict(zip(best["base"], best["imputed"]))
+        df = load_leaderboard_csv(str(path.resolve()))
+        col = _FIXED_COLUMN_SPECS[label][0] if value_column is not None else column
+        if col not in df.columns:
+            continue
+        rows = _subset_rows(df, col)
+        if not rows.empty:
+            loaded.append((label, group, rows))
+
+    if not loaded:
+        return gr.HTML("<p>No overview data available.</p>")
+
+    # The leading column decides which variant stands for each model, so the kept row is the one
+    # the reader is ranking by.
+    keep = None
+    if one_per_model:
+        lead = next((r for lbl, _, r in loaded if lbl not in _FIXED_COLUMN_SPECS), loaded[0][2])
+        best = _subset_best(lead, higher_is_better)
+        keep = set(zip(best["base"], best["variant"]))
+
+    for label, group, rows in loaded:
+        keys = list(zip(rows["base"], rows["variant"]))
+        if keep is not None:
+            rows = rows[[k in keep for k in keys]]
+            keys = [k for k in keys if k in keep]
+        if rows.empty:
+            continue
+        val_by_col[label] = dict(zip(keys, rows["val"]))
+        imp_by_col[label] = dict(zip(keys, rows["imputed"]))
         present.append((label, group))
-        for _, row in best.iterrows():
+        for key, (_, row) in zip(keys, rows.iterrows()):
             meta.setdefault(
-                row["base"],
+                key,
                 {
                     "type_name": row["TypeName"],
                     "emoji": row["Type"],
@@ -1293,8 +1331,8 @@ def make_cross_subset_overview(
 
     sort_label = present[0][0]
     worst_sort = float("-inf") if higher_is_better else float("inf")
-    bases = sorted(
-        meta, key=lambda b: val_by_col[sort_label].get(b, worst_sort), reverse=higher_is_better
+    entries = sorted(
+        meta, key=lambda k: val_by_col[sort_label].get(k, worst_sort), reverse=higher_is_better
     )
 
     rank_by_col, bounds = {}, {}
@@ -1322,10 +1360,10 @@ def make_cross_subset_overview(
 
     # -- Body
     body = []
-    for base in bases:
-        m = meta[base]
+    for key in entries:
+        m = meta[key]
         color = model_type_color.get(m["type_name"], "#9e9e9e")
-        name = html.escape(base)
+        name = html.escape(key[0])
         if m["variant"]:
             name += f' <span class="ta-variant">({html.escape(m["variant"])})</span>'
         if m.get("verified") == "✔️":
@@ -1343,7 +1381,7 @@ def make_cross_subset_overview(
             f'<td class="ta-model-cell">{name_html}</td>',
         ]
         for label, _ in present:
-            val = val_by_col[label].get(base)
+            val = val_by_col[label].get(key)
             if val is None:
                 cells.append('<td class="ta-na">–</td>')
                 continue
@@ -1357,10 +1395,10 @@ def make_cross_subset_overview(
             else:
                 frac_best = (val - lo) / (hi - lo) if col_higher_better else (hi - val) / (hi - lo)
             bg = _interp_color(1 - frac_best)
-            medal = _MEDALS.get(rank_by_col[label].get(base), "")
+            medal = _MEDALS.get(rank_by_col[label].get(key), "")
             imp = (
                 '<sup class="ta-imp" title="Score is (partly) imputed">*</sup>'
-                if imp_by_col[label].get(base)
+                if imp_by_col[label].get(key)
                 else ""
             )
             sel = " ta-col-sel" if label in highlighted else ""
@@ -1372,10 +1410,15 @@ def make_cross_subset_overview(
         body.append(f"<tr>{''.join(cells)}</tr>")
 
     direction = "Higher is better" if higher_is_better else "Lower is better"
+    scope = (
+        "Each model shows its best-performing variant."
+        if one_per_model
+        else "One row per model variant, so a row's cost is the cost of the run it scores."
+    )
     caption = (
-        f'<div class="ta-cap">Best <b>{html.escape(metric)}</b> per model across subsets '
+        f'<div class="ta-cap"><b>{html.escape(metric)}</b> per entrant across subsets '
         f"(with imputation, all repeats). {direction}; 🥇🥈🥉 mark the top 3 in each column. "
-        "Each model shows its best-performing variant.</div>"
+        f"{scope}</div>"
         '<div class="ta-cap ta-cap-legend">✔️ = verified implementation &nbsp;·&nbsp; '
         '<span class="ta-imp">*</span> = (partly) imputed score &nbsp;·&nbsp; '
         "💡 Click any <u>underlined</u> model name (↗) to open its paper or code.</div>"

website_texts.py

@@ -643,10 +643,19 @@ YOUR_BENCHMARK_LINKS = [
 ]
 
 VERSION_HISTORY_BUTTON_TEXT = """
-**Current Version: TabArena-v0.1.8.1**
+**Current Version: TabArena-v0.1.8.2**
 
 The following details updates to the leaderboard (date format is YYYY/MM/DD):
 
+* 2026/08/17-v0.1.8.2:
+    * The *Performance across leaderboards* table now has a row per model variant, so default,
+      tuned and tuned + ensembled each get their own line and the cost of tuning is on the page.
+      Its new *One per model* toggle collapses each method to its best variant, as the win-rate
+      matrix's button of the same name does.
+    * Fixed the *Fit (s/1K)* and *Infer (s/1K)* columns of that table, which reported the cheapest
+      variant of each model rather than the variant on the row. A row labelled
+      *(tuned + ensembled)* showed its default's fit time, roughly 200x too low, because tuning
+      fits 200 configurations where the default fits one.
 * 2026/08/10-v0.1.8.1:
     * Updated verified model: ChimeraBoost, now at version 0.30.0.
 * 2026/08/06-v0.1.8: