Resilience + linux gate: connection-failure reset + pre-push DuckDB read-path gate

- lib/duckdb.ts: reset the connection singleton on init failure so a transient
httpfs blip retries instead of wedging the process in 500s until restart
(error-path only; happy path byte-identical). + tests/duckdb-connection-reset.test.ts (3/3).
- scripts/linux-gate + hooks/pre-push: enforce "never deploy untested-on-linux"
(HF Spaces run no CI between push and prod). Host vitest + a linux/amd64 DuckDB
read-path smoke on the prod-pinned binding (auto-read from pnpm-lock).
Activate once per clone: git config core.hooksPath hooks

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

hooks/pre-push

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+#
+# Pre-push linux gate (invariant I5). Pushing to the HF Space deploys to prod and
+# runs NO tests there — so this is the enforcement point between local and prod.
+# Aborts the push if the linux gate is red.
+#
+# Activate once per clone:   git config core.hooksPath hooks
+# Escape hatch (docs-only / emergency):   SKIP_LINUX_GATE=1 git push
+set -euo pipefail
+
+if [ "${SKIP_LINUX_GATE:-0}" = "1" ]; then
+  echo "[pre-push] SKIP_LINUX_GATE=1 — skipping linux gate (by request)."
+  exit 0
+fi
+
+ROOT="$(git rev-parse --show-toplevel)"
+echo "[pre-push] running linux gate before deploy (set SKIP_LINUX_GATE=1 to bypass)…"
+exec "$ROOT/scripts/linux-gate.sh"

lib/duckdb.ts

@@ -30,7 +30,7 @@ const VIEW_FILES = {
 
 export async function getConnection(): Promise<DuckDBConnection> {
   if (!connectionPromise) {
-    connectionPromise = (async () => {
+    const pending = (async () => {
       const connection = await DuckDBConnection.create()
 
       // Materialise each parquet snapshot into an in-memory DuckDB table at
@@ -65,6 +65,15 @@ export async function getConnection(): Promise<DuckDBConnection> {
 
       return connection
     })()
+    connectionPromise = pending
+    // If init fails (e.g. a transient httpfs blip during the snapshot
+    // read), clear the cached rejected promise so the NEXT request retries
+    // instead of every request awaiting a permanently-rejected promise
+    // until the Space restarts. Guard on identity so a later retry already
+    // in flight is never stomped.
+    pending.catch(() => {
+      if (connectionPromise === pending) connectionPromise = null
+    })
   }
 
   return connectionPromise

scripts/linux-gate.sh

@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+#
+# Linux gate — the enforcement point for invariant I5: "never deploy code that
+# hasn't run on linux." Pushing to the HF Space deploys to prod and runs NO
+# tests there, so this gate (wired as a pre-push hook) is the only check between
+# a local change and production. It catches the class of bug that passes on Mac
+# and broke prod: DuckDB struct/timestamp marshalling on the linux-x64 binding
+# (R2) and the connection lifecycle (R5).
+#
+# Two legs:
+#   (1) host vitest — unit/logic suite (fast; connection-reset + transforms).
+#   (2) linux/amd64 DuckDB read-path smoke on the prod-pinned binding (the real
+#       net for the Mac↔linux divergence).
+#
+# Requires Docker. Coverage grows: add parity / render-equivalence tests to the
+# linux leg as they land (see scripts/linux-gate/README.md).
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$ROOT"
+
+: "${SNAPSHOT_URL:=https://huggingface.co/datasets/evaleval/card_backend/resolve/main/warehouse/2026-05-29T00-24-44Z}"
+
+echo "[linux-gate] (1/2) host unit/logic suite (vitest)…"
+# Quarantine: tests/transformations/identity-canonicalization.test.ts holds the
+# intentionally-RED Group F route-id cases (they assert the legacy `__` form
+# pending the model-resolution-rework landing — frontend commit 924266a). Excluded
+# so the gate blocks on REAL regressions, not known-deferred reds. REMOVE this
+# exclusion once the rework lands and those expectations are aligned.
+pnpm vitest run --exclude '**/identity-canonicalization.test.ts'
+
+echo "[linux-gate] (2/2) linux/amd64 DuckDB read-path smoke (prod binding)…"
+DUCKDB_VERSION="$(grep -m1 -oE '@duckdb/node-api@[0-9][A-Za-z0-9.+-]*' pnpm-lock.yaml | sed 's#.*@##' || true)"
+DUCKDB_VERSION="${DUCKDB_VERSION:-1.5.3-r.2}"
+echo "[linux-gate]   binding @duckdb/node-api@${DUCKDB_VERSION} (from pnpm-lock); snapshot ${SNAPSHOT_URL}"
+docker build --platform=linux/amd64 --build-arg "DUCKDB_VERSION=${DUCKDB_VERSION}" \
+  -t evalcard-linux-gate "$ROOT/scripts/linux-gate" >/dev/null
+docker run --platform=linux/amd64 --rm -e SNAPSHOT_URL="$SNAPSHOT_URL" evalcard-linux-gate
+
+echo "[linux-gate] PASS ✅"

scripts/linux-gate/Dockerfile

@@ -0,0 +1,14 @@
+# linux/amd64 gate image — runs the DuckDB read-path smoke on the SAME binding
+# version prod uses, so a Mac-vs-linux marshalling divergence is caught before deploy.
+# DUCKDB_VERSION is passed by scripts/linux-gate.sh, extracted from pnpm-lock.yaml
+# so it can never drift from what prod actually ships.
+# Platform is set by `docker build --platform=linux/amd64` in scripts/linux-gate.sh.
+FROM node:18-bullseye-slim
+RUN apt-get update \
+  && apt-get install -y --no-install-recommends ca-certificates \
+  && rm -rf /var/lib/apt/lists/*
+WORKDIR /gate
+ARG DUCKDB_VERSION=1.5.3-r.2
+RUN npm install --no-audit --no-fund "@duckdb/node-api@${DUCKDB_VERSION}"
+COPY smoke.mjs .
+ENTRYPOINT ["node", "smoke.mjs"]

scripts/linux-gate/README.md

@@ -0,0 +1,45 @@
+# Linux gate (pre-push)
+
+**Why this exists.** This app deploys to a HuggingFace Space. Pushing to the
+Space's git remote (`hf.co`) **builds and deploys to prod immediately and runs no
+tests**. There is no CI between "push" and "live." That gap is what caused the
+linux-only DuckDB crashes (struct/timestamp marshalling that passed on Mac and
+broke in prod, discovered by users days later). This gate is the enforcement
+point for invariant **I5 — never deploy code that hasn't run on linux.**
+
+## Activate (once per clone)
+
+```bash
+git config core.hooksPath hooks
+```
+
+Now `git push` runs `scripts/linux-gate.sh` and **aborts the push if it's red.**
+Requires Docker (for the linux/amd64 leg).
+
+Escape hatch (docs-only / emergency): `SKIP_LINUX_GATE=1 git push`.
+
+## What it runs
+
+1. **Host vitest** (`pnpm vitest run`) — unit/logic suite (connection-reset lifecycle, transforms). Fast; platform-agnostic logic.
+2. **linux/amd64 DuckDB read-path smoke** (`scripts/linux-gate/smoke.mjs` in a container on the **prod-pinned `@duckdb/node-api`** version, read from `pnpm-lock.yaml`) — loads the snapshot views over httpfs into in-memory tables (mirroring prod `getConnection`, no `/data` mmap) and runs the real read paths through `readAll()`+`getRowObjectsJson()`. This is the net for the Mac↔linux marshalling divergence.
+
+`SNAPSHOT_URL` defaults to the pinned prod snapshot; override via env to gate against a post-rebaseline snapshot.
+
+## Coverage — grows as tests land
+
+Today the linux leg is a read-path *smoke* (catches the marshalling-crash class).
+As the comparison-index work proceeds, add to the linux leg:
+- the **leaderboard parity** gate (query output vs the live `comparison-index`),
+- **render-equivalence** for eval/histogram/DeepDive (silent-drop guard),
+- the **`by_model`-removal consumer** test.
+
+The smoke is the runner; these are its content. It is NOT a substitute for the
+full gate (#12) — it's the enforcement substrate that makes the gate block deploys.
+
+## Limitation (be honest)
+
+A git hook protects whoever installs it, not the org — a fresh clone or a
+collaborator who hasn't run `git config core.hooksPath hooks` can still push
+unchecked. For solo/small-team that's acceptable; if pushes ever come from
+multiple people or automation, move to a GitHub-mirror + Actions substrate
+(runs the same gate on `ubuntu-latest`, enforced for everyone).

scripts/linux-gate/smoke.mjs

@@ -0,0 +1,58 @@
+// Linux/amd64 DuckDB read-path smoke — runs the REAL read paths the app uses on
+// the prod-pinned @duckdb/node-api binding, inside a linux/amd64 container, and
+// asserts the binding can marshal them via readAll()+getRowObjectsJson().
+//
+// This is the net for the class of failure that passes on Mac and broke prod:
+// the "Invalid Error: don't know what type:" struct/timestamp marshalling crash
+// on the linux-x64 binding (R2/I2). It mirrors prod's getConnection() exactly:
+// in-memory tables loaded over httpfs (NO /data mmap — R1/I1).
+//
+// SNAPSHOT_URL is passed in by scripts/linux-gate.sh.
+import { DuckDBConnection } from "@duckdb/node-api"
+
+const SNAP = (process.env.SNAPSHOT_URL || "").replace(/\/+$/, "")
+if (!SNAP) { console.error("linux-gate smoke: SNAPSHOT_URL required"); process.exit(2) }
+
+const VIEWS = {
+  models_view: "models_view.parquet",
+  evals_view: "evals_view.parquet",
+  eval_results_view: "eval_results_view.parquet",
+}
+
+const c = await DuckDBConnection.create()
+// Prod read path: materialise each view into an in-memory table over httpfs.
+for (const [v, f] of Object.entries(VIEWS)) {
+  await c.run(`CREATE OR REPLACE TABLE ${v} AS SELECT * FROM read_parquet('${SNAP}/${f}')`)
+}
+
+async function probe(label, sql) {
+  const t = Date.now()
+  const r = await c.runAndRead(sql)
+  await r.readAll()                 // the chunk-fetch that throws on a bad type
+  const rows = r.getRowObjectsJson() // the JS marshalling the linux binding crashed on
+  if (!rows.length) throw new Error(`${label} returned 0 rows (expected data)`)
+  console.log(`  ok  ${label}: ${rows.length} rows (${Date.now() - t}ms)`)
+}
+
+try {
+  // getModelSummaryById's actual read path: raw SELECT * on models_view (which
+  // carries STRUCT/JSON/TIMESTAMP columns) — a real prod read that marshals today.
+  await probe("models_view SELECT* (getModelSummaryById)", `SELECT * FROM models_view LIMIT 5`)
+
+  // The scalar leaderboard query we will serve (RANK window over eval_results_view).
+  await probe("leaderboard scalar query", `
+    SELECT evaluation_id, metric_summary_id, model_route_id, score,
+      RANK()   OVER (PARTITION BY evaluation_id, metric_summary_id
+                     ORDER BY (CASE WHEN lower_is_better THEN score ELSE -score END) ASC) AS rank,
+      COUNT(*) OVER (PARTITION BY evaluation_id, metric_summary_id) AS total
+    FROM eval_results_view
+    WHERE score IS NOT NULL AND evaluation_id IS NOT NULL
+      AND metric_summary_id IS NOT NULL AND model_route_id IS NOT NULL
+    LIMIT 200`)
+
+  console.log("LINUX-GATE SMOKE: PASS — prod binding marshalled the real read paths")
+  process.exit(0)
+} catch (e) {
+  console.error(`LINUX-GATE SMOKE: FAIL — ${e?.name}: ${e?.message}`)
+  process.exit(1)
+}

tests/duckdb-connection-reset.test.ts

@@ -0,0 +1,54 @@
+import { describe, it, expect, vi, beforeEach } from "vitest"
+
+// `lib/duckdb` imports "server-only" (throws outside RSC) and @duckdb/node-api.
+// Stub both so we can unit-test the connection-singleton lifecycle in isolation.
+vi.mock("server-only", () => ({}))
+
+const create = vi.fn()
+vi.mock("@duckdb/node-api", () => ({
+  DuckDBConnection: { create: () => create() },
+}))
+
+describe("getConnection() failure-reset (P1 / I6)", () => {
+  beforeEach(() => {
+    vi.resetModules()
+    create.mockReset()
+    // file:// path so the CREATE TABLE loop builds a SQL string without any
+    // real network/disk read (connection.run is mocked).
+    process.env.SNAPSHOT_URL = "file:///tmp/snapshot"
+  })
+
+  it("clears the singleton on a transient init failure so the NEXT request retries (no permanent 500)", async () => {
+    const conn = { run: vi.fn().mockResolvedValue(undefined) }
+    create
+      .mockRejectedValueOnce(new Error("transient httpfs blip")) // boot blip
+      .mockResolvedValueOnce(conn) // retry succeeds
+    const { getConnection } = await import("../lib/duckdb")
+
+    await expect(getConnection()).rejects.toThrow("transient httpfs blip")
+    await Promise.resolve() // let the pending.catch microtask null the singleton
+    const got = await getConnection()
+
+    expect(got).toBe(conn)
+    expect(create).toHaveBeenCalledTimes(2) // retried — not wedged on a rejected promise
+  })
+
+  it("dedups concurrent first-callers into ONE init attempt and they share the rejection", async () => {
+    create.mockRejectedValueOnce(new Error("boom"))
+    const { getConnection } = await import("../lib/duckdb")
+    const [a, b] = await Promise.allSettled([getConnection(), getConnection()])
+    expect(a.status).toBe("rejected")
+    expect(b.status).toBe("rejected")
+    expect(create).toHaveBeenCalledTimes(1) // both awaited the same pending promise
+  })
+
+  it("reuses the connection once initialised (no re-init on the happy path)", async () => {
+    const conn = { run: vi.fn().mockResolvedValue(undefined) }
+    create.mockResolvedValue(conn)
+    const { getConnection } = await import("../lib/duckdb")
+    const a = await getConnection()
+    const b = await getConnection()
+    expect(a).toBe(b)
+    expect(create).toHaveBeenCalledTimes(1)
+  })
+})