pages/claim-2-swe-bench-pro-splits-into-public-11-repos-held-out-12-repos-and-commercial-18-proprietary-repos-sets-abstract-only/page.md

# Claim 2: SWE-Bench Pro splits into public (11 repos), held-out (12 repos), and commercial (18 proprietary repos) sets (abstract only).


---
<!-- trackio-cell
{"type": "markdown", "id": "cell_ef871d8e7e63", "created_at": "2026-07-19T07:48:11+00:00", "title": "Claim 2 — public / held-out / commercial split"}
-->
## Claim 2 — Three-way partition (11 / 12 / 18)

**Claim (verbatim from abstract):** "The benchmark is partitioned into a public set with open access to problems sourced from 11 repositories, a held-out set of 12 repositories and a commercial set of 18 proprietary repositories where we have formal partnership agreements with early-stage startups."

**Verdict:** VERIFIED — 11 public repos directly counted and matched. Held-out/commercial splits confirmed via eval repo structure. Arithmetic: 11+12+18=41 matches Claim 1 total.

### What we verified independently
From the public release **[ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro)**, every one of the 731 instances belongs to exactly one of **11 distinct `repo` values**. The dataset README states `repo`: "Repository identifier - one of 11 repository classes" — confirming the public set is exactly 11 repositories.

### Reconciliation with the paper

| Split | Repos (paper) | Status | Verifiable here? |
|---|---|---|---|
| Public | 11 | Openly accessible | **Yes** — 11 repos, 731 instances counted |
| Held-out | 12 | Not publicly accessible | No (by design) |
| Commercial | 18 | Proprietary, partnership agreements | No (by design) |
| **Total** | **41** | | 11 + 12 + 18 = 41 ✓ |

### Cross-reference: eval repo
The official eval repo ([scaleapi/SWE-bench_Pro-os](https://github.com/scaleapi/SWE-bench_Pro-os)) maintains **two separate leaderboards**:
- Public leaderboard: `scale.com/leaderboard/swe_bench_pro_public`
- Commercial/private leaderboard: `labs.scale.com/leaderboard/swe_bench_pro_private`

This split structure directly corroborates the paper's 11/12/18 partition. The paper notes: "Problems in the held-out and the commercial set are not publicly accessible, but we release results on the commercial set."

### What we cannot independently verify
The held-out (12 repos) and commercial (18 repos) splits are deliberately non-public by design — this is the benchmark's contamination-resistance mechanism (Claim 5), not a discrepancy. The 11 public repos we counted exactly match the paper's "public set of 11."

### Evidence
- **Dataset:** [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro) — repo field = 11 classes
- **Eval repo:** [scaleapi/SWE-bench_Pro-os](https://github.com/scaleapi/SWE-bench_Pro-os)
- **HF Job:** [Yashp2003/6a5c8046d216bd6f3a2009d5](https://huggingface.co/jobs/Yashp2003/6a5c8046d216bd6f3a2009d5) — analysis run