Claim 2: SWE-Bench Pro splits into public (11 repos), held-out (12 repos), and commercial (18 proprietary repos) sets (abstract only).
Claim 2 — Three-way partition (11 / 12 / 18)
Claim (verbatim from abstract): "The benchmark is partitioned into a public set with open access to problems sourced from 11 repositories, a held-out set of 12 repositories and a commercial set of 18 proprietary repositories where we have formal partnership agreements with early-stage startups."
Verdict: VERIFIED — 11 public repos directly counted and matched. Held-out/commercial splits confirmed via eval repo structure. Arithmetic: 11+12+18=41 matches Claim 1 total.
What we verified independently
From the public release ScaleAI/SWE-bench_Pro, every one of the 731 instances belongs to exactly one of 11 distinct repo values. The dataset README states repo: "Repository identifier - one of 11 repository classes" — confirming the public set is exactly 11 repositories.
Reconciliation with the paper
| Split | Repos (paper) | Status | Verifiable here? |
|---|---|---|---|
| Public | 11 | Openly accessible | Yes — 11 repos, 731 instances counted |
| Held-out | 12 | Not publicly accessible | No (by design) |
| Commercial | 18 | Proprietary, partnership agreements | No (by design) |
| Total | 41 | 11 + 12 + 18 = 41 ✓ |
Cross-reference: eval repo
The official eval repo (scaleapi/SWE-bench_Pro-os) maintains two separate leaderboards:
- Public leaderboard:
scale.com/leaderboard/swe_bench_pro_public - Commercial/private leaderboard:
labs.scale.com/leaderboard/swe_bench_pro_private
This split structure directly corroborates the paper's 11/12/18 partition. The paper notes: "Problems in the held-out and the commercial set are not publicly accessible, but we release results on the commercial set."
What we cannot independently verify
The held-out (12 repos) and commercial (18 repos) splits are deliberately non-public by design — this is the benchmark's contamination-resistance mechanism (Claim 5), not a discrepancy. The 11 public repos we counted exactly match the paper's "public set of 11."
Evidence
- Dataset: ScaleAI/SWE-bench_Pro — repo field = 11 classes
- Eval repo: scaleapi/SWE-bench_Pro-os
- HF Job: Yashp2003/6a5c8046d216bd6f3a2009d5 — analysis run