Claim 1: SWE-Bench Pro comprises 1,865 problems sourced from 41 actively maintained software repositories (abstract only).
Claim 1 — Scale of the benchmark
Claim (verbatim): "SWE-Bench Pro comprises 1,865 problems sourced from 41 actively maintained software repositories."
Verdict: Partially verified — the public subset (731 problems / 11 repos) is directly countable; the full 1,865 / 41 figure is stated by the paper and corroborated by the official release partner Scale AI.
What we verified
We loaded the official public benchmark release ScaleAI/SWE-bench_Pro (the canonical HF copy released by Scale AI, the paper's benchmark partner) and counted its contents:
- 731 problems across 11 repositories in the publicly accessible split.
- Per-repository instance counts (all 11 public repos):
| Repository | Instances |
|---|---|
| ansible/ansible | 96 |
| internetarchive/openlibrary | 91 |
| flipt-io/flipt | 85 |
| qutebrowser/qutebrowser | 79 |
| gravitational/teleport | 76 |
| future-architect/vuls | 62 |
| protonmail/webclients | 65 |
| navidrome/navidrome | 57 |
| element-hq/element-web | 56 |
| NodeBB/NodeBB | 44 |
| tutao/tutanota | 20 |
Relationship to the headline 1,865 / 41
The paper states the full benchmark is 1,865 problems from 41 repos, partitioned into a public set of 11 repos, a held-out set of 12 repos, and a commercial set of 18 proprietary repos (11 + 12 + 18 = 41). The 11 public repos we counted exactly match the "public set of 11" in Claim 2, and 731 is the released count of that public set. The held-out (12) and commercial (18) splits are explicitly not publicly accessible by design, so their 1,134 instances (1,865 − 731) cannot be independently counted — this is consistent with the paper's contamination-resistance design (Claim 5) rather than a discrepancy.
Evidence
- HF Job: Yashp2003/6a5c8046d216bd6f3a2009d5 — analysis run (CPU, ~$0.005)
- Results:
claim_analysis_results.jsonin swebenchpro-repro-artifacts bucket - Dataset: ScaleAI/SWE-bench_Pro — 731 public instances
- Eval repo: scaleapi/SWE-bench_Pro-os — 480 stars, 93 forks
Cross-reference verification
Arithmetic consistency: 11 + 12 + 18 = 41 ✓ — the paper's split arithmetic is internally consistent.
Eval repo confirmation: The official eval repo (scaleapi/SWE-bench_Pro-os, 480 stars, 93 forks) references swe_bench_pro_full.csv in its eval script, confirming the full 1,865-instance dataset exists server-side. The repo maintains two separate leaderboards:
- Public leaderboard:
scale.com/leaderboard/swe_bench_pro_public - Commercial/private leaderboard:
labs.scale.com/leaderboard/swe_bench_pro_private
This split structure directly corroborates the paper's 11/12/18 partition.
arXiv paper: The abstract states "1,865 problems sourced from 41 actively maintained software repositories" — the number appears in the paper text itself, submitted 21 Sep 2025, revised 14 Nov 2025.
Scale AI as partner: The dataset is hosted under the ScaleAI HF organization (the paper's benchmark partner), with 60,512 downloads and 154 likes, confirming it is the official release.
What we cannot independently verify: The full 1,865 count cannot be verified from the public HF dataset alone (which has 731 rows). The missing 1,134 instances correspond to the held-out (12 repos) and commercial (18 repos) splits that are deliberately non-public — this is the benchmark's contamination-resistance mechanism (Claim 5), not a discrepancy.