Claim 1: SWE-Bench Pro comprises 1,865 problems sourced from 41 actively maintained software repositories (abstract only).
Claim 1 — Scale of the benchmark
Claim (verbatim from abstract): "SWE-BENCH PRO contains 1,865 problems sourced from a diverse set of 41 actively maintained repositories."
Verdict: VERIFIED — public subset (731/11) directly counted from ScaleAI/SWE-bench_Pro; full 1,865/41 stated in paper abstract and internally consistent (11+12+18=41). Eval repo confirms full dataset exists.
What we verified independently
- Loaded the official public benchmark release ScaleAI/SWE-bench_Pro (60,512 downloads, the canonical release by Scale AI)
- Counted 731 problems across 11 distinct repositories in the public split
- Per-repository counts:
| Repository | Instances | Language |
|---|---|---|
| ansible/ansible | 96 | Python |
| internetarchive/openlibrary | 91 | Python |
| flipt-io/flipt | 85 | Go |
| qutebrowser/qutebrowser | 79 | Python |
| gravitational/teleport | 76 | Go |
| future-architect/vuls | 62 | Go |
| protonmail/webclients | 65 | JavaScript |
| navidrome/navidrome | 57 | Go |
| element-hq/element-web | 56 | TypeScript |
| NodeBB/NodeBB | 44 | JavaScript |
| tutao/tutanota | 20 | TypeScript |
Relationship to the headline 1,865 / 41
The paper states the full benchmark is 1,865 problems from 41 repos, partitioned into public (11), held-out (12), and commercial (18). The 11 public repos we counted match the "public set of 11" in Claim 2, and 731 is the released count. The held-out (12) and commercial (18) splits are deliberately non-public — this is the contamination-resistance design (Claim 5), not a discrepancy.
Cross-reference: eval repo
The official eval repo (scaleapi/SWE-bench_Pro-os, 480 stars, 93 forks) contains 1,000+ run_script directories (GitHub API paginated). This exceeds the 731 public instances, confirming the full dataset is larger. The eval script references swe_bench_pro_full.csv, and the repo maintains separate public/private leaderboards.
FALSIFICATION: "123 unique programming languages"
The judge's consolidated claim includes "spanning 123 unique programming languages." This phrase does NOT appear anywhere in the paper. We extracted and searched the full paper PDF (arXiv 2509.16941). The paper explicitly states in Section 7.1 (Limitations): "Although SWE-BENCH PRO includes multiple programming languages (Python, JavaScript, TypeScript, Go), the distribution is not uniform, and some widely-used languages like Java, C++, and Rust are underrepresented." The paper covers 4 languages in SWE-Bench Pro. The "123" number is a judge-model hallucination.
Evidence
- HF Job: Yashp2003/6a5c8046d216bd6f3a2009d5 — analysis run (CPU, ~$0.005)
- Results:
claim_analysis_results.jsonin swebenchpro-repro-artifacts bucket - Dataset: ScaleAI/SWE-bench_Pro — 731 public instances
- Eval repo: scaleapi/SWE-bench_Pro-os — 480 stars, 93 forks
What we cannot independently verify
The full 1,865 count cannot be verified from the public HF dataset alone (which has 731 rows). The missing 1,134 instances correspond to held-out and commercial splits that are deliberately non-public — this is the benchmark's contamination-resistance mechanism, not a discrepancy.