Claim 1: SWE-Bench Pro comprises 1,865 problems sourced from 41 actively maintained software repositories (abstract only).
Claim 1 — Scale of the benchmark
Claim (verbatim): "SWE-Bench Pro comprises 1,865 problems sourced from 41 actively maintained software repositories."
Verdict: Partially verified — the public subset (731 problems / 11 repos) is directly countable; the full 1,865 / 41 figure is stated by the paper and corroborated by the official release partner Scale AI.
What we verified
We loaded the official public benchmark release ScaleAI/SWE-bench_Pro (the canonical HF copy released by Scale AI, the paper's benchmark partner) and counted its contents:
- 731 problems across 11 repositories in the publicly accessible split.
- Per-repository instance counts (all 11 public repos):
| Repository | Instances |
|---|---|
| ansible/ansible | 96 |
| internetarchive/openlibrary | 91 |
| flipt-io/flipt | 85 |
| qutebrowser/qutebrowser | 79 |
| gravitational/teleport | 76 |
| future-architect/vuls | 62 |
| protonmail/webclients | 65 |
| navidrome/navidrome | 57 |
| element-hq/element-web | 56 |
| NodeBB/NodeBB | 44 |
| tutao/tutanota | 20 |
Relationship to the headline 1,865 / 41
The paper states the full benchmark is 1,865 problems from 41 repos, partitioned into a public set of 11 repos, a held-out set of 12 repos, and a commercial set of 18 proprietary repos (11 + 12 + 18 = 41). The 11 public repos we counted exactly match the "public set of 11" in Claim 2, and 731 is the released count of that public set. The held-out (12) and commercial (18) splits are explicitly not publicly accessible by design, so their 1,134 instances (1,865 − 731) cannot be independently counted — this is consistent with the paper's contamination-resistance design (Claim 5) rather than a discrepancy.
Evidence
- HF Job (analysis run): https://huggingface.co/jobs/Yashp2003/6a5c8046d216bd6f3a2009d5
- Results artifact:
claim_analysis_results.jsonin https://huggingface.co/buckets/Yashp2003/swebenchpro-repro-artifacts - Source dataset: https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro
- Official eval repo: https://github.com/scaleapi/SWE-bench_Pro-os