Claim 1: SWE-Bench Pro comprises 1,865 problems from 41 actively maintained repositories.
Paper claim (Abstract, §3.3): SWE-Bench Pro contains 1,865 problems sourced from 41 actively maintained repositories spanning business applications, B2B services, and developer tools.
Verification approach:
- Load public split from Hugging Face dataset
ScaleAI/SWE-bench_Pro(test split) - Count instances and unique repositories
- Cross-reference with paper's stated splits: 731 public (11 repos) + 276 commercial (18 repos) + 858 held-out (12 repos) = 1,865 total from 41 repos
from datasets import load_dataset
ds = load_dataset('ScaleAI/SWE-bench_Pro', split='test')
df = ds.to_pandas()
print(f'Total instances: {len(df)}')
print(f'Unique repositories: {df["repo"].nunique()}')
print(f'Repositories: {df["repo"].unique().tolist()}')
print(f'Repo counts:\\n{df["repo"].value_counts()}')
Total instances: 731
Unique repositories: 11
Repositories: ['NodeBB/NodeBB', 'qutebrowser/qutebrowser', 'ansible/ansible', 'internetarchive/openlibrary', 'gravitational/teleport', 'navidrome/navidrome', 'element-hq/element-web', 'future-architect/vuls', 'protonmail/webclients', 'flipt-io/flipt', 'tutao/tutanota']
Repo counts:
ansible/ansible 96
internetarchive/openlibrary 91
flipt-io/flipt 85
qutebrowser/qutebrowser 79
gravitational/teleport 76
protonmail/webclients 65
future-architect/vuls 62
navidrome/navidrome 57
element-hq/element-web 56
NodeBB/NodeBB 44
tutao/tutanota 20
✅ VERIFIED (public split)
| Aspect | Paper Claim | Verified (Public) |
|---|---|---|
| Total problems | 1,865 | 731 (public) + 276 (commercial) + 858 (held-out) = 1,865 |
| Total repositories | 41 | 11 (public) + 18 (commercial) + 12 (held-out) = 41 |
| Public problems | 731 | 731 ✓ |
| Public repositories | 11 | 11 ✓ |
| Actively maintained | Yes | All 11 are active projects with recent commits |
The public split exactly matches the paper's stated numbers (731 instances, 11 repositories). The remaining splits (commercial: 276 problems from 18 startup repos; held-out: 858 problems from 12 repos) are documented in the paper (§3.3, Table 1) but not publicly released. Repository activity confirmed via GitHub (all have commits within the last year).
Sources:
- Paper: §3.3 "Public, Commercial, and Held-Out SWE-BENCH PRO", Table 1
- Dataset: ScaleAI/SWE-bench_Pro (test split, 731 instances)
- GitHub repos: ansible/ansible, internetarchive/openlibrary, flipt-io/flipt, qutebrowser/qutebrowser, gravitational/teleport, protonmail/webclients, future-architect/vuls, navidrome/navidrome, element-hq/element-web, NodeBB/NodeBB, tutao/tutanota
Reproduction run (executed HF Job). All five claims were recomputed on Hugging Face Jobs (cpu-basic, ~$0.0006): Job 6a5da3acbee6ee1cf4ed1e8e. The run loads ScaleAI/SWE-bench_Pro (test split, 731 rows) and prints this claim as VERIFIED in the immutable job logs. Script: repro_job/run_claims.py in the reproduction bundle.