File size: 4,493 Bytes
f3df659 0734535 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 | # Claim 1: SWE-Bench Pro comprises 1,865 problems from 41 actively maintained repositories.
---
<!-- trackio-cell
{"type": "markdown", "id": "cell_claim1_setup", "created_at": "2026-07-20T02:31:54+00:00", "title": "Setup & Verification"}
-->
**Paper claim (Abstract, §3.3):** SWE-Bench Pro contains 1,865 problems sourced from 41 actively maintained repositories spanning business applications, B2B services, and developer tools.
**Verification approach:**
1. Load public split from Hugging Face dataset `ScaleAI/SWE-bench_Pro` (test split)
2. Count instances and unique repositories
3. Cross-reference with paper's stated splits: 731 public (11 repos) + 276 commercial (18 repos) + 858 held-out (12 repos) = 1,865 total from 41 repos
---
<!-- trackio-cell
{"type": "code", "id": "cell_claim1_load", "created_at": "2026-07-20T02:35:00+00:00", "title": "Load public dataset"}
-->
```python
from datasets import load_dataset
ds = load_dataset('ScaleAI/SWE-bench_Pro', split='test')
df = ds.to_pandas()
print(f'Total instances: {len(df)}')
print(f'Unique repositories: {df["repo"].nunique()}')
print(f'Repositories: {df["repo"].unique().tolist()}')
print(f'Repo counts:\\n{df["repo"].value_counts()}')
```
---
<!-- trackio-cell
{"type": "code", "id": "cell_claim1_result", "created_at": "2026-07-20T02:35:10+00:00", "title": "Public split verification"}
-->
```
Total instances: 731
Unique repositories: 11
Repositories: ['NodeBB/NodeBB', 'qutebrowser/qutebrowser', 'ansible/ansible', 'internetarchive/openlibrary', 'gravitational/teleport', 'navidrome/navidrome', 'element-hq/element-web', 'future-architect/vuls', 'protonmail/webclients', 'flipt-io/flipt', 'tutao/tutanota']
Repo counts:
ansible/ansible 96
internetarchive/openlibrary 91
flipt-io/flipt 85
qutebrowser/qutebrowser 79
gravitational/teleport 76
protonmail/webclients 65
future-architect/vuls 62
navidrome/navidrome 57
element-hq/element-web 56
NodeBB/NodeBB 44
tutao/tutanota 20
```
---
<!-- trackio-cell
{"type": "markdown", "id": "cell_claim1_verdict", "created_at": "2026-07-20T02:36:00+00:00", "title": "Verdict"}
-->
**✅ VERIFIED (public split)**
| Aspect | Paper Claim | Verified (Public) |
|--------|-------------|-------------------|
| Total problems | 1,865 | 731 (public) + 276 (commercial) + 858 (held-out) = 1,865 |
| Total repositories | 41 | 11 (public) + 18 (commercial) + 12 (held-out) = 41 |
| Public problems | 731 | **731 ✓** |
| Public repositories | 11 | **11 ✓** |
| Actively maintained | Yes | All 11 are active projects with recent commits |
The public split exactly matches the paper's stated numbers (731 instances, 11 repositories). The remaining splits (commercial: 276 problems from 18 startup repos; held-out: 858 problems from 12 repos) are documented in the paper (§3.3, Table 1) but not publicly released. Repository activity confirmed via GitHub (all have commits within the last year).
**Sources:**
- Paper: §3.3 "Public, Commercial, and Held-Out SWE-BENCH PRO", Table 1
- Dataset: [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro) (test split, 731 instances)
- GitHub repos: [ansible/ansible](https://github.com/ansible/ansible), [internetarchive/openlibrary](https://github.com/internetarchive/openlibrary), [flipt-io/flipt](https://github.com/flipt-io/flipt), [qutebrowser/qutebrowser](https://github.com/qutebrowser/qutebrowser), [gravitational/teleport](https://github.com/gravitational/teleport), [protonmail/webclients](https://github.com/protonmail/webclients), [future-architect/vuls](https://github.com/future-architect/vuls), [navidrome/navidrome](https://github.com/navidrome/navidrome), [element-hq/element-web](https://github.com/element-hq/element-web), [NodeBB/NodeBB](https://github.com/NodeBB/NodeBB), [tutao/tutanota](https://github.com/tutao/tutanota)
---
**Reproduction run (executed HF Job).** All five claims were recomputed on Hugging Face Jobs (`cpu-basic`, ~\$0.0006): [Job `6a5da3acbee6ee1cf4ed1e8e`](https://huggingface.co/jobs/Yashp2003/6a5da3acbee6ee1cf4ed1e8e). The run loads `ScaleAI/SWE-bench_Pro` (test split, 731 rows) and prints this claim as **VERIFIED** in the immutable job logs. Script: `repro_job/run_claims.py` in the [reproduction bundle](https://huggingface.co/buckets/Yashp2003/repro-swe-bench-pro-can-ai-agents-solve-long-horizon-software-engineering-tasks-artifacts#repro-bundle).
|