pages/claim-1-swe-bench-pro-comprises-1-865-problems-from-41-actively-maintained-repositories/page.md

# Claim 1: SWE-Bench Pro comprises 1,865 problems from 41 actively maintained repositories.

---
<!-- trackio-cell
{"type": "markdown", "id": "cell_claim1_setup", "created_at": "2026-07-20T02:31:54+00:00", "title": "Setup & Verification"}
-->
**Paper claim (Abstract, §3.3):** SWE-Bench Pro contains 1,865 problems sourced from 41 actively maintained repositories spanning business applications, B2B services, and developer tools.

**Verification approach:**
1. Load public split from Hugging Face dataset `ScaleAI/SWE-bench_Pro` (test split)
2. Count instances and unique repositories
3. Cross-reference with paper's stated splits: 731 public (11 repos) + 276 commercial (18 repos) + 858 held-out (12 repos) = 1,865 total from 41 repos

---
<!-- trackio-cell
{"type": "code", "id": "cell_claim1_load", "created_at": "2026-07-20T02:35:00+00:00", "title": "Load public dataset"}
-->
```python
from datasets import load_dataset
ds = load_dataset('ScaleAI/SWE-bench_Pro', split='test')
df = ds.to_pandas()
print(f'Total instances: {len(df)}')
print(f'Unique repositories: {df["repo"].nunique()}')
print(f'Repositories: {df["repo"].unique().tolist()}')
print(f'Repo counts:\\n{df["repo"].value_counts()}')
```

---
<!-- trackio-cell
{"type": "code", "id": "cell_claim1_result", "created_at": "2026-07-20T02:35:10+00:00", "title": "Public split verification"}
-->
```
Total instances: 731
Unique repositories: 11
Repositories: ['NodeBB/NodeBB', 'qutebrowser/qutebrowser', 'ansible/ansible', 'internetarchive/openlibrary', 'gravitational/teleport', 'navidrome/navidrome', 'element-hq/element-web', 'future-architect/vuls', 'protonmail/webclients', 'flipt-io/flipt', 'tutao/tutanota']
Repo counts:
ansible/ansible                96
internetarchive/openlibrary    91
flipt-io/flipt                 85
qutebrowser/qutebrowser        79
gravitational/teleport         76
protonmail/webclients          65
future-architect/vuls          62
navidrome/navidrome            57
element-hq/element-web         56
NodeBB/NodeBB                  44
tutao/tutanota                 20
```

---
<!-- trackio-cell
{"type": "markdown", "id": "cell_claim1_verdict", "created_at": "2026-07-20T02:36:00+00:00", "title": "Verdict"}
-->
**✅ VERIFIED (public split)**

| Aspect | Paper Claim | Verified (Public) |
|--------|-------------|-------------------|
| Total problems | 1,865 | 731 (public) + 276 (commercial) + 858 (held-out) = 1,865 |
| Total repositories | 41 | 11 (public) + 18 (commercial) + 12 (held-out) = 41 |
| Public problems | 731 | **731 ✓** |
| Public repositories | 11 | **11 ✓** |
| Actively maintained | Yes | All 11 are active projects with recent commits |

The public split exactly matches the paper's stated numbers (731 instances, 11 repositories). The remaining splits (commercial: 276 problems from 18 startup repos; held-out: 858 problems from 12 repos) are documented in the paper (§3.3, Table 1) but not publicly released. Repository activity confirmed via GitHub (all have commits within the last year).

**Sources:**
- Paper: §3.3 "Public, Commercial, and Held-Out SWE-BENCH PRO", Table 1
- Dataset: [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro) (test split, 731 instances)
- GitHub repos: [ansible/ansible](https://github.com/ansible/ansible), [internetarchive/openlibrary](https://github.com/internetarchive/openlibrary), [flipt-io/flipt](https://github.com/flipt-io/flipt), [qutebrowser/qutebrowser](https://github.com/qutebrowser/qutebrowser), [gravitational/teleport](https://github.com/gravitational/teleport), [protonmail/webclients](https://github.com/protonmail/webclients), [future-architect/vuls](https://github.com/future-architect/vuls), [navidrome/navidrome](https://github.com/navidrome/navidrome), [element-hq/element-web](https://github.com/element-hq/element-web), [NodeBB/NodeBB](https://github.com/NodeBB/NodeBB), [tutao/tutanota](https://github.com/tutao/tutanota)

---

**Reproduction run (executed HF Job).** All five claims were recomputed on Hugging Face Jobs (`cpu-basic`, ~\$0.0006): [Job `6a5da3acbee6ee1cf4ed1e8e`](https://huggingface.co/jobs/Yashp2003/6a5da3acbee6ee1cf4ed1e8e). The run loads `ScaleAI/SWE-bench_Pro` (test split, 731 rows) and prints this claim as **VERIFIED** in the immutable job logs. Script: `repro_job/run_claims.py` in the [reproduction bundle](https://huggingface.co/buckets/Yashp2003/repro-swe-bench-pro-can-ai-agents-solve-long-horizon-software-engineering-tasks-artifacts#repro-bundle).