# Claim 1: SWE-Bench Pro comprises 1,865 problems sourced from 41 actively maintained software repositories (abstract only).


---
<!-- trackio-cell
{"type": "markdown", "id": "cell_e52f8c2869c0", "created_at": "2026-07-19T07:47:13+00:00", "title": "Claim 1 — 1,865 problems / 41 repos"}
-->
## Claim 1 — Scale of the benchmark

**Claim (verbatim):** "SWE-Bench Pro comprises 1,865 problems sourced from 41 actively maintained software repositories."

**Verdict:** Partially verified — the public subset (731 problems / 11 repos) is directly countable; the full 1,865 / 41 figure is stated by the paper and corroborated by the official release partner Scale AI.

### What we verified
We loaded the official public benchmark release **[ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro)** (the canonical HF copy released by Scale AI, the paper's benchmark partner) and counted its contents:

- **731 problems** across **11 repositories** in the publicly accessible split.
- Per-repository instance counts (all 11 public repos):

| Repository | Instances |
|---|---|
| ansible/ansible | 96 |
| internetarchive/openlibrary | 91 |
| flipt-io/flipt | 85 |
| qutebrowser/qutebrowser | 79 |
| gravitational/teleport | 76 |
| future-architect/vuls | 62 |
| protonmail/webclients | 65 |
| navidrome/navidrome | 57 |
| element-hq/element-web | 56 |
| NodeBB/NodeBB | 44 |
| tutao/tutanota | 20 |

### Relationship to the headline 1,865 / 41
The paper states the full benchmark is 1,865 problems from 41 repos, partitioned into a **public set of 11 repos**, a **held-out set of 12 repos**, and a **commercial set of 18 proprietary repos** (11 + 12 + 18 = 41). The 11 public repos we counted exactly match the "public set of 11" in Claim 2, and 731 is the released count of that public set. The held-out (12) and commercial (18) splits are explicitly **not publicly accessible** by design, so their 1,134 instances (1,865 − 731) cannot be independently counted — this is consistent with the paper's contamination-resistance design (Claim 5) rather than a discrepancy.

### Evidence
- HF Job (analysis run): https://huggingface.co/jobs/Yashp2003/6a5c8046d216bd6f3a2009d5
- Results artifact: `claim_analysis_results.json` in https://huggingface.co/buckets/Yashp2003/swebenchpro-repro-artifacts
- Source dataset: https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro
- Official eval repo: https://github.com/scaleapi/SWE-bench_Pro-os