| # Claim 1: SWE-Bench Pro comprises 1,865 problems sourced from 41 actively maintained software repositories (abstract only). |
|
|
|
|
| --- |
| <!-- trackio-cell |
| {"type": "markdown", "id": "cell_e52f8c2869c0", "created_at": "2026-07-19T07:47:13+00:00", "title": "Claim 1 — 1,865 problems / 41 repos"} |
| --> |
| ## Claim 1 — Scale of the benchmark |
|
|
| **Claim (verbatim):** "SWE-Bench Pro comprises 1,865 problems sourced from 41 actively maintained software repositories." |
|
|
| **Verdict:** Partially verified — the public subset (731 problems / 11 repos) is directly countable; the full 1,865 / 41 figure is stated by the paper and corroborated by the official release partner Scale AI. |
|
|
| ### What we verified |
| We loaded the official public benchmark release **[ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro)** (the canonical HF copy released by Scale AI, the paper's benchmark partner) and counted its contents: |
|
|
| - **731 problems** across **11 repositories** in the publicly accessible split. |
| - Per-repository instance counts (all 11 public repos): |
|
|
| | Repository | Instances | |
| |---|---| |
| | ansible/ansible | 96 | |
| | internetarchive/openlibrary | 91 | |
| | flipt-io/flipt | 85 | |
| | qutebrowser/qutebrowser | 79 | |
| | gravitational/teleport | 76 | |
| | future-architect/vuls | 62 | |
| | protonmail/webclients | 65 | |
| | navidrome/navidrome | 57 | |
| | element-hq/element-web | 56 | |
| | NodeBB/NodeBB | 44 | |
| | tutao/tutanota | 20 | |
|
|
| ### Relationship to the headline 1,865 / 41 |
| The paper states the full benchmark is 1,865 problems from 41 repos, partitioned into a **public set of 11 repos**, a **held-out set of 12 repos**, and a **commercial set of 18 proprietary repos** (11 + 12 + 18 = 41). The 11 public repos we counted exactly match the "public set of 11" in Claim 2, and 731 is the released count of that public set. The held-out (12) and commercial (18) splits are explicitly **not publicly accessible** by design, so their 1,134 instances (1,865 − 731) cannot be independently counted — this is consistent with the paper's contamination-resistance design (Claim 5) rather than a discrepancy. |
|
|
| ### Evidence |
| - HF Job (analysis run): https://huggingface.co/jobs/Yashp2003/6a5c8046d216bd6f3a2009d5 |
| - Results artifact: `claim_analysis_results.json` in https://huggingface.co/buckets/Yashp2003/swebenchpro-repro-artifacts |
| - Source dataset: https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro |
| - Official eval repo: https://github.com/scaleapi/SWE-bench_Pro-os |
|
|
