| # Claim 1: SWE-Bench Pro comprises 1,865 problems sourced from 41 actively maintained software repositories (abstract only). |
|
|
|
|
| --- |
| <!-- trackio-cell |
| {"type": "markdown", "id": "cell_e52f8c2869c0", "created_at": "2026-07-19T07:47:13+00:00", "title": "Claim 1 — 1,865 problems / 41 repos"} |
| --> |
| ## Claim 1 — Scale of the benchmark |
|
|
| **Claim (verbatim from abstract):** "SWE-BENCH PRO contains 1,865 problems sourced from a diverse set of 41 actively maintained repositories." |
|
|
| **Verdict:** VERIFIED — public subset (731/11) directly counted from [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro); full 1,865/41 stated in paper abstract and internally consistent (11+12+18=41). Eval repo confirms full dataset exists. |
|
|
| ### What we verified independently |
| - Loaded the official public benchmark release **[ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro)** (60,512 downloads, the canonical release by Scale AI) |
| - Counted **731 problems** across **11 distinct repositories** in the public split |
| - Per-repository counts: |
|
|
| | Repository | Instances | Language | |
| |---|---|---| |
| | ansible/ansible | 96 | Python | |
| | internetarchive/openlibrary | 91 | Python | |
| | flipt-io/flipt | 85 | Go | |
| | qutebrowser/qutebrowser | 79 | Python | |
| | gravitational/teleport | 76 | Go | |
| | future-architect/vuls | 62 | Go | |
| | protonmail/webclients | 65 | JavaScript | |
| | navidrome/navidrome | 57 | Go | |
| | element-hq/element-web | 56 | TypeScript | |
| | NodeBB/NodeBB | 44 | JavaScript | |
| | tutao/tutanota | 20 | TypeScript | |
|
|
| ### Relationship to the headline 1,865 / 41 |
| The paper states the full benchmark is 1,865 problems from 41 repos, partitioned into public (11), held-out (12), and commercial (18). The 11 public repos we counted match the "public set of 11" in Claim 2, and 731 is the released count. The held-out (12) and commercial (18) splits are deliberately non-public — this is the contamination-resistance design (Claim 5), not a discrepancy. |
|
|
| ### Cross-reference: eval repo |
| The official eval repo ([scaleapi/SWE-bench_Pro-os](https://github.com/scaleapi/SWE-bench_Pro-os), 480 stars, 93 forks) contains **1,000+ run_script directories** (GitHub API paginated). This exceeds the 731 public instances, confirming the full dataset is larger. The eval script references `swe_bench_pro_full.csv`, and the repo maintains separate public/private leaderboards. |
| |
| ### FALSIFICATION: "123 unique programming languages" |
| The judge's consolidated claim includes "spanning 123 unique programming languages." **This phrase does NOT appear anywhere in the paper.** We extracted and searched the full paper PDF (arXiv 2509.16941). The paper explicitly states in Section 7.1 (Limitations): "Although SWE-BENCH PRO includes multiple programming languages (Python, JavaScript, TypeScript, Go), the distribution is not uniform, and some widely-used languages like Java, C++, and Rust are underrepresented." The paper covers **4 languages** in SWE-Bench Pro. The "123" number is a judge-model hallucination. |
|
|
| ### Evidence |
| - **HF Job:** [Yashp2003/6a5c8046d216bd6f3a2009d5](https://huggingface.co/jobs/Yashp2003/6a5c8046d216bd6f3a2009d5) — analysis run (CPU, ~$0.005) |
| - **Results:** [`claim_analysis_results.json`](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/claim_analysis_results.json) in [swebenchpro-repro-artifacts](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts) bucket |
| - **Dataset:** [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro) — 731 public instances |
| - **Eval repo:** [scaleapi/SWE-bench_Pro-os](https://github.com/scaleapi/SWE-bench_Pro-os) — 480 stars, 93 forks |
|
|
| ### What we cannot independently verify |
| The full 1,865 count cannot be verified from the public HF dataset alone (which has 731 rows). The missing 1,134 instances correspond to held-out and commercial splits that are deliberately non-public — this is the benchmark's contamination-resistance mechanism, not a discrepancy. |
|
|
