File size: 3,998 Bytes
bd3727b c5966da bd3727b c5966da bd3727b c5966da 58e12ec c5966da 58e12ec bd3727b ef3f9a0 6497101 c5966da | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | # Claim 1: SWE-Bench Pro comprises 1,865 problems sourced from 41 actively maintained software repositories (abstract only).
---
<!-- trackio-cell
{"type": "markdown", "id": "cell_e52f8c2869c0", "created_at": "2026-07-19T07:47:13+00:00", "title": "Claim 1 — 1,865 problems / 41 repos"}
-->
## Claim 1 — Scale of the benchmark
**Claim (verbatim from abstract):** "SWE-BENCH PRO contains 1,865 problems sourced from a diverse set of 41 actively maintained repositories."
**Verdict:** VERIFIED — public subset (731/11) directly counted from [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro); full 1,865/41 stated in paper abstract and internally consistent (11+12+18=41). Eval repo confirms full dataset exists.
### What we verified independently
- Loaded the official public benchmark release **[ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro)** (60,512 downloads, the canonical release by Scale AI)
- Counted **731 problems** across **11 distinct repositories** in the public split
- Per-repository counts:
| Repository | Instances | Language |
|---|---|---|
| ansible/ansible | 96 | Python |
| internetarchive/openlibrary | 91 | Python |
| flipt-io/flipt | 85 | Go |
| qutebrowser/qutebrowser | 79 | Python |
| gravitational/teleport | 76 | Go |
| future-architect/vuls | 62 | Go |
| protonmail/webclients | 65 | JavaScript |
| navidrome/navidrome | 57 | Go |
| element-hq/element-web | 56 | TypeScript |
| NodeBB/NodeBB | 44 | JavaScript |
| tutao/tutanota | 20 | TypeScript |
### Relationship to the headline 1,865 / 41
The paper states the full benchmark is 1,865 problems from 41 repos, partitioned into public (11), held-out (12), and commercial (18). The 11 public repos we counted match the "public set of 11" in Claim 2, and 731 is the released count. The held-out (12) and commercial (18) splits are deliberately non-public — this is the contamination-resistance design (Claim 5), not a discrepancy.
### Cross-reference: eval repo
The official eval repo ([scaleapi/SWE-bench_Pro-os](https://github.com/scaleapi/SWE-bench_Pro-os), 480 stars, 93 forks) contains **1,000+ run_script directories** (GitHub API paginated). This exceeds the 731 public instances, confirming the full dataset is larger. The eval script references `swe_bench_pro_full.csv`, and the repo maintains separate public/private leaderboards.
### FALSIFICATION: "123 unique programming languages"
The judge's consolidated claim includes "spanning 123 unique programming languages." **This phrase does NOT appear anywhere in the paper.** We extracted and searched the full paper PDF (arXiv 2509.16941). The paper explicitly states in Section 7.1 (Limitations): "Although SWE-BENCH PRO includes multiple programming languages (Python, JavaScript, TypeScript, Go), the distribution is not uniform, and some widely-used languages like Java, C++, and Rust are underrepresented." The paper covers **4 languages** in SWE-Bench Pro. The "123" number is a judge-model hallucination.
### Evidence
- **HF Job:** [Yashp2003/6a5c8046d216bd6f3a2009d5](https://huggingface.co/jobs/Yashp2003/6a5c8046d216bd6f3a2009d5) — analysis run (CPU, ~$0.005)
- **Results:** [`claim_analysis_results.json`](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/claim_analysis_results.json) in [swebenchpro-repro-artifacts](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts) bucket
- **Dataset:** [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro) — 731 public instances
- **Eval repo:** [scaleapi/SWE-bench_Pro-os](https://github.com/scaleapi/SWE-bench_Pro-os) — 480 stars, 93 forks
### What we cannot independently verify
The full 1,865 count cannot be verified from the public HF dataset alone (which has 731 rows). The missing 1,134 instances correspond to held-out and commercial splits that are deliberately non-public — this is the benchmark's contamination-resistance mechanism, not a discrepancy.
|