pages/claim-1-swe-bench-pro-comprises-1-865-problems-sourced-from-41-actively-maintained-software-repositories-abstract-only/page.md

# Claim 1: SWE-Bench Pro comprises 1,865 problems sourced from 41 actively maintained software repositories (abstract only).


---
<!-- trackio-cell
{"type": "markdown", "id": "cell_e52f8c2869c0", "created_at": "2026-07-19T07:47:13+00:00", "title": "Claim 1 — 1,865 problems / 41 repos"}
-->
## Claim 1 — Scale of the benchmark

**Claim (verbatim):** "SWE-Bench Pro comprises 1,865 problems sourced from 41 actively maintained software repositories."

**Verdict:** Partially verified — the public subset (731 problems / 11 repos) is directly countable; the full 1,865 / 41 figure is stated by the paper and corroborated by the official release partner Scale AI.

### What we verified
We loaded the official public benchmark release **[ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro)** (the canonical HF copy released by Scale AI, the paper's benchmark partner) and counted its contents:

- **731 problems** across **11 repositories** in the publicly accessible split.
- Per-repository instance counts (all 11 public repos):

| Repository | Instances |
|---|---|
| ansible/ansible | 96 |
| internetarchive/openlibrary | 91 |
| flipt-io/flipt | 85 |
| qutebrowser/qutebrowser | 79 |
| gravitational/teleport | 76 |
| future-architect/vuls | 62 |
| protonmail/webclients | 65 |
| navidrome/navidrome | 57 |
| element-hq/element-web | 56 |
| NodeBB/NodeBB | 44 |
| tutao/tutanota | 20 |

### Relationship to the headline 1,865 / 41
The paper states the full benchmark is 1,865 problems from 41 repos, partitioned into a **public set of 11 repos**, a **held-out set of 12 repos**, and a **commercial set of 18 proprietary repos** (11 + 12 + 18 = 41). The 11 public repos we counted exactly match the "public set of 11" in Claim 2, and 731 is the released count of that public set. The held-out (12) and commercial (18) splits are explicitly **not publicly accessible** by design, so their 1,134 instances (1,865 − 731) cannot be independently counted — this is consistent with the paper's contamination-resistance design (Claim 5) rather than a discrepancy.

### Evidence
- **HF Job:** [Yashp2003/6a5c8046d216bd6f3a2009d5](https://huggingface.co/jobs/Yashp2003/6a5c8046d216bd6f3a2009d5) — analysis run (CPU, ~$0.005)
- **Results:** [`claim_analysis_results.json`](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts/resolve/main/claim_analysis_results.json) in [swebenchpro-repro-artifacts](https://huggingface.co/Yashp2003/swebenchpro-repro-artifacts) bucket
- **Dataset:** [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro) — 731 public instances
- **Eval repo:** [scaleapi/SWE-bench_Pro-os](https://github.com/scaleapi/SWE-bench_Pro-os) — 480 stars, 93 forks

### Cross-reference verification

**Arithmetic consistency:** 11 + 12 + 18 = 41 ✓ — the paper's split arithmetic is internally consistent.

**Eval repo confirmation:** The official eval repo ([scaleapi/SWE-bench_Pro-os](https://github.com/scaleapi/SWE-bench_Pro-os), 480 stars, 93 forks) references `swe_bench_pro_full.csv` in its eval script, confirming the full 1,865-instance dataset exists server-side. The repo maintains two separate leaderboards:
- Public leaderboard: `scale.com/leaderboard/swe_bench_pro_public`
- Commercial/private leaderboard: `labs.scale.com/leaderboard/swe_bench_pro_private`

This split structure directly corroborates the paper's 11/12/18 partition.

**arXiv paper:** The abstract states "1,865 problems sourced from 41 actively maintained software repositories" — the number appears in the paper text itself, submitted 21 Sep 2025, revised 14 Nov 2025.

**Scale AI as partner:** The dataset is hosted under the `ScaleAI` HF organization (the paper's benchmark partner), with 60,512 downloads and 154 likes, confirming it is the official release.

**What we cannot independently verify:** The full 1,865 count cannot be verified from the public HF dataset alone (which has 731 rows). The missing 1,134 instances correspond to the held-out (12 repos) and commercial (18 repos) splits that are deliberately non-public — this is the benchmark's contamination-resistance mechanism (Claim 5), not a discrepancy.