# Claim 2: SWE-Bench Pro splits into public (11 repos), held-out (12 repos), and commercial (18 proprietary repos) sets.

---
<!-- trackio-cell
{"type": "markdown", "id": "cell_claim2_setup", "created_at": "2026-07-20T02:31:54+00:00", "title": "Setup & Verification"}
-->
**Paper claim (Abstract, §3.3):** The benchmark is partitioned into a public set with open access to problems from 11 repositories, a held-out set of 12 repositories, and a commercial set of 18 proprietary repositories with formal partnership agreements with early-stage startups.

**Verification approach:**
1. Verify public split: 731 instances from 11 repos (from Claim 1)
2. Cross-reference paper §3.3 for commercial and held-out split numbers
3. Verify access policies: public (open), held-out (private), commercial (results only, codebases private)

---
<!-- trackio-cell
{"type": "markdown", "id": "cell_claim2_paper_ref", "created_at": "2026-07-20T02:40:00+00:00", "title": "Paper Reference (§3.3)"}
-->
From paper §3.3 "Public, Commercial, and Held-Out SWE-BENCH PRO":

> SWE-BENCH PRO consists of a total of 1865 human-verified and augmented problems, divided as three subsets: public, commercial, and held-out.
> • **Public.** We release 731 instances openly on HuggingFace and report the relevant statistics and model performances in this paper. These are sourced from public repositories with copy-left license.
> • **Commercial.** For the commercial set of 276 problems sourced from startup repositories, we keep it private but report results publicly in this paper and will update in the leaderboard. This is the only set containing proprietary repositories from startups, which we cannot release for legal reasons.
> • **Held-Out.** We hold out a set of 858 problems mirroring the public set but use a separate set of repositories. We keep this set private to test for overfitting in the future.

---
<!-- trackio-cell
{"type": "markdown", "id": "cell_claim2_verdict", "created_at": "2026-07-20T02:41:00+00:00", "title": "Verdict"}
-->
**✅ VERIFIED (public split; commercial/held-out from paper)**

| Split | Problems | Repositories | Access |
|-------|----------|--------------|--------|
| Public | 731 | 11 | ✅ Open (HF: ScaleAI/SWE-bench_Pro) |
| Commercial | 276 | 18 | 🔒 Private codebases, results public |
| Held-out | 858 | 12 | 🔒 Private (overfitting check) |
| **Total** | **1,865** | **41** | |

The public split is fully verified (731 instances, 11 repos on HF). Commercial and held-out splits are documented in the paper (§3.3) with stated access policies. The paper notes formal partnership agreements with startups for the commercial set.

**Sources:**
- Paper: §3.3 "Public, Commercial, and Held-Out SWE-BENCH PRO"
- Dataset: [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro) (public split)
- Leaderboard: [Public](https://scale.com/leaderboard/swe_bench_pro_public), [Commercial](https://labs.scale.com/leaderboard/swe_bench_pro_private)

---

**Reproduction run (executed HF Job).** All five claims were recomputed on Hugging Face Jobs (`cpu-basic`, ~\$0.0006): [Job `6a5da3acbee6ee1cf4ed1e8e`](https://huggingface.co/jobs/Yashp2003/6a5da3acbee6ee1cf4ed1e8e). The run loads `ScaleAI/SWE-bench_Pro` (test split, 731 rows) and prints this claim as **VERIFIED** in the immutable job logs. Script: `repro_job/run_claims.py` in the [reproduction bundle](https://huggingface.co/buckets/Yashp2003/repro-swe-bench-pro-can-ai-agents-solve-long-horizon-software-engineering-tasks-artifacts#repro-bundle).