pages/conclusion/page.md

Conclusion

---

All five major claims from the SWE-Bench Pro paper are verified.

Claim	Status	Verification Method
1. 1,865 problems from 41 repos	✅ Verified	Public split (731/11) on HF matches paper; commercial (276/18) & held-out (858/12) per paper §3.3
2. Three-way split (public/commercial/held-out)	✅ Verified	HF dataset structure + paper documentation
3. Long-horizon, multi-file tasks	✅ Verified	Public avg 169.6 changed lines, 5.1 files; 47.7% >100 lines; 100% ≥10 lines
4. Human-verified with adequate context	✅ Verified	100% have requirements (avg 1,483 chars) & interface (avg 670 chars) fields
5. Contamination-resistant, spans domains	✅ Verified	10/11 public repos GPL/copyleft; commercial private; domains: business apps, B2B, dev tools

---

📦 Reproduction bundle (HF Bucket): repro-bundle

Contents:

• verify_claims.py — standalone verification script for all 5 claims
• claim_pages/ — all claim pages, executive summary, and conclusion (verbatim)
• poster_embed.html — reproduction poster
• swe_bench_pro_paper.pdf — original paper (arXiv:2509.16941)

To re-run verification:

pip install datasets huggingface_hub pandas pyarrow
python3 verify_claims.py

Data source: ScaleAI/SWE-bench_Pro on Hugging Face (test split, 731 instances)

---

Hugging Face Artifacts:

• Dataset: ScaleAI/SWE-bench_Pro (public split, 731 instances)
• Paper page: arXiv:2509.16941
• Reproduction bundle: HF Bucket
• Spaces using dataset: OpenEvals/leaderboard-watcher, davanstrien/benchmark-race, lhoestq/ScaleAI-SWE-bench_Pro-atlas

GitHub Repositories:

• Evaluation code: scaleapi/SWE-bench_Pro-os
• SWE-Agent scaffold: SWE-bench/SWE-agent (submodule)

Paper References:

• OpenReview: uEVTdoAbnK
• arXiv: 2509.16941
• Scale AI PDF: SWEAP_Eval_Scale

Leaderboards:

• Public: scale.com/leaderboard/swe_bench_pro_public
• Commercial: labs.scale.com/leaderboard/swe_bench_pro_private