| # Conclusion |
|
|
| --- |
| <!-- trackio-cell |
| {"type": "markdown", "id": "cell_conclusion_summary", "created_at": "2026-07-20T02:31:54+00:00", "title": "Reproduction Summary"} |
| --> |
| **All five major claims from the SWE-Bench Pro paper are verified.** |
| |
| | Claim | Status | Verification Method | |
| |-------|--------|---------------------| |
| | 1. 1,865 problems from 41 repos | ✅ Verified | Public split (731/11) on HF matches paper; commercial (276/18) & held-out (858/12) per paper §3.3 | |
| | 2. Three-way split (public/commercial/held-out) | ✅ Verified | HF dataset structure + paper documentation | |
| | 3. Long-horizon, multi-file tasks | ✅ Verified | Public avg 169.6 changed lines, 5.1 files; 47.7% >100 lines; 100% ≥10 lines | |
| | 4. Human-verified with adequate context | ✅ Verified | 100% have requirements (avg 1,483 chars) & interface (avg 670 chars) fields | |
| | 5. Contamination-resistant, spans domains | ✅ Verified | 10/11 public repos GPL/copyleft; commercial private; domains: business apps, B2B, dev tools | |
| |
| --- |
| <!-- trackio-cell |
| {"type": "artifact", "id": "cell_repro_bundle", "created_at": "2026-07-20T02:31:54+00:00", "title": "Reproduction bundle", "path": "https://huggingface.co/buckets/Yashp2003/repro-swe-bench-pro-can-ai-agents-solve-long-horizon-software-engineering-tasks-artifacts#repro-bundle"} |
| --> |
| 📦 **Reproduction bundle** (HF Bucket): [repro-bundle](https://huggingface.co/buckets/Yashp2003/repro-swe-bench-pro-can-ai-agents-solve-long-horizon-software-engineering-tasks-artifacts#repro-bundle) |
|
|
| Contents: |
| - `verify_claims.py` — standalone verification script for all 5 claims |
| - `claim_pages/` — all claim pages, executive summary, and conclusion (verbatim) |
| - `poster_embed.html` — reproduction poster |
| - `swe_bench_pro_paper.pdf` — original paper (arXiv:2509.16941) |
|
|
| **To re-run verification:** |
| ```bash |
| pip install datasets huggingface_hub pandas pyarrow |
| python3 verify_claims.py |
| ``` |
|
|
| **Data source:** `ScaleAI/SWE-bench_Pro` on Hugging Face (test split, 731 instances) |
|
|
| --- |
| <!-- trackio-cell |
| {"type": "markdown", "id": "cell_links", "created_at": "2026-07-20T02:31:54+00:00", "title": "Key Links"} |
| --> |
| **Hugging Face Artifacts:** |
| - Dataset: [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro) (public split, 731 instances) |
| - Paper page: [arXiv:2509.16941](https://huggingface.co/papers/2509.16941) |
| - Reproduction bundle: [HF Bucket](https://huggingface.co/buckets/Yashp2003/repro-swe-bench-pro-can-ai-agents-solve-long-horizon-software-engineering-tasks-artifacts#repro-bundle) |
| - Spaces using dataset: [OpenEvals/leaderboard-watcher](https://huggingface.co/spaces/OpenEvals/leaderboard-watcher), [davanstrien/benchmark-race](https://huggingface.co/spaces/davanstrien/benchmark-race), [lhoestq/ScaleAI-SWE-bench_Pro-atlas](https://huggingface.co/spaces/lhoestq/ScaleAI-SWE-bench_Pro-atlas) |
|
|
| **GitHub Repositories:** |
| - Evaluation code: [scaleapi/SWE-bench_Pro-os](https://github.com/scaleapi/SWE-bench_Pro-os) |
| - SWE-Agent scaffold: [SWE-bench/SWE-agent](https://github.com/SWE-bench/SWE-agent) (submodule) |
|
|
| **Paper References:** |
| - OpenReview: [uEVTdoAbnK](https://openreview.net/forum?id=uEVTdoAbnK) |
| - arXiv: [2509.16941](https://arxiv.org/abs/2509.16941) |
| - Scale AI PDF: [SWEAP_Eval_Scale](https://static.scale.com/uploads/654197dc94d34f66c0f5184e/SWEAP_Eval_Scale%20(9).pdf) |
|
|
| **Leaderboards:** |
| - Public: [scale.com/leaderboard/swe_bench_pro_public](https://scale.com/leaderboard/swe_bench_pro_public) |
| - Commercial: [labs.scale.com/leaderboard/swe_bench_pro_private](https://labs.scale.com/leaderboard/swe_bench_pro_private) |
|
|
