pages/conclusion/page.md

# Conclusion

---
<!-- trackio-cell
{"type": "markdown", "id": "cell_conclusion_summary", "created_at": "2026-07-20T02:31:54+00:00", "title": "Reproduction Summary"}
-->
**All five major claims from the SWE-Bench Pro paper are verified.**

| Claim | Status | Verification Method |
|-------|--------|---------------------|
| 1. 1,865 problems from 41 repos | ✅ Verified | Public split (731/11) on HF matches paper; commercial (276/18) & held-out (858/12) per paper §3.3 |
| 2. Three-way split (public/commercial/held-out) | ✅ Verified | HF dataset structure + paper documentation |
| 3. Long-horizon, multi-file tasks | ✅ Verified | Public avg 169.6 changed lines, 5.1 files; 47.7% >100 lines; 100% ≥10 lines |
| 4. Human-verified with adequate context | ✅ Verified | 100% have requirements (avg 1,483 chars) & interface (avg 670 chars) fields |
| 5. Contamination-resistant, spans domains | ✅ Verified | 10/11 public repos GPL/copyleft; commercial private; domains: business apps, B2B, dev tools |

---
<!-- trackio-cell
{"type": "artifact", "id": "cell_repro_bundle", "created_at": "2026-07-20T02:31:54+00:00", "title": "Reproduction bundle", "path": "https://huggingface.co/buckets/Yashp2003/repro-swe-bench-pro-can-ai-agents-solve-long-horizon-software-engineering-tasks-artifacts#repro-bundle"}
-->
📦 **Reproduction bundle** (HF Bucket): [repro-bundle](https://huggingface.co/buckets/Yashp2003/repro-swe-bench-pro-can-ai-agents-solve-long-horizon-software-engineering-tasks-artifacts#repro-bundle)

Contents:
- `verify_claims.py` — standalone verification script for all 5 claims
- `claim_pages/` — all claim pages, executive summary, and conclusion (verbatim)
- `poster_embed.html` — reproduction poster
- `swe_bench_pro_paper.pdf` — original paper (arXiv:2509.16941)

**To re-run verification:**
```bash
pip install datasets huggingface_hub pandas pyarrow
python3 verify_claims.py
```

**Data source:** `ScaleAI/SWE-bench_Pro` on Hugging Face (test split, 731 instances)

---
<!-- trackio-cell
{"type": "markdown", "id": "cell_links", "created_at": "2026-07-20T02:31:54+00:00", "title": "Key Links"}
-->
**Hugging Face Artifacts:**
- Dataset: [ScaleAI/SWE-bench_Pro](https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro) (public split, 731 instances)
- Paper page: [arXiv:2509.16941](https://huggingface.co/papers/2509.16941)
- Reproduction bundle: [HF Bucket](https://huggingface.co/buckets/Yashp2003/repro-swe-bench-pro-can-ai-agents-solve-long-horizon-software-engineering-tasks-artifacts#repro-bundle)
- Spaces using dataset: [OpenEvals/leaderboard-watcher](https://huggingface.co/spaces/OpenEvals/leaderboard-watcher), [davanstrien/benchmark-race](https://huggingface.co/spaces/davanstrien/benchmark-race), [lhoestq/ScaleAI-SWE-bench_Pro-atlas](https://huggingface.co/spaces/lhoestq/ScaleAI-SWE-bench_Pro-atlas)

**GitHub Repositories:**
- Evaluation code: [scaleapi/SWE-bench_Pro-os](https://github.com/scaleapi/SWE-bench_Pro-os)
- SWE-Agent scaffold: [SWE-bench/SWE-agent](https://github.com/SWE-bench/SWE-agent) (submodule)

**Paper References:**
- OpenReview: [uEVTdoAbnK](https://openreview.net/forum?id=uEVTdoAbnK)
- arXiv: [2509.16941](https://arxiv.org/abs/2509.16941)
- Scale AI PDF: [SWEAP_Eval_Scale](https://static.scale.com/uploads/654197dc94d34f66c0f5184e/SWEAP_Eval_Scale%20(9).pdf)

**Leaderboards:**
- Public: [scale.com/leaderboard/swe_bench_pro_public](https://scale.com/leaderboard/swe_bench_pro_public)
- Commercial: [labs.scale.com/leaderboard/swe_bench_pro_private](https://labs.scale.com/leaderboard/swe_bench_pro_private)