Executive summary
All five major claims from the SWE-Bench Pro paper are verified. The public dataset on Hugging Face (ScaleAI/SWE-bench_Pro) contains 731 instances from 11 repositories, matching the paper's public split. The paper's full benchmark comprises 1,865 problems from 41 repositories (731 public + 276 commercial + 858 held-out), split across public (11 repos), held-out (12 repos), and commercial (18 proprietary repos) sets. Tasks are confirmed long-horizon: public instances average 170 changed lines of patch across 5.1 files (paper reports 107 lines across 4.1 files for full set), with 349/731 tasks (47.7%) exceeding 100 changed lines. All 731 public instances have human-augmented requirements and interface fields. The benchmark is contamination-resistant: public/held-out repos use GPL/copyleft licenses, commercial repos are private startup codebases. Repositories span business applications (Ansible, OpenLibrary), B2B services (Flipt, Teleport, Tutanota), and developer tools (qutebrowser, Navidrome, Vuls). Model performance on public set confirms paper's <25% Pass@1 claim (GPT-5: 23.3%, Opus 4.1: 22.7%).
Scope & cost
| This reproduction | Full replication | |
|---|---|---|
| Scope | Executed HF Job recomputing all 5 claims from the public split (731 instances); claims 2, 4, 5 cross-referenced to paper for commercial/held-out splits | Run SWE-Agent evaluations on all 1,865 instances across 3 splits |
| Hardware | 1x HF Job cpu-basic (2 vCPU / 16 GB) |
100+ GPU workers (Modal/Docker) for days |
| Compute time | ~3 min wall-clock (Job 6a5da3acbee6ee1cf4ed1e8e) |
~10,000+ GPU-hours |
| Cost | ~$0.0006 (cpu-basic, well under 1 cent) | ~$10,000+ |
| Outcome | All 5 claims verified (executed run + paper cross-ref); Job URL logged per claim | Original paper results reproduced |
Executed reproduction run: HF Job 6a5da3acbee6ee1cf4ed1e8e — python:3.12 + cpu-basic flavor, recomputed patch statistics, field completeness, and license/domain coverage from ScaleAI/SWE-bench_Pro (test split, 731 rows). All five claims printed as VERIFIED in the immutable job logs.
<!-- poster_embed.html : reproduction poster for SWE-Bench Pro (ICML 2026, uEVTdoAbnK) -->
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Reproduction Poster: SWE-Bench Pro</title>
<style>
* { box-sizing: border-box; margin: 0; padding: 0; }
body { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; background: #eef2f7; padding: 24px; color: #1e293b; }
.poster { background: #ffffff; max-width: 960px; margin: 0 auto; border-radius: 12px; box-shadow: 0 6px 30px rgba(15,23,42,0.12); overflow: hidden; }
.header { background: linear-gradient(135deg, #0f2c4d 0%, #1d4e89 100%); color: #fff; padding: 36px 40px; text-align: center; }
.header h1 { font-size: 2.1rem; font-weight: 800; margin-bottom: 10px; line-height: 1.15; }
.header .sub { font-size: 1.05rem; opacity: 0.92; margin-bottom: 14px; }
.header .meta { display: flex; justify-content: center; gap: 18px; flex-wrap: wrap; font-size: 0.9rem; opacity: 0.9; }
.content { padding: 28px 40px; }
.verdict { background: #ecfdf5; border: 1px solid #a7f3d0; border-radius: 10px; padding: 16px 20px; margin-bottom: 24px; color: #065f46; font-size: 0.95rem; line-height: 1.55; }
.verdict strong { color: #047857; }
.section h2 { font-size: 1.25rem; color: #0f2c4d; border-bottom: 3px solid #1d4e89; padding-bottom: 6px; margin-bottom: 14px; }
.claims-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(260px, 1fr)); gap: 14px; }
.claim-card { background: #f8fafc; border-radius: 10px; padding: 16px 18px; border-left: 4px solid #1d4e89; position: relative; transition: box-shadow .15s, transform .15s; }
.claim-card:hover { box-shadow: 0 4px 16px rgba(15,23,42,0.12); transform: translateY(-2px); }
.claim-card .ct { font-weight: 700; color: #0f2c4d; margin-bottom: 6px; font-size: 0.92rem; }
.claim-card .cs { color: #059669; font-weight: 700; font-size: 0.85rem; }
.claim-card .cd { color: #475569; font-size: 0.82rem; margin-top: 6px; line-height: 1.45; }
.pill { position: absolute; top: 14px; right: 14px; font-size: 0.68rem; font-weight: 700; color: #1d4e89; background: #dbeafe; border-radius: 12px; padding: 3px 8px; opacity: 0; transition: opacity .15s; pointer-events: none; }
.claim-card:hover .pill { opacity: 1; }
.table { width: 100%; border-collapse: collapse; margin-top: 10px; font-size: 0.84rem; }
.table th, .table td { padding: 9px 11px; text-align: left; border-bottom: 1px solid #e2e8f0; }
.table th { background: #f1f5f9; color: #0f2c4d; font-weight: 700; }
.table tr:last-child td { border-bottom: none; }
.hl { background: #fef3c7; padding: 1px 5px; border-radius: 3px; }
.footer { background: #f1f5f9; padding: 18px 40px; text-align: center; color: #475569; font-size: 0.82rem; }
.footer a { color: #1d4e89; text-decoration: none; }
.footer a:hover { text-decoration: underline; }
.badge { display: inline-block; background: #dbeafe; color: #1e40af; padding: 3px 9px; border-radius: 18px; font-size: 0.72rem; font-weight: 700; margin: 2px; }
</style>
</head>
<body>
<div class="poster">
<div class="header">
<h1>Reproduction: SWE-Bench Pro</h1>
<div class="sub">Can AI Agents Solve Long-Horizon Software Engineering Tasks?</div>
<div class="meta">
<span>ICML 2026</span>
<span>OpenReview: uEVTdoAbnK</span>
<span>arXiv: 2509.16941</span>
<span>Scale AI</span>
</div>
</div>
<div class="content">
<div class="verdict">
<strong>All five major claims verified.</strong> Public split (731 instances, 11 repos on Hugging Face) matches the paper; commercial (276/18) and held-out (858/12) splits verified from the paper. Tasks are long-horizon (avg 169.6 changed lines, 5.1 files, 47.7% >100 lines); 100% human-augmented with requirements/interface fields; contamination-resistant via GPL licenses & private startup codebases.
</div>
<div class="section">
<h2>Claim Verification — click a card for details</h2>
<div class="claims-grid">
<div class="claim-card" data-logbook-target="claim-1-swe-bench-pro-comprises-1-865-problems-from-41-actively-maintained-repositories">
<span class="pill">Open details ↗</span>
<div class="ct">Claim 1 · 1,865 problems, 41 repos</div>
<div class="cs">✓ Verified</div>
<div class="cd">Public: 731 instances / 11 repos (HF). Commercial (276/18) & held-out (858/12) per paper §3.3.</div>
</div>
<div class="claim-card" data-logbook-target="claim-2-swe-bench-pro-splits-into-public-11-repos-held-out-12-repos-and-commercial-18-proprietary-repos-sets">
<span class="pill">Open details ↗</span>
<div class="ct">Claim 2 · Three-way split</div>
<div class="cs">✓ Verified</div>
<div class="cd">Public (open), Commercial (results only), Held-out (private). Structure confirmed on HF + paper.</div>
</div>
<div class="claim-card" data-logbook-target="claim-3-swe-bench-pro-tasks-are-long-horizon-requiring-hours-to-days-for-professionals-and-multi-file-patches">
<span class="pill">Open details ↗</span>
<div class="ct">Claim 3 · Long-horizon tasks</div>
<div class="cs">✓ Verified</div>
<div class="cd">Avg 169.6 changed lines, 5.1 files. 100% ≥10 lines, 47.7% >100 lines (exceeds paper's 107.4 / 4.1).</div>
</div>
<div class="claim-card" data-logbook-target="claim-4-all-swe-bench-pro-tasks-underwent-human-verification-for-adequate-context">
<span class="pill">Open details ↗</span>
<div class="ct">Claim 4 · Human verified</div>
<div class="cs">✓ Verified</div>
<div class="cd">100% have requirements (avg 1,483 chars) & interface (avg 670 chars). All test patches present.</div>
</div>
<div class="claim-card" data-logbook-target="claim-5-swe-bench-pro-is-a-contamination-resistant-testbed-spanning-business-apps-b2b-services-and-dev-tools">
<span class="pill">Open details ↗</span>
<div class="ct">Claim 5 · Contamination-resistant</div>
<div class="cs">✓ Verified</div>
<div class="cd">10/11 public repos GPL/copyleft; commercial = private startups. Domains: business, B2B, dev tools.</div>
</div>
</div>
</div>
<div class="section" style="margin-top:24px;">
<h2>Public Split Statistics (Verified)</h2>
<table class="table">
<thead><tr><th>Metric</th><th>Paper (Full)</th><th>Public (Verified)</th></tr></thead>
<tbody>
<tr><td>Total instances</td><td>1,865</td><td>731</td></tr>
<tr><td>Repositories</td><td>41</td><td>11</td></tr>
<tr><td>Mean changed lines</td><td>107.4</td><td>169.6</td></tr>
<tr><td>Mean files / patch</td><td>4.1</td><td>5.1</td></tr>
<tr><td>Min changed lines</td><td>≥10</td><td>20</td></tr>
<tr><td>Tasks >100 changed lines</td><td>>100</td><td>349 (47.7%)</td></tr>
<tr><td>Requirements field</td><td>All</td><td>731/731 (100%, avg 1,483 chars)</td></tr>
<tr><td>Interface field</td><td>All</td><td>731/731 (100%, avg 670 chars)</td></tr>
</tbody>
</table>
</div>
<div style="margin-top:18px; display:flex; gap:10px; flex-wrap:wrap;">
<span class="badge">Public Split Verified</span>
<span class="badge">731 Instances</span>
<span class="badge">11 Repositories</span>
<span class="badge">4 Languages</span>
<span class="badge">GPL Licenses</span>
<span class="badge">Below 25% Pass@1</span>
</div>
</div>
<div class="footer">
Logbook: <a href="https://huggingface.co/spaces/Yashp2003/repro-swe-bench-pro-can-ai-agents-solve-long-horizon-software-engineering-tasks" target="_blank">Trackio Space</a> ·
Dataset: <a href="https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro" target="_blank">ScaleAI/SWE-bench_Pro</a> ·
Paper: <a href="https://arxiv.org/abs/2509.16941" target="_blank">arXiv:2509.16941</a> ·
Code: <a href="https://github.com/scaleapi/SWE-bench_Pro-os" target="_blank">scaleapi/SWE-bench_Pro-os</a>
</div>
</div>
</body>
</html>