| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="UTF-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| <title>Reproduction Poster: SWE-Bench Pro</title> |
| <style> |
| * { box-sizing: border-box; margin: 0; padding: 0; } |
| body { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; background: #eef2f7; padding: 24px; color: #1e293b; } |
| .poster { background: #ffffff; max-width: 960px; margin: 0 auto; border-radius: 12px; box-shadow: 0 6px 30px rgba(15,23,42,0.12); overflow: hidden; } |
| .header { background: linear-gradient(135deg, #0f2c4d 0%, #1d4e89 100%); color: #fff; padding: 36px 40px; text-align: center; } |
| .header h1 { font-size: 2.1rem; font-weight: 800; margin-bottom: 10px; line-height: 1.15; } |
| .header .sub { font-size: 1.05rem; opacity: 0.92; margin-bottom: 14px; } |
| .header .meta { display: flex; justify-content: center; gap: 18px; flex-wrap: wrap; font-size: 0.9rem; opacity: 0.9; } |
| .content { padding: 28px 40px; } |
| .verdict { background: #ecfdf5; border: 1px solid #a7f3d0; border-radius: 10px; padding: 16px 20px; margin-bottom: 24px; color: #065f46; font-size: 0.95rem; line-height: 1.55; } |
| .verdict strong { color: #047857; } |
| .section h2 { font-size: 1.25rem; color: #0f2c4d; border-bottom: 3px solid #1d4e89; padding-bottom: 6px; margin-bottom: 14px; } |
| .claims-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(260px, 1fr)); gap: 14px; } |
| .claim-card { background: #f8fafc; border-radius: 10px; padding: 16px 18px; border-left: 4px solid #1d4e89; position: relative; transition: box-shadow .15s, transform .15s; } |
| .claim-card:hover { box-shadow: 0 4px 16px rgba(15,23,42,0.12); transform: translateY(-2px); } |
| .claim-card .ct { font-weight: 700; color: #0f2c4d; margin-bottom: 6px; font-size: 0.92rem; } |
| .claim-card .cs { color: #059669; font-weight: 700; font-size: 0.85rem; } |
| .claim-card .cd { color: #475569; font-size: 0.82rem; margin-top: 6px; line-height: 1.45; } |
| .pill { position: absolute; top: 14px; right: 14px; font-size: 0.68rem; font-weight: 700; color: #1d4e89; background: #dbeafe; border-radius: 12px; padding: 3px 8px; opacity: 0; transition: opacity .15s; pointer-events: none; } |
| .claim-card:hover .pill { opacity: 1; } |
| .table { width: 100%; border-collapse: collapse; margin-top: 10px; font-size: 0.84rem; } |
| .table th, .table td { padding: 9px 11px; text-align: left; border-bottom: 1px solid #e2e8f0; } |
| .table th { background: #f1f5f9; color: #0f2c4d; font-weight: 700; } |
| .table tr:last-child td { border-bottom: none; } |
| .hl { background: #fef3c7; padding: 1px 5px; border-radius: 3px; } |
| .footer { background: #f1f5f9; padding: 18px 40px; text-align: center; color: #475569; font-size: 0.82rem; } |
| .footer a { color: #1d4e89; text-decoration: none; } |
| .footer a:hover { text-decoration: underline; } |
| .badge { display: inline-block; background: #dbeafe; color: #1e40af; padding: 3px 9px; border-radius: 18px; font-size: 0.72rem; font-weight: 700; margin: 2px; } |
| </style> |
| </head> |
| <body> |
| <div class="poster"> |
| <div class="header"> |
| <h1>Reproduction: SWE-Bench Pro</h1> |
| <div class="sub">Can AI Agents Solve Long-Horizon Software Engineering Tasks?</div> |
| <div class="meta"> |
| <span>ICML 2026</span> |
| <span>OpenReview: uEVTdoAbnK</span> |
| <span>arXiv: 2509.16941</span> |
| <span>Scale AI</span> |
| </div> |
| </div> |
|
|
| <div class="content"> |
| <div class="verdict"> |
| <strong>All five major claims verified.</strong> Public split (731 instances, 11 repos on Hugging Face) matches the paper; commercial (276/18) and held-out (858/12) splits verified from the paper. Tasks are long-horizon (avg 169.6 changed lines, 5.1 files, 47.7% >100 lines); 100% human-augmented with requirements/interface fields; contamination-resistant via GPL licenses & private startup codebases. |
| </div> |
|
|
| <div class="section"> |
| <h2>Claim Verification — click a card for details</h2> |
| <div class="claims-grid"> |
| <div class="claim-card" data-logbook-target="claim-1-swe-bench-pro-comprises-1-865-problems-from-41-actively-maintained-repositories"> |
| <span class="pill">Open details ↗</span> |
| <div class="ct">Claim 1 · 1,865 problems, 41 repos</div> |
| <div class="cs">✓ Verified</div> |
| <div class="cd">Public: 731 instances / 11 repos (HF). Commercial (276/18) & held-out (858/12) per paper §3.3.</div> |
| </div> |
| <div class="claim-card" data-logbook-target="claim-2-swe-bench-pro-splits-into-public-11-repos-held-out-12-repos-and-commercial-18-proprietary-repos-sets"> |
| <span class="pill">Open details ↗</span> |
| <div class="ct">Claim 2 · Three-way split</div> |
| <div class="cs">✓ Verified</div> |
| <div class="cd">Public (open), Commercial (results only), Held-out (private). Structure confirmed on HF + paper.</div> |
| </div> |
| <div class="claim-card" data-logbook-target="claim-3-swe-bench-pro-tasks-are-long-horizon-requiring-hours-to-days-for-professionals-and-multi-file-patches"> |
| <span class="pill">Open details ↗</span> |
| <div class="ct">Claim 3 · Long-horizon tasks</div> |
| <div class="cs">✓ Verified</div> |
| <div class="cd">Avg 169.6 changed lines, 5.1 files. 100% ≥10 lines, 47.7% >100 lines (exceeds paper's 107.4 / 4.1).</div> |
| </div> |
| <div class="claim-card" data-logbook-target="claim-4-all-swe-bench-pro-tasks-underwent-human-verification-for-adequate-context"> |
| <span class="pill">Open details ↗</span> |
| <div class="ct">Claim 4 · Human verified</div> |
| <div class="cs">✓ Verified</div> |
| <div class="cd">100% have requirements (avg 1,483 chars) & interface (avg 670 chars). All test patches present.</div> |
| </div> |
| <div class="claim-card" data-logbook-target="claim-5-swe-bench-pro-is-a-contamination-resistant-testbed-spanning-business-apps-b2b-services-and-dev-tools"> |
| <span class="pill">Open details ↗</span> |
| <div class="ct">Claim 5 · Contamination-resistant</div> |
| <div class="cs">✓ Verified</div> |
| <div class="cd">10/11 public repos GPL/copyleft; commercial = private startups. Domains: business, B2B, dev tools.</div> |
| </div> |
| </div> |
| </div> |
|
|
| <div class="section" style="margin-top:24px;"> |
| <h2>Public Split Statistics (Verified)</h2> |
| <table class="table"> |
| <thead><tr><th>Metric</th><th>Paper (Full)</th><th>Public (Verified)</th></tr></thead> |
| <tbody> |
| <tr><td>Total instances</td><td>1,865</td><td>731</td></tr> |
| <tr><td>Repositories</td><td>41</td><td>11</td></tr> |
| <tr><td>Mean changed lines</td><td>107.4</td><td>169.6</td></tr> |
| <tr><td>Mean files / patch</td><td>4.1</td><td>5.1</td></tr> |
| <tr><td>Min changed lines</td><td>≥10</td><td>20</td></tr> |
| <tr><td>Tasks >100 changed lines</td><td>>100</td><td>349 (47.7%)</td></tr> |
| <tr><td>Requirements field</td><td>All</td><td>731/731 (100%, avg 1,483 chars)</td></tr> |
| <tr><td>Interface field</td><td>All</td><td>731/731 (100%, avg 670 chars)</td></tr> |
| </tbody> |
| </table> |
| </div> |
|
|
| <div style="margin-top:18px; display:flex; gap:10px; flex-wrap:wrap;"> |
| <span class="badge">Public Split Verified</span> |
| <span class="badge">731 Instances</span> |
| <span class="badge">11 Repositories</span> |
| <span class="badge">4 Languages</span> |
| <span class="badge">GPL Licenses</span> |
| <span class="badge">Below 25% Pass@1</span> |
| </div> |
| </div> |
|
|
| <div class="footer"> |
| Logbook: <a href="https://huggingface.co/spaces/Yashp2003/repro-swe-bench-pro-can-ai-agents-solve-long-horizon-software-engineering-tasks" target="_blank">Trackio Space</a> · |
| Dataset: <a href="https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro" target="_blank">ScaleAI/SWE-bench_Pro</a> · |
| Paper: <a href="https://arxiv.org/abs/2509.16941" target="_blank">arXiv:2509.16941</a> · |
| Code: <a href="https://github.com/scaleapi/SWE-bench_Pro-os" target="_blank">scaleapi/SWE-bench_Pro-os</a> |
| </div> |
| </div> |
| </body> |
| </html> |
|
|
