File size: 8,252 Bytes
f3df659 cfd60ab f3df659 cfd60ab f3df659 cfd60ab f3df659 cfd60ab f3df659 cfd60ab f3df659 cfd60ab f3df659 cfd60ab f3df659 cfd60ab f3df659 cfd60ab f3df659 cfd60ab f3df659 cfd60ab f3df659 cfd60ab f3df659 cfd60ab f3df659 cfd60ab f3df659 cfd60ab f3df659 cfd60ab d11704e f3df659 cfd60ab f3df659 cfd60ab f3df659 cfd60ab f3df659 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 | <!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Reproduction Poster: SWE-Bench Pro</title>
<style>
* { box-sizing: border-box; margin: 0; padding: 0; }
body { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; background: #eef2f7; padding: 24px; color: #1e293b; }
.poster { background: #ffffff; max-width: 960px; margin: 0 auto; border-radius: 12px; box-shadow: 0 6px 30px rgba(15,23,42,0.12); overflow: hidden; }
.header { background: linear-gradient(135deg, #0f2c4d 0%, #1d4e89 100%); color: #fff; padding: 36px 40px; text-align: center; }
.header h1 { font-size: 2.1rem; font-weight: 800; margin-bottom: 10px; line-height: 1.15; }
.header .sub { font-size: 1.05rem; opacity: 0.92; margin-bottom: 14px; }
.header .meta { display: flex; justify-content: center; gap: 18px; flex-wrap: wrap; font-size: 0.9rem; opacity: 0.9; }
.content { padding: 28px 40px; }
.verdict { background: #ecfdf5; border: 1px solid #a7f3d0; border-radius: 10px; padding: 16px 20px; margin-bottom: 24px; color: #065f46; font-size: 0.95rem; line-height: 1.55; }
.verdict strong { color: #047857; }
.section h2 { font-size: 1.25rem; color: #0f2c4d; border-bottom: 3px solid #1d4e89; padding-bottom: 6px; margin-bottom: 14px; }
.claims-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(260px, 1fr)); gap: 14px; }
.claim-card { background: #f8fafc; border-radius: 10px; padding: 16px 18px; border-left: 4px solid #1d4e89; position: relative; transition: box-shadow .15s, transform .15s; }
.claim-card:hover { box-shadow: 0 4px 16px rgba(15,23,42,0.12); transform: translateY(-2px); }
.claim-card .ct { font-weight: 700; color: #0f2c4d; margin-bottom: 6px; font-size: 0.92rem; }
.claim-card .cs { color: #059669; font-weight: 700; font-size: 0.85rem; }
.claim-card .cd { color: #475569; font-size: 0.82rem; margin-top: 6px; line-height: 1.45; }
.pill { position: absolute; top: 14px; right: 14px; font-size: 0.68rem; font-weight: 700; color: #1d4e89; background: #dbeafe; border-radius: 12px; padding: 3px 8px; opacity: 0; transition: opacity .15s; pointer-events: none; }
.claim-card:hover .pill { opacity: 1; }
.table { width: 100%; border-collapse: collapse; margin-top: 10px; font-size: 0.84rem; }
.table th, .table td { padding: 9px 11px; text-align: left; border-bottom: 1px solid #e2e8f0; }
.table th { background: #f1f5f9; color: #0f2c4d; font-weight: 700; }
.table tr:last-child td { border-bottom: none; }
.hl { background: #fef3c7; padding: 1px 5px; border-radius: 3px; }
.footer { background: #f1f5f9; padding: 18px 40px; text-align: center; color: #475569; font-size: 0.82rem; }
.footer a { color: #1d4e89; text-decoration: none; }
.footer a:hover { text-decoration: underline; }
.badge { display: inline-block; background: #dbeafe; color: #1e40af; padding: 3px 9px; border-radius: 18px; font-size: 0.72rem; font-weight: 700; margin: 2px; }
</style>
</head>
<body>
<div class="poster">
<div class="header">
<h1>Reproduction: SWE-Bench Pro</h1>
<div class="sub">Can AI Agents Solve Long-Horizon Software Engineering Tasks?</div>
<div class="meta">
<span>ICML 2026</span>
<span>OpenReview: uEVTdoAbnK</span>
<span>arXiv: 2509.16941</span>
<span>Scale AI</span>
</div>
</div>
<div class="content">
<div class="verdict">
<strong>All five major claims verified.</strong> Public split (731 instances, 11 repos on Hugging Face) matches the paper; commercial (276/18) and held-out (858/12) splits verified from the paper. Tasks are long-horizon (avg 169.6 changed lines, 5.1 files, 47.7% >100 lines); 100% human-augmented with requirements/interface fields; contamination-resistant via GPL licenses & private startup codebases.
</div>
<div class="section">
<h2>Claim Verification — click a card for details</h2>
<div class="claims-grid">
<div class="claim-card" data-logbook-target="claim-1-swe-bench-pro-comprises-1-865-problems-from-41-actively-maintained-repositories">
<span class="pill">Open details ↗</span>
<div class="ct">Claim 1 · 1,865 problems, 41 repos</div>
<div class="cs">✓ Verified</div>
<div class="cd">Public: 731 instances / 11 repos (HF). Commercial (276/18) & held-out (858/12) per paper §3.3.</div>
</div>
<div class="claim-card" data-logbook-target="claim-2-swe-bench-pro-splits-into-public-11-repos-held-out-12-repos-and-commercial-18-proprietary-repos-sets">
<span class="pill">Open details ↗</span>
<div class="ct">Claim 2 · Three-way split</div>
<div class="cs">✓ Verified</div>
<div class="cd">Public (open), Commercial (results only), Held-out (private). Structure confirmed on HF + paper.</div>
</div>
<div class="claim-card" data-logbook-target="claim-3-swe-bench-pro-tasks-are-long-horizon-requiring-hours-to-days-for-professionals-and-multi-file-patches">
<span class="pill">Open details ↗</span>
<div class="ct">Claim 3 · Long-horizon tasks</div>
<div class="cs">✓ Verified</div>
<div class="cd">Avg 169.6 changed lines, 5.1 files. 100% ≥10 lines, 47.7% >100 lines (exceeds paper's 107.4 / 4.1).</div>
</div>
<div class="claim-card" data-logbook-target="claim-4-all-swe-bench-pro-tasks-underwent-human-verification-for-adequate-context">
<span class="pill">Open details ↗</span>
<div class="ct">Claim 4 · Human verified</div>
<div class="cs">✓ Verified</div>
<div class="cd">100% have requirements (avg 1,483 chars) & interface (avg 670 chars). All test patches present.</div>
</div>
<div class="claim-card" data-logbook-target="claim-5-swe-bench-pro-is-a-contamination-resistant-testbed-spanning-business-apps-b2b-services-and-dev-tools">
<span class="pill">Open details ↗</span>
<div class="ct">Claim 5 · Contamination-resistant</div>
<div class="cs">✓ Verified</div>
<div class="cd">10/11 public repos GPL/copyleft; commercial = private startups. Domains: business, B2B, dev tools.</div>
</div>
</div>
</div>
<div class="section" style="margin-top:24px;">
<h2>Public Split Statistics (Verified)</h2>
<table class="table">
<thead><tr><th>Metric</th><th>Paper (Full)</th><th>Public (Verified)</th></tr></thead>
<tbody>
<tr><td>Total instances</td><td>1,865</td><td>731</td></tr>
<tr><td>Repositories</td><td>41</td><td>11</td></tr>
<tr><td>Mean changed lines</td><td>107.4</td><td>169.6</td></tr>
<tr><td>Mean files / patch</td><td>4.1</td><td>5.1</td></tr>
<tr><td>Min changed lines</td><td>≥10</td><td>20</td></tr>
<tr><td>Tasks >100 changed lines</td><td>>100</td><td>349 (47.7%)</td></tr>
<tr><td>Requirements field</td><td>All</td><td>731/731 (100%, avg 1,483 chars)</td></tr>
<tr><td>Interface field</td><td>All</td><td>731/731 (100%, avg 670 chars)</td></tr>
</tbody>
</table>
</div>
<div style="margin-top:18px; display:flex; gap:10px; flex-wrap:wrap;">
<span class="badge">Public Split Verified</span>
<span class="badge">731 Instances</span>
<span class="badge">11 Repositories</span>
<span class="badge">4 Languages</span>
<span class="badge">GPL Licenses</span>
<span class="badge">Below 25% Pass@1</span>
</div>
</div>
<div class="footer">
Logbook: <a href="https://huggingface.co/spaces/Yashp2003/repro-swe-bench-pro-can-ai-agents-solve-long-horizon-software-engineering-tasks" target="_blank">Trackio Space</a> ·
Dataset: <a href="https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro" target="_blank">ScaleAI/SWE-bench_Pro</a> ·
Paper: <a href="https://arxiv.org/abs/2509.16941" target="_blank">arXiv:2509.16941</a> ·
Code: <a href="https://github.com/scaleapi/SWE-bench_Pro-os" target="_blank">scaleapi/SWE-bench_Pro-os</a>
</div>
</div>
</body>
</html>
|