pages/index.md

Reproduction: SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

HF paper page

Pages

Page
Executive summary
Claim 1: SWE-Bench Pro comprises 1,865 problems sourced from 41 actively maintained software repositories (abstract only).
Claim 2: SWE-Bench Pro splits into public (11 repos), held-out (12 repos), and commercial (18 proprietary repos) sets (abstract only).
Claim 3: Tasks are long-horizon, potentially requiring hours to days for professional engineers and often spanning multiple files (abstract only).
Claim 4: All SWE-Bench Pro tasks underwent human verification for adequate resolution context (abstract only).
Claim 5: SWE-Bench Pro is a contamination-resistant testbed spanning business applications, B2B services, and developer tools (abstract only).
Conclusion
Deep Verification: Contamination Resistance, Agent Solve Rate, and Eval Repo Evidence