pages/index.md

Reproduction: SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

HF paper page

Pages

Page
Executive summary
Claim 1: SWE-Bench Pro comprises 1,865 problems from 41 actively maintained repositories.
Claim 2: SWE-Bench Pro splits into public (11 repos), held-out (12 repos), and commercial (18 proprietary repos) sets.
Claim 3: SWE-Bench Pro tasks are long-horizon, requiring hours to days for professionals and multi-file patches.
Claim 4: All SWE-Bench Pro tasks underwent human verification for adequate context.
Claim 5: SWE-Bench Pro is a contamination-resistant testbed spanning business apps, B2B services, and dev tools.
Conclusion