# Reproduction: SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?

## Pages

| Page |
| --- |
| [Executive summary](#/executive-summary) |
| [Claim 1: The SWE-fficiency benchmark comprises 498 real-world code optimization tasks drawn from 9 widely used Python repositories including numpy, pandas, and scipy (abstract only).](#/claim-1-the-swe-fficiency-benchmark-comprises-498-real-world-code-optimization-tasks-drawn-from-9-widely-used-python-repositories-including-numpy-pandas-and-scipy-abstract-only) |
| [Claim 2: Agents are tasked with investigating code semantics, localizing bottlenecks and relevant tests, and producing a patch that matches or exceeds expert speedup while passing the same unit tests (abstract only).](#/claim-2-agents-are-tasked-with-investigating-code-semantics-localizing-bottlenecks-and-relevant-tests-and-producing-a-patch-that-matches-or-exceeds-expert-speedup-while-passing-the-same-unit-tests-abstract-only) |
| [Claim 3: The top-performing agents evaluated on SWE-fficiency achieve less than 0.23x of the expert-level speedup on average across the 498 tasks (abstract only).](#/claim-3-the-top-performing-agents-evaluated-on-swe-fficiency-achieve-less-than-0-23x-of-the-expert-level-speedup-on-average-across-the-498-tasks-abstract-only) |
| [Claim 4: Agents struggle specifically with localizing optimization opportunities in large codebases, reasoning about execution flow across multiple functions, and maintaining correctness of proposed optimizations (abstract only).](#/claim-4-agents-struggle-specifically-with-localizing-optimization-opportunities-in-large-codebases-reasoning-about-execution-flow-across-multiple-functions-and-maintaining-correctness-of-proposed-optimizations-abstract-only) |
| [Conclusion](#/conclusion) |