# Claim 2: Agents are tasked with investigating code semantics, localizing bottlenecks and relevant tests, and producing a patch that matches or exceeds expert speedup while passing the same unit tests (abstract only).


---
<!-- trackio-cell
{"type": "markdown", "id": "cell_c2_v1", "created_at": "2026-07-22T14:50:08+00:00", "title": "Verification"}
-->
## Verification

**Verdict: ✅ VERIFIED (full)**

Confirmed by reading the paper (Sections 2.3 and 3) and the official project documentation at [swefficiency.com](https://swefficiency.com).

### Task Formulation (from Section 2.3)

An agent is given:
1. **Complete codebase** — full repository at the base commit
2. **Performance workload** — executable Python script exercising slow code paths

The agent must:
- Investigate code semantics and execution flow
- Localize performance bottlenecks (functions, files)
- Identify relevant unit tests from the repo's own test suite
- Produce a patch (git diff) that speeds up the workload
- Ensure all covering tests still pass

### Evaluation Metric: Speedup Ratio (SR)

SR = Speedup_LM / Speedup_gold

Where Speedup = T_pre / T_post. Harmonic mean is used for aggregation. If tests fail, SR = 1/Speedup_gold (a penalty).

### Agent Scaffolds Evaluated

- **OpenHands (CodeActAgent v0.51.1)** — primary harness, provides bash terminal + file editing tools
- **SWE-agent (v1.1.0)** — secondary harness for cross-validation

Both configured with 3-hour time limit, 30-min step timeout, max 100 actions.

### Toy-Scale Reproduction

To demonstrate the task formulation is reproducible, a toy-scale experiment was conducted on Modal:

- **Model**: Qwen/Qwen3-4B-Instruct-2507 on L4 GPU (24GB)
- **Tasks**: 5 SWE-fficiency instances (from pandas, dask, numpy)
- **Result**: 5/5 file localization, 5/5 diff-format output
- **Cost**: ~$0.03, 192.7 seconds total
- **Caveat**: Gold patch was included in prompt (reference task), so this tests re-statement ability, not novel generation

Modal run: [ap-60SOIgcBeHQx2VJU4jnEpk](https://modal.com/apps/ypa7979/main/ap-60SOIgcBeHQx2VJU4jnEpk)

### Source

Paper: [arXiv:2511.06090 Sections 2-3](https://arxiv.org/abs/2511.06090)