# SWE-bench Cascade: All 5 Fixes — Final Summary

**Date:** 2026-05-11

## What We Did

Five corrections to the original cascade analysis, plus a live agent build.

## Fix 1: Fair Baseline — DONE

**Original:** Cascade 416 solves vs Frontier-single 391 solves (+25). Cascade cheaper ($76.48 vs $158.34).

**Corrected:** Frontier-with-retry (Claude → gpt-5.2-medium) solves 420, cascade solves 416. Frontier retry wins on quality by 4 instances, but costs 2.3x more. The solve difference is NOT statistically significant (-0.9pp, 95% CI [-2.8, +1.0]). Cascade is significantly cheaper (-$0.22/inst, 95% CI [-$0.26, -$0.19]).

**Verdict:** Cascade dominates on cost-efficiency. Same quality within noise, 56% cheaper.

## Fix 2: Patch Inspection — INCONCLUSIVE

The 5 cascade-only instances (T2 solves, no T4 model does) span diverse repos and have moderate-complexity gold patches. This is consistent with genuine model diversity, not benchmark noise. But SWE-Router datasets lack extracted patches — we can't compare T2 patches to T4 patches without Docker execution. **Needs Docker to complete.**

## Fix 3: Provider Routing Model — FIXED

The $585 figure was a unit error. Corrected: provider routing saves $18.16 (Bedrock vs Anthropic direct for T4 calls). Cascade with Bedrock: $68.17 vs $86.33 base. Real but secondary.

## Fix 4: Live Docker Agent — PARTIAL

Built a Docker-less cascade agent that clones repos, sets up environments, runs T1 (Llama-3.1-8B) → T2 (Llama-3.3-70B), and produces patches. On 10 astropy instances: 10/10 produced patches, 9 at T1, 1 at T2. Verification fails because `git apply` needs the exact `/testbed` path structure from Docker images. Full Docker agent code is ready; needs Docker daemon to run.

**To run on Docker:** `python cascade_agent_docker.py --batch 50 --strategy cascade` on a host with Docker daemon, Anaconda, and API keys.

## Fix 5: Paired Significance — DONE

1000 bootstrap resamples, 95% CI, per-repo breakdown:
- Cascade solve rate CI: [80.0%, 86.2%]
- Frontier-retry solve rate CI: [80.6%, 87.2%]
- Overlap: YES — no significant quality difference
- Cost difference: significant at p < 0.025

## What The Data Actually Says

| Metric | Cascade | Frontier-retry | Difference |
|---|---|---|---|
| Solved | 416/500 (83.2%) | 420/500 (84.0%) | -4 (not sig) |
| Cost | $86.33 | $196.77 | -56.1% (significant) |
| $/solved | $0.2075 | $0.4685 | -55.7% |
| $/solved (Bedrock) | $0.1638 | — | — |

## Next Steps

1. Run Docker agent on 50 instances (needs Docker daemon + API keys)
2. Manually inspect the 5 cascade-only T2 patches
3. Test frontier-with-equal-retries in live agent
4. Add gpt-5.2-medium as tier in cascade