Upload FIXES_COMPLETE.md

FIXES_COMPLETE.md

@@ -0,0 +1,53 @@
+# SWE-bench Cascade: All 5 Fixes — Final Summary
+
+**Date:** 2026-05-11
+
+## What We Did
+
+Five corrections to the original cascade analysis, plus a live agent build.
+
+## Fix 1: Fair Baseline — DONE
+
+**Original:** Cascade 416 solves vs Frontier-single 391 solves (+25). Cascade cheaper ($76.48 vs $158.34).
+
+**Corrected:** Frontier-with-retry (Claude → gpt-5.2-medium) solves 420, cascade solves 416. Frontier retry wins on quality by 4 instances, but costs 2.3x more. The solve difference is NOT statistically significant (-0.9pp, 95% CI [-2.8, +1.0]). Cascade is significantly cheaper (-$0.22/inst, 95% CI [-$0.26, -$0.19]).
+
+**Verdict:** Cascade dominates on cost-efficiency. Same quality within noise, 56% cheaper.
+
+## Fix 2: Patch Inspection — INCONCLUSIVE
+
+The 5 cascade-only instances (T2 solves, no T4 model does) span diverse repos and have moderate-complexity gold patches. This is consistent with genuine model diversity, not benchmark noise. But SWE-Router datasets lack extracted patches — we can't compare T2 patches to T4 patches without Docker execution. **Needs Docker to complete.**
+
+## Fix 3: Provider Routing Model — FIXED
+
+The $585 figure was a unit error. Corrected: provider routing saves $18.16 (Bedrock vs Anthropic direct for T4 calls). Cascade with Bedrock: $68.17 vs $86.33 base. Real but secondary.
+
+## Fix 4: Live Docker Agent — PARTIAL
+
+Built a Docker-less cascade agent that clones repos, sets up environments, runs T1 (Llama-3.1-8B) → T2 (Llama-3.3-70B), and produces patches. On 10 astropy instances: 10/10 produced patches, 9 at T1, 1 at T2. Verification fails because `git apply` needs the exact `/testbed` path structure from Docker images. Full Docker agent code is ready; needs Docker daemon to run.
+
+**To run on Docker:** `python cascade_agent_docker.py --batch 50 --strategy cascade` on a host with Docker daemon, Anaconda, and API keys.
+
+## Fix 5: Paired Significance — DONE
+
+1000 bootstrap resamples, 95% CI, per-repo breakdown:
+- Cascade solve rate CI: [80.0%, 86.2%]
+- Frontier-retry solve rate CI: [80.6%, 87.2%]
+- Overlap: YES — no significant quality difference
+- Cost difference: significant at p < 0.025
+
+## What The Data Actually Says
+
+| Metric | Cascade | Frontier-retry | Difference |
+|---|---|---|---|
+| Solved | 416/500 (83.2%) | 420/500 (84.0%) | -4 (not sig) |
+| Cost | $86.33 | $196.77 | -56.1% (significant) |
+| $/solved | $0.2075 | $0.4685 | -55.7% |
+| $/solved (Bedrock) | $0.1638 | — | — |
+
+## Next Steps
+
+1. Run Docker agent on 50 instances (needs Docker daemon + API keys)
+2. Manually inspect the 5 cascade-only T2 patches
+3. Test frontier-with-equal-retries in live agent
+4. Add gpt-5.2-medium as tier in cascade