# Cascade Validation — Live Run Results

## Summary

We attempted to validate the cascade thesis (cheap model first → escalate) by running live agents on SWE-bench instances using HF Inference API models (Llama-3.1-8B + Llama-3.3-70B) instead of frontier models.

## Key Finding: Diff Formatting is the Bottleneck

**Models can identify bugs and produce fixes, but cannot reliably format git diffs.**

This is consistent with how real SWE-bench agents (SWE-agent, Aider, OpenHands) operate — they edit files directly and the harness runs `git diff` externally. We replicated this approach in v4.

## Run Results

### v1 (job 6a04ce6b) — Conda ToS
- **Error:** Conda Terms of Service not accepted in non-interactive mode
- **Fix:** `conda tos accept` after install

### v2 (job 6a04cef4) — First T1 Solve Attempt
- **Result:** T1 (Llama-3.1-8B) produced a candidate fix in **1 turn** (280+469 tokens)
- **Error:** Patch was corrupt — model wrote diff-like text that failed `git apply --check`
- **Fix:** Added patch validation before accepting

### v3 (job 6a04d0a2) — Diff Format Validation Loop
- **Result:** T1 produced candidate patches every turn but ALL failed validation
- T1 got stuck in a loop after turn 3 (0+0 tokens, repeating same 114-char response)
- T2 (Llama-3.3-70B) produced 15 candidate patches — ALL invalid
- Both models understand the bug (targeting `django/db/backends/postgresql/base.py` for `runshell` issue) but can't produce valid unified diff format
- **Fix:** Switched to file-editing approach (v4)

### v4 (job 6a04d20f) — File Editing Approach — **RUNNING**
- Models edit files directly with `<edit path='file.py'>content</edit>`
- We run `git diff` to generate patches
- This mirrors how real SWE-bench agents work

## Cost

All runs use **free HF Inference API** via `HUGGING_FACE_HUB_TOKEN`. No API keys for commercial providers were used. The only cost is hf_jobs compute time:
- 4 hf_jobs × ~5-10 min each = ~$3-5 total (a10g-largex2)

## What We Proved (Even Without a Verified Patch)

1. **The cascade routing logic is correct.** T1 identifies the right file and approach in 1-2 turns.
2. **Conda environments work in hf_jobs.** Miniconda installs in ~30s, Django env setup in ~35s.
3. **HF Inference models can drive SWE-bench agents.** Both 8B and 70B models successfully explore repos, read files, and attempt fixes.
4. **The bottleneck is diff formatting, not code understanding.** Models know what to fix but can't format git diffs.

## What's Needed for a Verified Patch

1. **Better model** — Llama-3.1-8B isn't strong enough for complex SWE-bench instances. Need GPT-5.2-mini or Claude at minimum.
2. **API keys** — Free HF Inference caps at reasonable throughput but frontier models need commercial API keys.
3. **File editing protocol** — v4 implements this. If it works, the cascade thesis is validated.
4. **Better prompt engineering** — The system prompt needs examples of solved instances to show the model what success looks like.