Upload docs/deep_literature_review.md

docs/deep_literature_review.md

@@ -0,0 +1,91 @@
+# Agent Cost Optimization: Deep Literature Review (2024–2026)
+
+**Date: 2026-06-03**  
+**Part of: [agent-cost-optimizer](https://huggingface.co/narcolepticchicken/agent-cost-optimizer)**
+
+## Summary Table (Ranked by Impact)
+
+| Rank | Paper | Method | Key Result | Applies to Agents? |
+|------|-------|--------|-----------|---------------------|
+| 1 | In-Context Distillation + Self-Consistency Cascades (2512.02543) | ICL retrieval + student voting + teacher fallback | **2.5-3.5× cost reduction** at iso-quality | ✅ Directly (ALFWorld, AppWorld) |
+| 2 | Make Every Penny Count: DSC (2408.13457) | Difficulty-adaptive sample count | **65.29% cost reduction** vs standard SC | ⚠️ Single-turn reasoning |
+| 3 | BAVT: Budget-Aware Value Tree Search (2603.12634) | Residual value critic + budget-annealed search | Matches baseline at **4× budget** | ✅ Multi-hop tool-calling |
+| 4 | Efficient Agents (2508.02694) | Systematic ablation across agent components | **42.7% cost reduction**, 96.7% quality | ✅ Directly (GAIA) |
+| 5 | Agentic Plan Caching (2506.14852) | Structured plan reuse + template adaptation | **50.31% cost reduction**, 27% latency | ✅ Multi-step Plan-Act |
+| 6 | SpecEyes (2603.23483) | Speculative perception for agentic MLLMs | **1.1-3.35× speedup**, +6.7% accuracy | ⚠️ Vision agents only |
+| 7 | R2-Router (2602.02823) | Joint model + length budget selection | **4-5× cheaper** than existing routers | ⚠️ Single-turn, length budget applicable |
+| 8 | RouteLLM (2405.03860) | BERT routers beat LLM routers on cost | **2× cost reduction**, $0.8/hr vs $5.23/hr | ✅ Model routing |
+
+## Ablation Findings from Efficient Agents (2508.02694)
+
+**The only systematic ablation across ALL agent cost components:**
+
+| Component | Finding | Cost Impact |
+|-----------|---------|-------------|
+| Backbone selection | Claude 3.7 costs **3.6×** more than GPT-4.1, **27×** more than Qwen3-30B-A3B | Dominates everything |
+| Best-of-N (N=4 vs N=1) | +0.61% accuracy for +34% tokens | **Waste — don't use** |
+| Max steps (12 vs 8) | +0.6% accuracy for +27% cost | 8 is sweet spot |
+| Complex memory | No improvement on GAIA | Simple memory is optimal |
+| Multi-search tools | Helps only at L2-L3 | Gating needed |
+| Planning interval | Interval=1 is optimal | No optimization opportunity |
+
+**Optimal config:** GPT-4.1 backbone, 8 max steps, plan interval=1, BoN=1, simple memory.
+
+## Prompt Caching Discounts
+
+| Provider | Discount | Mechanism | Cache Lifetime |
+|----------|----------|-----------|----------------|
+| Anthropic | **90%** on reads | Explicit breakpoints | 5 min TTL |
+| OpenAI | **50%** automatic | 1024+ token prefix match | 5-60 min |
+| Google Gemini | **75%** on cached input | Explicit context caching API | Configurable |
+| DeepSeek | Unknown | — | — |
+
+**Critical:** Use the Hub kernels library (`kernels-community/flash-attn2`) instead of building flash-attn from source. Building takes minutes to hours.
+
+## Router Architectures (2025–2026)
+
+| Router | Approach | Innovation | Cost Claim |
+|--------|----------|-----------|------------|
+| Router-R1 (2506.09033) | RL-trained LLM, multi-round | Unseen model generalization | — |
+| R2-Router (2602.02823) | Joint model + length budget | "Powerful+constrained > weak" | 4-5× cheaper |
+| CARROT (2502.03261) | Optimal routing theory | Cost-rate-optimal proofs | — |
+| xRouter (2510.08439) | RL tool-calling router | Cost-aware reward function | — |
+| BaRP (2510.07429) | Online bandit routing | Partial feedback learning | Outperforms offline |
+| **RouteLLM** (2405.03860) | **BERT classifier** | **$0.8/hr deployment** | **2× cost reduction** |
+
+## Token-Level Cost Prediction
+
+| Method | Approach | Key Result |
+|--------|----------|------------|
+| LenVM (2604.27039) | Token-level value model | 63% at 200-token vs 6% baseline |
+| Budget Guidance (2506.13752) | Gamma distribution predictor | +26% accuracy under tight budgets |
+| TALE (2412.18547) | Binary search budget | Token elasticity — cutoff too far = costlier |
+| BudgetThinker (2508.17196) | Control tokens + SFT/RL | Budget-conditioned generation |
+
+## Speculative Agent Actions
+
+| Paper | Method | Result |
+|-------|--------|--------|
+| ForeAgent (2601.05930) | Predict ML code execution | 6× convergence, +6% over execution-only |
+| AWO/Meta-tools (2601.22037) | Recurring tool sequences → meta-tools | 11.9% fewer LLM calls |
+| SpecEyes (2603.23483) | Cognitive gating via answer separability | Key insight transferable to text agents |
+
+## What's Missing from Literature
+
+1. **No speculative decoding for tool-calling agents** — draft-then-verify is proven for text, never applied to tool calls. Genuine research gap.
+2. **No real-time cost monitoring with action** — all existing work is static/offline optimization.
+3. **No combined routing + compression + caching optimizer** — each technique studied in isolation.
+4. **No provider-aware cache routing** — which provider to use based on cache hit probability.
+5. **No learned cost-quality frontiers per task type** — all routing is task-agnostic.
+
+## Implementation Priorities for ACO
+
+1. **Self-consistency cascade** (training-free, 2.5-3.5× reduction, best ROI) — IMPLEMENTED: `aco/self_consistency_cascade.py`
+2. **Trained BERT routers** (ModernBERT, focal loss, threshold calibration) — TRAINING: v2 specialist models
+3. **Cache-aware prompt layout** (90% Anthropic discount, zero risk) — IMPLEMENTED: `aco/cache_layout.py`
+4. **Context compression** (ACON pattern, improves success rate) — IMPLEMENTED: `aco/context_compression.py`
+5. **Tool-use gating** (ModernBERT binary, 93% of ToolACE doesn't need tools) — TRAINING: v2 tool gater
+6. **Plan caching** (50% cost reduction, requires integration) — PENDING
+7. **Verifier gating with Intervention Paradox protection** — TRAINING: v2 verifier gater
+8. **Budget-aware value tree search** (for multi-hop agents) — PENDING
+9. **Token-length prediction** (LenVM-style, stop early) — PENDING