Delete router_models/baarbundle.pkl, router_models/productionbundle.pkl, router_models/routerbundle.pkl, router_models/router_bundle_v6.pkl, router_models/router_bundle_v8.pkl, router_models/router_bundle_v10_fixed.pkl, router_models/feat_keys.json, router_models/tier_config.json, eval/baar_results.json, eval/bert_vs_xgboost_results.json/*, docs/trained_router_report.md, aco/router_v10.py, aco/conformal.py, aco/learned_router.py, training/train_router_real.py, training/benchmark_final.py, training/benchmark_with_feedback.py, training/bfcl_eval.py, training/aco_eval.py, FINAL_RESULTS.md

FINAL_RESULTS.md

@@ -1,137 +0,0 @@
-# ACO (Agent Cost Optimizer) — Final Results
-
-## The Cascade Wins. Everything Else Is Seasoning.
-
-After trace analysis, anti-oracle testing, causal simulation, and multi-module benchmarking
-across 500 SWE-bench instances, here is the definitive ranking:
-
-```
-┌─────────────────────────────────────────┬────────┬──────────┬──────────┬──────────────┐
-│ Strategy                                │ Solved │ Cost     │ $/Solved │ Ship?        │
-├─────────────────────────────────────────┼────────┼──────────┼──────────┼──────────────┤
-│ Always Frontier                         │ 391    │ $158.34  │ $0.405   │ baseline     │
-│ Cascade T1→T2→T4                        │ 416    │ $76.48   │ $0.184   │ ✅ PROD      │
-│ Safe Proposal T1→T4                     │ 411    │ $82.10   │ $0.200   │ 🧪 Experimental│
-│ Safe Proposal + T2                      │ 411    │ $71.50   │ $0.174   │ 🧪 Best cost │
-│ Per-Step Oracle (cheating)              │ 411    │ $81.78   │ $0.199   │ ❌ Lookahead │
-└─────────────────────────────────────────┴────────┴──────────┴──────────┴──────────────┘
-```
-
-**The cascade beats frontier on BOTH cost AND quality** — 416 solved vs 391, at 51.7% less cost.
-This is the core result. Safe proposal variants save more money but lose 5 instances
-that T2 would have caught.
-
-## 5 Things We Built and Tested
-
-### 1. Live Cascade Causal Simulation
-- **Job 6a0131f2**: Causal simulation where ACOLiveAgent makes routing decisions at
-  pre_turn(), outcomes looked up from SWE-Router oracle datasets
-- **Result**: 416/500 solved, $76.48, $0.184/solved, 51.7% savings
-- **Key insight**: Cascade solves 25 MORE instances than frontier because T1/T2
-  catch cases Claude misses. Different models have different failure modes.
-- **Tier distribution**: T1=316 (63.2%), T2=43 (8.6%), T4=141 (28.2%)
-- **Cost per API call**: T1=$0.000277, T2=$0.002163, T4=$0.017307 (62.4x ratio)
-
-### 2. Cache-Aware Prompt Layout
-- **Job 6a01332b**: Analyzed 500 traces for cacheable vs dynamic content
-- **Result**: Only 1.5% of tokens are static (135 of 9,066). The system prompt
-  in SWE-bench agents is tiny. Almost everything is in the conversation transcript.
-- **Cache hit rate**: Already 97.6% across turns — the prior conversation IS the cache.
-- **Conclusion**: Cache-aware layout adds negligible value for SWE-bench agents.
-  The prefix caching optimization matters for different workloads (RAG, long system prompts).
-
-### 3. Macro Tool Mining
-- **Job 6a013424**: Extracted 109K 2-command and 147K 3-command sequences
-- **Result**: 6 macro tools identified, 13,578 turns saved, $44.43 saved
-- **Top macros**:
-  - `repo_search`: find + cat (179 occurrences)
-  - `locate_symbol`: grep + cat (565 occurrences)
-  - `run_test_and_summarize`: pytest + grep failures (1,187 occurrences — most valuable)
-  - `read_and_patch`: cat + sed (334 occurrences)
-  - `submit_patch`: git diff + submit (123 occurrences)
-- **220,189 sequences are 100% exploration** — can stay entirely on T1
-- **Implementation**: Each macro replaces 2-3 LLM turns with one deterministic subprocess call
-
-### 4. Doom Rescue Policy
-- **Job 6a0134d4**: Analyzed 1,942 error streaks across 1,000 traces
-- **Result**: **58-72% of "doomed" runs eventually recover and resolve the instance**
-- **Never terminate at 3 errors**: At streak≥3, 72% (T4) and 58% (T1) eventually succeed.
-  Terminating is destroying value, not saving it.
-- **Rescue policy**: One T4 "reset + diagnose" call ($0.025) → expected value over $0.20
-  - Summarize state into clean scratchpad
-  - Classify error: dependency, wrong_file, bad_patch, test_env, repeating
-  - Apply targeted recovery strategy
-  - Terminate only if rescue produces no new plan
-- **Net savings**: $124.88 across 500 instances (recovers 452 solves, costs $18.40 in rescues)
-- **Most fatal errors**: `not_found` (55% fatality), `timeout` (51%), `syntax` (45%)
-  vs least fatal: `permission` (11%), `other` (29%)
-
-### 5. Provider Routing
-- **Job 6a0135e8**: Compared 7 providers across 3 tiers
-- **Per-call costs** (15K input / 2K output):
-  - T1 DeepSeek Direct: $0.00266 (cheapest, 800ms)
-  - T2 OpenAI Direct: $0.00345 live, $0.00255 with cache (600ms)
-  - T4 AWS Bedrock: $0.30 live, $0.17 with cache hit (2,200ms, 20% cheaper than Anthropic)
-- **Batch API**: 50% off on OpenAI/Anthropic for offline workloads
-  - Verifier 411 calls: $64.73 live → $32.37 batch (saves $32.36)
-- **Best provider stack**: DeepSeek (T1) + OpenAI (T2) + AWS Bedrock (T4)
-- **Agent loops**: Live calls only. Latency matters. Use Anthropic Direct if 2.2s is too slow.
-- **Eval/verifier/training**: Use Batch APIs. 50% discount.
-
-## What We Learned (The Hard Way)
-
-1. **ML routing is unnecessary**: The gap between cascade (83.2%) and oracle (86.8%) is 19 instances.
-   Too narrow for ML to add value. XGBoost and BERT both failed to beat static cascade.
-
-2. **Per-step prediction routing doesn't work**: 98.3% false positive rate on edit detection.
-   Can't predict command type from conversation state. Safe proposal model is the fix.
-
-3. **Cache-aware layout adds nothing for SWE-bench**: Only 1.5% static content. Different story
-   for RAG agents with large system prompts.
-
-4. **Never kill a run at 3 errors**: 58-72% recover. Rescue, don't terminate.
-
-5. **Command-type prediction is a red herring**: Don't predict what the model will do.
-   Let it propose, then gate the dangerous proposals.
-
-## What's Production-Ready
-
-```python
-from aco_live import ACOLiveAgent
-
-# Best all-around: Cascade T1→T2→T4
-aco = ACOLiveAgent(strategy='cascade', max_cost=2.0)
-
-# Before each turn:
-decision = aco.pre_turn(messages)
-# → {'model': 'deepseek-v4-flash', 'tier': 1}
-
-response = call_llm(model=decision['model'], messages=...)
-
-# After each turn:
-result = aco.post_turn(response, cost=cost, success=success)
-# → {'action': 'continue'|'escalate'|'review_needed'|'done'}
-```
-
-## Remaining Unknowns
-
-1. **Causal divergence in live agents**: Would T1's exploration discover different bugs than T4's?
-   Can only be tested in actual Docker-execution agent runs.
-2. **Multi-harness validation**: Only tested on SWE-bench coding tasks. Research agents, RAG agents,
-   personal assistants may have different optimal strategies.
-3. **Model version drift**: Models improve. Cascade order needs periodic revalidation.
-4. **Live safe proposal**: T1 proposing + T4 reviewing edits in a real agent loop (not simulation).
-
-## Code Location
-
-- `aco/aco_live.py` — Drop-in ACOLiveAgent wrapper (cascade, safe_proposal, safe_proposal_t2)
-- `aco/per_step_router.py` — Command classifier + per-step router
-- `aco/classifier.py` — Task cost classifier
-- `aco/telemetry.py` — Cost telemetry collector
-- `aco/tool_gate.py` — Tool duplicate detection
-- `aco/doom_detector.py` — Early termination (use rescue policy, not termination!)
-- `aco/verifier_budgeter.py` — Selective verifier calls
-- `aco/retry_optimizer.py` — Error-type-specific recovery
-- `aco/context_budgeter.py` — Context compression
-- `aco/meta_tool_miner.py` — Macro tool extraction
-- `aco/cache_layout.py` — Cache-aware prompt layout

aco/conformal.py

@@ -1,112 +0,0 @@
-"""Conformal calibration for escalation thresholds.
-
-Implements RouteNLP-style conformal risk control:
-  P(failure AND no escalation) <= alpha
-
-Method:
-1. Compute nonconformity scores from calibrated P(success)
-2. Find conformal quantile threshold
-3. Guarantee coverage under exchangeability
-"""
-import numpy as np
-from typing import Dict, List, Optional, Tuple
-
-
-class ConformalEscalationCalibrator:
-    """Calibrate escalation thresholds with distribution-free coverage guarantees.
-    
-    Based on RouteNLP (arxiv 2604.23577) and Conformal Risk Control
-    (Angelopoulos et al., arxiv 2208.02814).
-    
-    Guarantee: P(y=fail AND no_escalation) <= alpha
-    """
-    
-    def __init__(self, alpha: float = 0.05):
-        self.alpha = alpha
-        self.thresholds: Dict[int, float] = {}
-        self.calibrated = False
-    
-    def calibrate(
-        self,
-        psuccess: Dict[int, np.ndarray],
-        outcomes: Dict[int, np.ndarray],
-    ) -> Dict[int, float]:
-        """Calibrate per-tier escalation thresholds.
-        
-        Args:
-            psuccess: {tier: array of calibrated P(success)}
-            outcomes: {tier: array of binary outcomes (1=success, 0=fail)}
-        
-        Returns:
-            {tier: conformal_threshold}
-        """
-        for tier in sorted(psuccess.keys()):
-            p = psuccess[tier]
-            y = outcomes[tier]
-            n = len(y)
-            
-            # Nonconformity: 1 - P(success) for failed examples
-            # These are the scores we want to bound
-            failed_mask = y == 0
-            if failed_mask.sum() == 0:
-                self.thresholds[tier] = 1.0
-                continue
-            
-            # Conformal risk control: find threshold lam such that
-            # R_hat(lam) <= alpha, where R_hat = (1/n) * sum 1[p >= lam AND y=0]
-            # This means: fraction of examples with P(success) >= lam that actually failed <= alpha
-            
-            # Sort P(success) values
-            sorted_p = np.sort(p[failed_mask])
-            
-            # Conformal quantile: ceiling of (1-alpha)*(n+1)/n
-            q = int(np.ceil((1 - self.alpha) * (n + 1) / n))
-            q = min(q, len(sorted_p))
-            
-            # Threshold: if P(success) < this, escalate
-            # We want the (1-alpha) quantile of failure nonconformity scores
-            threshold = sorted_p[q - 1] if q > 0 else 0.0
-            self.thresholds[tier] = float(threshold)
-        
-        self.calibrated = True
-        return self.thresholds
-    
-    def should_escalate(self, tier: int, psuccess: float) -> bool:
-        """Decide whether to escalate from this tier.
-        
-        Returns True if P(success) is below conformal threshold,
-        meaning we can't guarantee success at this tier with 1-alpha coverage.
-        """
-        if not self.calibrated:
-            return psuccess < 0.65  # fallback to heuristic
-        threshold = self.thresholds.get(tier, 0.65)
-        return psuccess < threshold
-    
-    def coverage_check(
-        self,
-        psuccess: Dict[int, np.ndarray],
-        outcomes: Dict[int, np.ndarray],
-    ) -> Dict[int, Dict[str, float]]:
-        """Verify conformal coverage on test data."""
-        results = {}
-        for tier in sorted(psuccess.keys()):
-            p = psuccess[tier]
-            y = outcomes[tier]
-            threshold = self.thresholds.get(tier, 0.65)
-            
-            no_escalate = p >= threshold
-            failed_no_escalate = (y == 0) & no_escalate
-            n_no_escalate = no_escalate.sum()
-            
-            violation_rate = failed_no_escalate.sum() / max(n_no_escalate, 1)
-            escalation_rate = 1 - no_escalate.mean()
-            
-            results[tier] = {
-                "violation_rate": float(violation_rate),
-                "escalation_rate": float(escalation_rate),
-                "threshold": float(threshold),
-                "n_no_escalate": int(n_no_escalate),
-                "n_violations": int(failed_no_escalate.sum()),
-                "covered": violation_rate <= self.alpha,
-            }
-        return results

aco/learned_router.py

@@ -1,151 +0,0 @@
-"""Trained Production Router - Replaces heuristic routing.
-
-Architecture: difficulty-first + ML confirmation + safety floors.
-
-Usage:
-    from aco.learned_router import TrainedRouter
-    
-    router = TrainedRouter.from_pretrained("narcolepticchicken/agent-cost-optimizer")
-    tier, confidence = router.predict("Write a Python function", "coding", difficulty=3)
-"""
-
-import json
-import os
-import pickle
-from typing import Dict, List, Optional, Tuple
-from dataclasses import dataclass
-from collections import defaultdict
-
-try:
-    import numpy as np
-    import xgboost as xgb
-    HAS_ML = True
-except ImportError:
-    HAS_ML = False
-
-
-TASK_TYPES = ["quick_answer","coding","research","document_drafting",
-              "legal_regulated","tool_heavy","retrieval_heavy",
-              "long_horizon","unknown_ambiguous"]
-TT2IDX = {t:i for i,t in enumerate(TASK_TYPES)}
-
-CODE_KW = ["python","javascript","code","function","bug","debug","refactor",
-           "implement","test","compile","runtime","class","module","async","thread"]
-LEGAL_KW = ["contract","legal","compliance","gdpr","privacy","policy","regulatory","liability"]
-RESEARCH_KW = ["research","find sources","literature","investigate","compare","analyze","survey"]
-TOOL_KW = ["search","fetch","retrieve","query","api","database","scrape","aggregate"]
-LONG_KW = ["plan","project","roadmap","orchestrate","multi-step","migrate","pipeline","deploy"]
-MATH_KW = ["calculate","compute","solve","equation","formula","optimize","probability"]
-
-# Default safety floors per task type
-TASK_FLOOR = {
-    "legal_regulated":4,"long_horizon":3,"research":3,"coding":3,
-    "unknown_ambiguous":3,"quick_answer":1,"document_drafting":2,
-    "tool_heavy":2,"retrieval_heavy":2,
-}
-
-
-class TrainedRouter:
-    """Production trained router: difficulty-first + ML confirmation + safety floors."""
-
-    def __init__(self, tier_clfs: Dict, feat_keys: List[str], 
-                 tier_config: Dict, escalation_threshold: float = 0.55):
-        self.tier_clfs = tier_clfs
-        self.feat_keys = feat_keys
-        self.tier_config = tier_config
-        self.tier_cost = {int(k):v for k,v in tier_config["tier_cost"].items()}
-        self.task_floor = tier_config.get("task_floor", TASK_FLOOR)
-        self.escalation_threshold = escalation_threshold
-        self._trained = True
-
-    def extract_features(self, request: str, task_type: str, difficulty: int = 3) -> Dict:
-        r = request.lower()
-        f = {"req_len":len(request),"num_words":len(request.split()),
-            "has_code":int(any(k in r for k in CODE_KW)),
-            "n_code":sum(1 for k in CODE_KW if k in r),
-            "has_legal":int(any(k in r for k in LEGAL_KW)),
-            "n_legal":sum(1 for k in LEGAL_KW if k in r),
-            "has_research":int(any(k in r for k in RESEARCH_KW)),
-            "n_research":sum(1 for k in RESEARCH_KW if k in r),
-            "has_tool":int(any(k in r for k in TOOL_KW)),
-            "n_tool":sum(1 for k in TOOL_KW if k in r),
-            "has_long":int(any(k in r for k in LONG_KW)),
-            "has_math":int(any(k in r for k in MATH_KW)),
-            "tt_idx":TT2IDX.get(task_type,8),"difficulty":difficulty}
-        for tt in TASK_TYPES:
-            f[f"tt_{tt}"] = int(task_type == tt)
-        return f
-
-    def _feats_to_vec(self, feats: Dict):
-        import numpy as np
-        return np.array([float(feats.get(k, 0.0)) for k in self.feat_keys], dtype=np.float32)
-
-    def predict(self, request: str, task_type: str, difficulty: int = 3,
-                escalation_threshold: Optional[float] = None) -> Tuple[int, float]:
-        """Predict optimal tier using difficulty-first + ML confirmation.
-
-        Returns: (tier, confidence)
-        """
-        threshold = escalation_threshold or self.escalation_threshold
-        
-        # Step 1: difficulty -> base_tier
-        base_tier = min(difficulty + 1, 5)
-        
-        # Step 2: apply safety floor
-        floor = self.task_floor.get(task_type, 2)
-        base_tier = max(base_tier, floor)
-        
-        if not HAS_ML or not self._trained:
-            return base_tier, 0.6
-
-        # Step 3: ML confirmation
-        feats = self.extract_features(request, task_type, difficulty)
-        x = self._feats_to_vec(feats).reshape(1, -1)
-        
-        p_success = self.tier_clfs[base_tier].predict_proba(x)[0, 1]
-        confidence = p_success
-        
-        # Step 4: escalate if P(success) too low
-        while p_success < threshold and base_tier < 5:
-            base_tier += 1
-            p_success = self.tier_clfs[base_tier].predict_proba(x)[0, 1]
-            confidence = p_success
-        
-        return base_tier, float(confidence)
-
-    @classmethod
-    def from_pretrained(cls, repo_id: str, escalation_threshold: float = 0.55,
-                        cache_dir: Optional[str] = None):
-        """Load trained router from HuggingFace Hub."""
-        from huggingface_hub import hf_hub_download
-        
-        bundle_path = hf_hub_download(
-            repo_id=repo_id, filename="router_models/router_bundle.pkl",
-            cache_dir=cache_dir,
-        )
-        
-        with open(bundle_path, "rb") as f:
-            import pickle
-            bundle = pickle.load(f)
-        
-        return cls(
-            tier_clfs={int(k): v for k, v in bundle["tier_clfs"].items()},
-            feat_keys=bundle["feat_keys"],
-            tier_config=bundle["tier_config"],
-            escalation_threshold=escalation_threshold,
-        )
-
-    @classmethod
-    def from_local(cls, model_dir: str, escalation_threshold: float = 0.55):
-        """Load from local directory."""
-        bundle_path = os.path.join(model_dir, "router_bundle.pkl")
-        with open(bundle_path, "rb") as f:
-            import pickle
-            bundle = pickle.load(f)
-        
-        return cls(
-            tier_clfs={int(k): v for k, v in bundle["tier_clfs"].items()},
-            feat_keys=bundle["feat_keys"],
-            tier_config=bundle["tier_config"],
-            escalation_threshold=escalation_threshold,
-        )

aco/router_v10.py

@@ -1,151 +0,0 @@
-"""ACO v10 Router: Trained on REAL SWE-Router execution data.
-
-Key difference from v8: Uses XGBoost models trained on 500 real
-SWE-bench tasks across 8 models, not synthetic data.
-
-Routes based on problem-statement features → per-tier P(success) →
-optimal tier selection. Supports cascade + feedback escalation.
-"""
-import numpy as np
-import pickle, os, json
-from typing import Dict, Optional, Tuple
-from dataclasses import dataclass
-
-CODE_KW = ["python","javascript","code","function","bug","debug","refactor",
-    "implement","test","compile","runtime","segfault","thread","async","class",
-    "module","import","error","traceback"]
-CRITICAL_KW = ["critical","production","urgent","emergency","live","deployed",
-    "safety","security"]
-SIMPLE_KW = ["typo","simple","quick","brief","minor","small","easy","trivial","just"]
-RESEARCH_KW = ["research","investigate","compare","analyze","survey","paper"]
-TOOL_KW = ["search","fetch","retrieve","query","api","database","scrape","aggregate"]
-LONG_KW = ["plan","project","roadmap","orchestrate","migrate","pipeline","deploy","architecture"]
-
-FEAT_KEYS = sorted([
-    'req_len','num_words','has_code','n_code','has_legal','has_research',
-    'has_tool','has_critical','has_simple','has_long','has_math',
-    'has_error_msg','has_file_path','n_lines','has_version','has_add',
-    'has_fix','has_change','has_remove','has_test','has_doc',
-    'has_see_also','has_steps_to_reproduce',
-])
-
-TIER_TO_MODEL = {
-    1: 'deepseek-v4-flash', 2: 'gpt-5-mini',
-    3: 'gemini-2.5-pro', 4: 'claude-opus-4.7', 5: 'gemini-3-pro',
-}
-
-TIER_COST = {1:0.01, 2:0.05, 3:0.15, 4:0.30, 5:0.50}
-
-@dataclass
-class V10RoutingDecision:
-    tier: int
-    model: str
-    confidence: float
-    cost_estimate: float
-    tier_probs: Dict[int, float]
-    escalated: bool = False
-
-class V10Router:
-    def __init__(self, model_path: str = None, success_threshold: float = 0.5):
-        self.success_threshold = success_threshold
-        self.tier_clfs = None
-        self.tier_calibs = None
-        self.opt_clf = None
-        self.feat_keys = FEAT_KEYS
-        if model_path and os.path.exists(model_path):
-            self._load(model_path)
-    
-    def _load(self, path):
-        bundle = pickle.load(open(path, 'rb'))
-        self.tier_clfs = {int(k):v for k,v in bundle.get('tier_clfs',{}).items()}
-        self.tier_calibs = {int(k):v for k,v in bundle.get('tier_calibrators',{}).items()}
-        self.opt_clf = bundle.get('opt_clf', None)
-        self.feat_keys = bundle.get('feat_keys', FEAT_KEYS)
-    
-    def _extract(self, text: str) -> np.ndarray:
-        r = text.lower()
-        feats = {
-            'req_len': len(text), 'num_words': len(text.split()),
-            'has_code': int(any(k in r for k in CODE_KW)),
-            'n_code': sum(1 for k in CODE_KW if k in r),
-            'has_legal': int(any(k in r for k in ["contract","legal","compliance"])),
-            'has_research': int(any(k in r for k in RESEARCH_KW)),
-            'has_tool': int(any(k in r for k in TOOL_KW)),
-            'has_critical': int(any(k in r for k in CRITICAL_KW)),
-            'has_simple': int(any(k in r for k in SIMPLE_KW)),
-            'has_long': int(any(k in r for k in LONG_KW)),
-            'has_math': int(any(k in r for k in ["calculate","compute","solve","equation"])),
-            'has_error_msg': int('error' in r or 'traceback' in r or 'exception' in r),
-            'has_file_path': int('/' in r),
-            'n_lines': text.count('\n') + 1,
-            'has_version': int('version' in r or 'update' in r),
-            'has_add': int('add' in r or 'new' in r or 'create' in r),
-            'has_fix': int('fix' in r or 'bug' in r or 'issue' in r),
-            'has_change': int('change' in r or 'modify' in r),
-            'has_remove': int('remove' in r or 'delete' in r),
-            'has_test': int('test' in r or 'spec' in r),
-            'has_doc': int('doc' in r or 'readme' in r),
-            'has_see_also': int('see also' in r or 'related' in r),
-            'has_steps_to_reproduce': int('reproduce' in r or 'steps' in r),
-        }
-        return np.array([float(feats.get(k,0.0)) for k in self.feat_keys], dtype=np.float32).reshape(1,-1)
-    
-    def route_cascade(self, text: str) -> V10RoutingDecision:
-        """Route to cheapest tier with P(success) >= threshold."""
-        x = self._extract(text)
-        tier_probs = {}
-        if self.tier_clfs:
-            for t in range(1, 6):
-                if t in self.tier_clfs:
-                    p_raw = self.tier_clfs[t].predict_proba(x)[0,1]
-                    p_cal = float(self.tier_calibs[t].transform([p_raw])[0])
-                    tier_probs[t] = p_cal
-                else:
-                    tier_probs[t] = 0.5
-        else:
-            tier_probs = {1:0.67,2:0.72,3:0.50,4:0.84,5:0.70}
-        
-        # Find cheapest tier above threshold
-        selected_tier = 5
-        for t in range(1, 6):
-            if tier_probs.get(t, 0) >= self.success_threshold:
-                selected_tier = t
-                break
-        
-        model = TIER_TO_MODEL.get(selected_tier, 'claude-opus-4.7')
-        return V10RoutingDecision(
-            tier=selected_tier, model=model,
-            confidence=tier_probs.get(selected_tier, 0.5),
-            cost_estimate=TIER_COST.get(selected_tier, 0.30),
-            tier_probs=tier_probs,
-        )
-    
-    def route_direct(self, text: str) -> V10RoutingDecision:
-        """Predict optimal tier directly."""
-        x = self._extract(text)
-        if self.opt_clf:
-            tier = int(self.opt_clf.predict(x)[0]) + 1
-        else:
-            tier = 4  # fallback
-        model = TIER_TO_MODEL.get(tier, 'claude-opus-4.7')
-        return V10RoutingDecision(
-            tier=tier, model=model,
-            confidence=0.8, cost_estimate=TIER_COST.get(tier, 0.30),
-            tier_probs={},
-        )
-    
-    def route_with_feedback(self, text: str, initial_success: bool = True) -> V10RoutingDecision:
-        """Route with feedback: start cheap, escalate on failure."""
-        initial = self.route_cascade(text)
-        if initial_success:
-            return initial
-        # Escalate
-        escalated_tier = min(initial.tier + 1, 5)
-        model = TIER_TO_MODEL.get(escalated_tier, 'claude-opus-4.7')
-        return V10RoutingDecision(
-            tier=escalated_tier, model=model,
-            confidence=initial.tier_probs.get(escalated_tier, 0.8),
-            cost_estimate=TIER_COST.get(initial.tier, 0.01) + TIER_COST.get(escalated_tier, 0.30),
-            tier_probs=initial.tier_probs,
-            escalated=True,
-        )

docs/trained_router_report.md

@@ -1,69 +0,0 @@
-# Trained Router Report
-
-## Architecture
-
-The trained router uses a **difficulty-first + ML confirmation + safety floor** architecture:
-
-1. Map task_type → difficulty (1-5)
-2. Compute base_tier = min(difficulty + 1, 5)
-3. Apply safety floor per task_type (e.g., legal_regulated → tier 4)
-4. Use per-tier XGBoost P(success) classifiers to confirm or escalate
-5. If P(success@base_tier) < threshold, escalate one tier at a time
-
-### Per-Tier XGBoost Classifiers
-
-5 binary classifiers, each predicting P(task succeeds | query, tier=X).
-
-Trained on 50,000 synthetic traces with ground-truth per-tier success labels.
-
-Features: 23 (request text signals + task type one-hot + difficulty)
-
-## Results (N=2,000 eval traces, seed=999)
-
-| Router | Success | AvgCost | CostRed vs Frontier | Unsafe | F-DONE |
-|--------|---------|---------|---------------------|--------|--------|
-| oracle | 99.8% | 0.4862 | 51.4% | 0.0% | 0.3% |
-| prod_t0.65 | 91.9% | 1.365 | -36.5% | 1.5% | 6.6% |
-| prod_t0.60 | 90.7% | 1.316 | -31.6% | 1.8% | 7.4% |
-| always_frontier | 88.8% | 1.000 | 0% | 2.5% | 8.7% |
-| prod_t0.55 | 85.5% | 1.107 | -10.7% | 4.1% | 10.4% |
-| heuristic_diff+1 | 83.4% | 0.940 | 6.0% | 4.9% | 11.7% |
-| heuristic_floor | 59.7% | 0.501 | 49.9% | 27.8% | 12.6% |
-| always_cheap | 20.9% | 0.050 | 95.0% | 79.0% | 0.0% |
-
-## Key Findings
-
-1. **Trained router at t=0.65 achieves 91.9% success — 3.1pp HIGHER than always-frontier (88.8%)**
-2. The unsafe rate drops from 2.5% (frontier) to 1.5% (trained)
-3. The cost is higher because the ML classifiers are conservative (they escalate more)
-4. The oracle shows 51.4% cost reduction is achievable with perfect routing
-
-## The Cost Problem
-
-The trained router OVER-ESCALATES because:
-- Per-tier P(success) classifiers for tiers 1-2 have low accuracy (f1 < 0.5)
-- They underpredict success at low tiers, causing unnecessary escalation
-- This is a training data problem: success at low tiers is inherently rare (22%, 40%)
-
-## Solutions (Ordered by Expected Impact)
-
-1. **Calibrate classifier probabilities** (Platt scaling or isotonic regression on held-out data)
-2. **Add more training data** for easy tasks (oversample quick_answer successes)
-3. **Use difficulty as direct feature** — already top-3 in feature importance
-4. **Fine-tune escalation threshold per task type** (lower for quick_answer, higher for legal)
-5. **Retrain with asymmetric sample weights** (5x penalty for underkill examples)
-
-## Current Recommendation
-
-Use **prod_t0.55** as default: 85.5% success, 10.7% cost increase vs frontier, 4.1% unsafe.
-This is conservative (prefers safety over savings) which is the right default for production.
-
-For cost-sensitive deployments, use **heuristic_diff+1**: 83.4% success, 6% savings.
-
-## Files
-
-- `router_models/router_bundle.pkl` — Pickled router with all 5 XGBoost classifiers
-- `router_models/tier_{1-5}_success.json` — Individual XGBoost model files
-- `router_models/feat_keys.json` — Feature key order
-- `router_models/tier_config.json` — Tier costs, strengths, task floors
-- `training/` — All training scripts (v1-v4)

eval/baar_results.json

@@ -1,62 +0,0 @@
-{
-  "router": "BAAR-style (BERT [CLS] + XGBoost)",
-  "n_tasks": 500,
-  "n_features": 791,
-  "n_bert_dims": 768,
-  "n_keyword_feats": 23,
-  "policies": {
-    "baar": {
-      "success": 383,
-      "success_rate": 0.766,
-      "avg_cost": 0.1878,
-      "cost_reduction_pct": 40.7
-    },
-    "baar_feedback": {
-      "success": 426,
-      "success_rate": 0.852,
-      "avg_cost": 0.2485,
-      "cost_reduction_pct": 21.5
-    },
-    "keyword_only": {
-      "success": 382,
-      "success_rate": 0.764,
-      "avg_cost": 0.1815,
-      "cost_reduction_pct": 42.7
-    },
-    "frontier": {
-      "success": 391,
-      "success_rate": 0.782,
-      "avg_cost": 0.3167,
-      "cost_reduction_pct": 0.0
-    },
-    "always_cheap": {
-      "success": 316,
-      "success_rate": 0.632,
-      "avg_cost": 0.0142,
-      "cost_reduction_pct": 95.5
-    },
-    "oracle": {
-      "success": 435,
-      "success_rate": 0.87,
-      "avg_cost": 0.0586,
-      "cost_reduction_pct": 81.5
-    }
-  },
-  "cv_f1_comparison": {
-    "keyword_only": {
-      "1": 0.7237525635605897,
-      "2": 0.787730931045319,
-      "3": 0.5728946347769878,
-      "4": 0.8946610840323498,
-      "5": 0.7593761150692749
-    },
-    "baar_bert_kw": {
-      "1": 0.7696474695238006,
-      "2": 0.8160680568523706,
-      "3": 0.5678540235539141,
-      "4": 0.9130434782608695,
-      "5": 0.8033430766327058
-    }
-  },
-  "opt_tier_accuracy": 1.0
-}

router_models/feat_keys.json

@@ -1 +0,0 @@
-["difficulty", "has_code", "has_legal", "has_long", "has_math", "has_research", "has_tool", "n_code", "n_legal", "n_research", "n_tool", "num_words", "req_len", "tt_coding", "tt_document_drafting", "tt_idx", "tt_legal_regulated", "tt_long_horizon", "tt_quick_answer", "tt_research", "tt_retrieval_heavy", "tt_tool_heavy", "tt_unknown_ambiguous"]

router_models/router_bundle_v10_fixed.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:2eaf00b52c9d01e991f336518704fe2d6913ed7deb86e7b7cab0a8d31ec81f0c
-size 223016

router_models/router_bundle_v6.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:30e4e5548187842da8b73a96879a783f5e9b7f7d83d6be38f51be6d222585b64
-size 1649115

router_models/router_bundle_v8.pkl

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:3156c8b66c23af926f96a6ef146482c7ebb185877fb5277a822d50d8e7de8cee
-size 1649282

router_models/tier_config.json

@@ -1,27 +0,0 @@
-{
-  "tier_cost": {
-    "1": 0.05,
-    "2": 0.15,
-    "3": 0.75,
-    "4": 1.0,
-    "5": 1.5
-  },
-  "tier_str": {
-    "1": 0.35,
-    "2": 0.55,
-    "3": 0.8,
-    "4": 0.93,
-    "5": 0.97
-  },
-  "task_floor": {
-    "legal_regulated": 4,
-    "long_horizon": 3,
-    "research": 3,
-    "coding": 3,
-    "unknown_ambiguous": 3,
-    "quick_answer": 1,
-    "document_drafting": 2,
-    "tool_heavy": 2,
-    "retrieval_heavy": 2
-  }
-}

training/aco_eval.py

@@ -1,181 +0,0 @@
-#!/usr/bin/env python3
-"""ACO Benchmark Evaluation: Full system test with simulated agent traces."""
-import sys,json,random,pickle,time
-sys.path.insert(0,"/app")
-from collections import defaultdict
-
-TIER_STR={1:0.35,2:0.55,3:0.80,4:0.93,5:0.97}
-TIER_COST={1:0.05,2:0.15,3:0.75,4:1.0,5:1.5}
-TASK_FLOOR={"legal_regulated":4,"long_horizon":3,"research":3,"coding":3,
-             "unknown_ambiguous":3,"quick_answer":1,"document_drafting":2,
-             "tool_heavy":2,"retrieval_heavy":2}
-
-CODE_KW=["python","javascript","code","function","bug","debug","refactor","implement","test"]
-CRITICAL_KW=["critical","production","urgent","now","emergency","live","deployed","safety","security"]
-SIMPLE_KW=["typo","simple","quick","brief","briefly","just","minor","small","easy","trivial","clarification"]
-
-from aco.classifier import TaskCostClassifier
-from aco.router import ModelCascadeRouter
-from aco.context_budgeter import ContextBudgeter
-from aco.tool_gate import ToolCostGate
-from aco.verifier_budgeter import VerifierBudgeter
-from aco.retry_optimizer import RetryOptimizer
-from aco.meta_tool_miner import MetaToolMiner
-from aco.doom_detector import DoomDetector
-
-TASKS={
-  "quick_answer":["What is 2+2?","Explain quantum computing briefly.","Just tell me what 2+2 is."],
-  "coding":["Write a Python function to reverse a linked list.","Fix a typo in the README.","Debug this critical production segfault NOW.","Just fix the typo in line 42."],
-  "research":["Research latest transformer advances.","Find sources comparing LoRA and full FT briefly."],
-  "document_drafting":["Draft project proposal for ML pipeline.","Write email to team about deployment."],
-  "legal_regulated":["Review this contract for liability clauses.","Check GDPR compliance for data pipeline urgently."],
-  "tool_heavy":["Search open issues and create summary.","Fetch API docs and generate client code."],
-  "retrieval_heavy":["Answer based on 50-page document.","Find all payment processing mentions."],
-  "long_horizon":["Plan 3-month roadmap.","Orchestrate complete multi-region deployment."],
-  "unknown_ambiguous":["Help me with this thing.","I need something about the server."],
-}
-
-TOOL_LIST=["web_search","code_search","file_read","file_write","code_execute","verify"]
-TOOL_COST_ESTIMATES={"web_search":{"cost":0.01},"code_search":{"cost":0.005},"file_read":{"cost":0.001},"file_write":{"cost":0.001},"code_execute":{"cost":0.01},"verify":{"cost":0.02}}
-VERIFIER_COST=0.02
-
-print("="*80)
-print("ACO FULL SYSTEM BENCHMARK EVALUATION")
-print("="*80)
-
-# Initialize modules
-classifier=TaskCostClassifier()
-router=ModelCascadeRouter(model_path="/app/router_models/router_bundle_v8.pkl")
-context_budgeter=ContextBudgeter()
-tool_gate=ToolCostGate()
-verifier_budgeter=VerifierBudgeter()
-retry_optimizer=RetryOptimizer()
-meta_tool_miner=MetaToolMiner()
-doom_detector=DoomDetector()
-
-# Simulate 2000 agent runs
-rng=random.Random(42)
-N=2000
-results_aco=[]
-results_frontier=[]
-results_heuristic=[]
-results_cheap=[]
-
-for i in range(N):
-    tt=rng.choice(list(TASKS.keys()))
-    req=rng.choice(TASKS[tt])
-    
-    # Classify
-    pred=classifier.classify(req)
-    # Route
-    routing=router.route(req, pred["task_type"], pred["difficulty"], pred)
-    # Context budget
-    budget=context_budgeter.budget(pred["task_type"],pred["difficulty"],pred["needs_retrieval"],pred["needs_tools"])
-    # Tool decisions
-    tool_decisions={}
-    for tool in TOOL_LIST:
-        if pred["needs_tools"] or tt in ("coding","tool_heavy","retrieval_heavy","research"):
-            td=tool_gate.gate(tool,{"query":req},tt,1,5,routing.confidence)
-            tool_decisions[tool]=td
-    # Verifier
-    vd=verifier_budgeter.should_verify(tt,pred["risk"],routing.confidence,False,False,routing.tier)
-    # Simulate success
-    ps=TIER_STR[routing.tier]**(pred["difficulty"]*0.6)
-    success=rng.random()<ps
-    # Compute cost
-    model_cost=TIER_COST[routing.tier]
-    tool_cost=sum(TOOL_COST_ESTIMATES.get(t,{}).get("cost",0.02) for t,td in tool_decisions.items() if td.action=="use")
-    ver_cost=VERIFIER_COST if vd.should_verify else 0
-    total_cost=model_cost+tool_cost+ver_cost
-    
-    results_aco.append({"tt":tt,"tier":routing.tier,"success":success,"cost":total_cost,
-                       "model_cost":model_cost,"tool_cost":tool_cost,"ver_cost":ver_cost,
-                       "context_tokens":budget.total_tokens,"verified":vd.should_verify,
-                       "tools_used":sum(1 for td in tool_decisions.values() if td.action=="use"),
-                       "escalated":routing.escalated,"downgraded":routing.downgraded})
-    
-    # Baseline: always frontier
-    ps_f=TIER_STR[4]**(pred["difficulty"]*0.6)
-    s_f=rng.random()<ps_f
-    results_frontier.append({"tt":tt,"tier":4,"success":s_f,"cost":1.0+tool_cost+VERIFIER_COST})
-    
-    # Baseline: heuristic
-    h_tier=min(pred["difficulty"]+1,5)
-    h_tier=max(h_tier,TASK_FLOOR.get(tt,2))
-    ps_h=TIER_STR[h_tier]**(pred["difficulty"]*0.6)
-    s_h=rng.random()<ps_h
-    results_heuristic.append({"tt":tt,"tier":h_tier,"success":s_h,"cost":TIER_COST[h_tier]+tool_cost+ver_cost})
-    
-    # Baseline: always cheap
-    ps_c=TIER_STR[1]**(pred["difficulty"]*0.6)
-    s_c=rng.random()<ps_c
-    results_cheap.append({"tt":tt,"tier":1,"success":s_c,"cost":0.05+tool_cost})
-    
-    verifier_budgeter.reset_run()
-
-
-# Compute metrics
-def compute_metrics(results, name):
-    n=len(results)
-    succ=sum(1 for r in results if r["success"])
-    cost=sum(r["cost"] for r in results)
-    model_cost=sum(r.get("model_cost",r["cost"]) for r in results)
-    tool_cost=sum(r.get("tool_cost",0) for r in results)
-    ver_cost=sum(r.get("ver_cost",0) for r in results)
-    ctx=sum(r.get("context_tokens",8000) for r in results)/n
-    verified=sum(1 for r in results if r.get("verified",True))
-    tools=sum(r.get("tools_used",0) for r in results)/n
-    escalations=sum(1 for r in results if r.get("escalated",False))
-    downgrades=sum(1 for r in results if r.get("downgraded",False))
-    return {"name":name,"success_rate":succ/n,"avg_cost":cost/n,
-            "model_cost":model_cost/n,"tool_cost":tool_cost/n,"ver_cost":ver_cost/n,
-            "avg_context_tokens":ctx,"verifications":verified,
-            "avg_tools":tools,"escalations":escalations,"downgrades":downgrades}
-
-m=compute_metrics(results_aco,"aco_v8")
-m_f=compute_metrics(results_frontier,"always_frontier")
-m_h=compute_metrics(results_heuristic,"heuristic")
-m_c=compute_metrics(results_cheap,"always_cheap")
-
-print(f"\n{'Router':<20} {'Success':>10} {'AvgCost':>10} {'CostRed':>10} {'ModelCost':>10} {'ToolCost':>10} {'VerCost':>10} {'Context':>10} {'Verifs':>8}")
-print("-"*100)
-for r in [m_f,m_h,m,m_c]:
-    cr=(1-r["avg_cost"]/m_f["avg_cost"])*100
-    print(f"{r['name']:<20} {r['success_rate']:>10.3f} {r['avg_cost']:>10.4f} {cr:>9.1f}% {r['model_cost']:>10.4f} {r['tool_cost']:>10.4f} {r['ver_cost']:>10.4f} {r['avg_context_tokens']:>10.0f} {r['verifications']:>8d}")
-
-# Per-task breakdown
-print(f"\n\nPer-task breakdown:")
-for tt in sorted(set(r["tt"] for r in results_aco)):
-    aco_tt=[r for r in results_aco if r["tt"]==tt]
-    front_tt=[r for r in results_frontier if r["tt"]==tt]
-    n_tt=len(aco_tt)
-    a_s=sum(1 for r in aco_tt if r["success"])/n_tt
-    a_c=sum(r["cost"] for r in aco_tt)/n_tt
-    f_c=sum(r["cost"] for r in front_tt)/n_tt
-    f_s=sum(1 for r in front_tt if r["success"])/n_tt
-    cr=(1-a_c/f_c)*100
-    print(f"  {tt:<20} n={n_tt:>4} aco_success={a_s:.3f} frontier_success={f_s:.3f} aco_cost={a_c:.4f} costRed={cr:.1f}%")
-
-# Cost-quality frontier
-print(f"\n\nCost-Quality Frontier:")
-frontier_points=[]
-for r in [m_c,m_h,m,m_f]:
-    frontier_points.append((r["avg_cost"],r["success_rate"],r["name"]))
-frontier_points.sort(key=lambda x:x[0])
-for cost,succ,name in frontier_points:
-    print(f"  {name:<20} cost={cost:.4f} success={succ:.3f}")
-
-# Key findings
-print(f"\n\nKEY FINDINGS:")
-print(f"  ACO v8 success rate: {m['success_rate']:.3f}")
-print(f"  ACO v8 cost reduction: {(1-m['avg_cost']/m_f['avg_cost'])*100:.1f}%")
-print(f"  ACO v8 avg context: {m['avg_context_tokens']:.0f} tokens")
-print(f"  ACO v8 verifications: {m['verifications']}/{N}")
-print(f"  Escalations: {m['escalations']} ({m['escalations']/N*100:.1f}%)")
-print(f"  Downgrades: {m['downgrades']} ({m['downgrades']/N*100:.1f}%)")
-
-# Save
-with open("/app/aco_benchmark_results.json","w") as f:
-    json.dump({"aco_v8":m,"frontier":m_f,"heuristic":m_h,"cheap":m_c},f,indent=2)
-print(f"\nSaved to /app/aco_benchmark_results.json")
-print("DONE!")

training/benchmark_final.py

@@ -1,143 +0,0 @@
-#!/usr/bin/env python3
-"""FINAL v9 benchmark: v8 router + selective execution feedback.
-
-Strategy: Use v8 router for initial tier selection. Then for tasks
-where v8 chose tier < 4 (not frontier), run cheap model first and
-use output confidence to decide whether to escalate. This catches
-the ~2% of cases where cheap model fails but could be saved by
-escalation, without running double-inference everywhere.
-"""
-import sys,random,math,pickle,json
-sys.path.insert(0,"/app")
-from collections import defaultdict
-from aco.classifier import TaskCostClassifier
-from aco.router import ModelCascadeRouter
-from aco.execution_feedback import ExecutionFeedbackRouter
-
-TIER_STR={1:0.35,2:0.55,3:0.80,4:0.93,5:0.97}
-TIER_COST={1:0.05,2:0.15,3:0.75,4:1.0,5:1.5}
-TASK_FLOOR={"legal_regulated":4,"long_horizon":3,"research":3,"coding":3,
-             "unknown_ambiguous":3,"quick_answer":1,"document_drafting":2,
-             "tool_heavy":2,"retrieval_heavy":2}
-
-TASKS = {
-  "quick_answer":["What is 2+2?","Explain quantum computing briefly.","Convert 100F to Celsius."],
-  "coding":["Write Python function to reverse linked list.","Fix typo in README.",
-      "Debug critical production segfault NOW.","Just fix typo in line 42."],
-  "research":["Research latest transformer advances."],
-  "document_drafting":["Draft project proposal for ML pipeline."],
-  "legal_regulated":["Review this contract for liability clauses."],
-  "tool_heavy":["Search open issues and create summary."],
-  "retrieval_heavy":["Answer based on 50-page document."],
-  "long_horizon":["Plan 3-month roadmap."],
-  "unknown_ambiguous":["Help me with this thing."],
-}
-
-classifier=TaskCostClassifier()
-router=ModelCascadeRouter(model_path="/app/router_models/router_bundle_v8.pkl")
-
-def sim_lps_correlated(tier, diff, success, rng):
-    """Better simulation: successful outputs have higher confidence."""
-    n=rng.randint(20,150)
-    if success:
-        # High confidence: mean logprob close to 0
-        base={1:-1.5,2:-1.0,3:-0.6,4:-0.3,5:-0.2}[tier]
-        noise_std=0.3+diff*0.05
-    else:
-        # Low confidence: more uncertain, lower logprobs
-        base={1:-4.0,2:-3.0,3:-2.0,4:-1.5,5:-1.0}[tier]
-        noise_std=0.8+diff*0.2
-    return [base+rng.gauss(0,noise_std) for _ in range(n)]
-
-N=3000
-results = {}
-rng = random.Random(42)
-
-# Method 1: always frontier
-rng.seed(42); succ=0; cost=0.0
-for i in range(N):
-    tt=rng.choice(list(TASKS.keys())); req=rng.choice(TASKS[tt])
-    pred=classifier.classify(req)
-    ps=TIER_STR[4]**(pred["difficulty"]*0.6)
-    if rng.random()<ps: succ+=1
-    cost+=TIER_COST[4]
-results["frontier"]={"success":succ/N,"cost":cost/N}
-
-# Method 2: v8 router
-rng.seed(42); succ=0; cost=0.0
-for i in range(N):
-    tt=rng.choice(list(TASKS.keys())); req=rng.choice(TASKS[tt])
-    pred=classifier.classify(req)
-    r=router.route(req,tt,pred["difficulty"],pred)
-    ps=TIER_STR[r.tier]**(r.dynamic_difficulty*0.6)
-    if rng.random()<ps: succ+=1
-    cost+=TIER_COST[r.tier]
-results["v8"]={"success":succ/N,"cost":cost/N}
-
-# Method 3: v9 = v8 + feedback on non-frontier tiers
-# Only use feedback when v8 selected tier < 4
-for ent_thr in [2.0,2.5,3.0]:
-    for lc_thr in [0.05,0.10,0.15]:
-        ef=ExecutionFeedbackRouter(entropy_threshold=ent_thr,
-            low_conf_ratio_threshold=lc_thr, tier_costs=TIER_COST)
-        rng.seed(42); succ=0; cost=0.0; escalated_count=0
-        for i in range(N):
-            tt=rng.choice(list(TASKS.keys())); req=rng.choice(TASKS[tt])
-            pred=classifier.classify(req)
-            r=router.route(req,tt,pred["difficulty"],pred)
-            tier=r.tier; diff=r.dynamic_difficulty
-            # Only use feedback for non-frontier tiers
-            if tier < 4:
-                ps=TIER_STR[tier]**(diff*0.6)
-                initial_success=rng.random()<ps
-                lps=sim_lps_correlated(tier,diff,initial_success,rng)
-                signal=ef.analyze_output(lps,task_type=tt,current_tier=tier)
-                if signal.should_escalate:
-                    final_tier=min(tier+1,5)
-                    final_tier=max(final_tier,TASK_FLOOR.get(tt,1))
-                    ps2=TIER_STR[final_tier]**(diff*0.6)
-                    c=TIER_COST[tier]+TIER_COST[final_tier]
-                    if rng.random()<ps2: succ+=1
-                    escalated_count+=1
-                else:
-                    c=TIER_COST[tier]
-                    if initial_success: succ+=1
-            else:
-                ps=TIER_STR[tier]**(diff*0.6)
-                if rng.random()<ps: succ+=1
-                c=TIER_COST[tier]
-            cost+=c
-        name=f"v9_e{ent_thr}_lc{lc_thr}"
-        results[name]={"success":succ/N,"cost":cost/N,
-                       "escalated":escalated_count,"ent_thr":ent_thr,"lc_thr":lc_thr}
-
-print("="*80)
-print("FINAL v9 RESULTS: v8 ROUTER + SELECTIVE FEEDBACK")
-print("="*80)
-fc=results["frontier"]["cost"]
-print(f"\n{'Method':<25} {'Success':>10} {'AvgCost':>10} {'CostRed':>10} {'Gap':>8}")
-print("-"*65)
-print(f"{'always_frontier':<25} {results['frontier']['success']:>10.3f} {results['frontier']['cost']:>10.4f} {'0.0%':>10} {0.901-results['frontier']['success']:>8.3f}")
-print(f"{'v8_router':<25} {results['v8']['success']:>10.3f} {results['v8']['cost']:>10.4f} {(1-results['v8']['cost']/fc)*100:>9.1f}% {0.901-results['v8']['success']:>8.3f}")
-
-best_v9=None
-for name,r in sorted(results.items(),key=lambda x:-x[1]["success"]):
-    if name.startswith("v9"):
-        cr=(1-r["cost"]/fc)*100
-        gap=0.901-r["success"]
-        print(f"{name:<25} {r['success']:>10.3f} {r['cost']:>10.4f} {cr:>9.1f}% {gap:>8.3f}")
-        # Pick the one closest to frontier quality with most savings
-        score=r["success"]*20-r["cost"]*5
-        if best_v9 is None or score>best_v9[0]:
-            best_v9=(score,name,r)
-
-print(f"\n\nBest v9: {best_v9[1]}")
-print(f"  success={best_v9[2]['success']:.3f}, cost={best_v9[2]['cost']:.4f}")
-print(f"  costRed={(1-best_v9[2]['cost']/fc)*100:.1f}%")
-print(f"  quality_gap={0.901-best_v9[2]['success']:.3f}")
-print(f"  v8 gap={0.901-results['v8']['success']:.3f}")
-print(f"  Improvement: {best_v9[2]['success']-results['v8']['success']:+.3f} success, {best_v9[2]['cost']-results['v8']['cost']:+.4f} cost")
-
-with open("/app/benchmark_v9_final.json","w") as f:
-    json.dump(results,f,indent=2,default=str)
-print("DONE!")

training/benchmark_with_feedback.py

@@ -1,199 +0,0 @@
-#!/usr/bin/env python3
-"""Comprehensive benchmark with execution-feedback routing."""
-import sys,json,random,math,pickle,time
-sys.path.insert(0,"/app")
-from collections import defaultdict
-
-TIER_STR={1:0.35,2:0.55,3:0.80,4:0.93,5:0.97}
-TIER_COST={1:0.05,2:0.15,3:0.75,4:1.0,5:1.5}
-TASK_FLOOR={"legal_regulated":4,"long_horizon":3,"research":3,"coding":3,
-             "unknown_ambiguous":3,"quick_answer":1,"document_drafting":2,
-             "tool_heavy":2,"retrieval_heavy":2}
-
-from aco.classifier import TaskCostClassifier
-from aco.router import ModelCascadeRouter
-from aco.execution_feedback import ExecutionFeedbackRouter, FeedbackSignal
-
-TASKS = {
-  "quick_answer":["What is 2+2?","Explain quantum computing briefly.",
-      "What is the capital of France?","Convert 100F to Celsius.",
-      "Small clarification on this formula."],
-  "coding":["Write a Python function to reverse a linked list.",
-      "Fix a typo in the README.","Debug this critical production segfault NOW.",
-      "Just fix the typo in line 42.","Refactor auth module to JWT.",
-      "Implement LRU cache in Go."],
-  "research":["Research latest transformer advances.",
-      "Find sources comparing LoRA and full FT briefly.",
-      "Investigate data center climate impact."],
-  "document_drafting":["Draft project proposal for ML pipeline.",
-      "Write email to team about deployment."],
-  "legal_regulated":["Review this contract for liability clauses.",
-      "Check GDPR compliance for data pipeline urgently."],
-  "tool_heavy":["Search open issues and create summary.",
-      "Fetch API docs and generate client code."],
-  "retrieval_heavy":["Answer based on 50-page document.",
-      "Find all payment processing mentions."],
-  "long_horizon":["Plan 3-month roadmap.",
-      "Orchestrate complete multi-region deployment."],
-  "unknown_ambiguous":["Help me with this thing.",
-      "I need something about the server."],
-}
-
-print("="*80)
-print("ACO v9 BENCHMARK: EXECUTION-FEEDBACK + DYNAMIC DIFFICULTY")
-print("="*80)
-
-classifier = TaskCostClassifier()
-router = ModelCascadeRouter(model_path="/app/router_models/router_bundle_v8.pkl")
-ef_router = ExecutionFeedbackRouter(tier_costs=TIER_COST, task_floors=TASK_FLOOR)
-
-rng = random.Random(42)
-N = 3000
-
-def sim_logprobs(tier, difficulty, success, rng):
-    """Simulate token logprobs based on tier and difficulty."""
-    n_tokens = rng.randint(20, 200)
-    base_lp = {1:-3.5, 2:-2.5, 3:-1.5, 4:-0.7, 5:-0.3}[tier]
-    base_lp *= (1 + difficulty * 0.15)
-    lps = []
-    for _ in range(n_tokens):
-        noise = rng.gauss(0, 1.0 + difficulty * 0.3)
-        if success:
-            lps.append(base_lp + noise * 0.3)
-        else:
-            lps.append(base_lp + noise * 0.8)
-    return lps
-
-def eval_method(name, route_fn):
-    succ = 0; cost = 0.0; unsafe = 0
-    per_tt = defaultdict(lambda: {"succ":0,"cost":0.0,"n":0})
-    for i in range(N):
-        tt = rng.choice(list(TASKS.keys()))
-        req = rng.choice(TASKS[tt])
-        pred = classifier.classify(req)
-        tier, s, c, u = route_fn(req, tt, pred)
-        if s: succ += 1
-        cost += c
-        if u: unsafe += 1
-        per_tt[tt]["succ"] += (1 if s else 0)
-        per_tt[tt]["cost"] += c
-        per_tt[tt]["n"] += 1
-    return {"name":name,"success":succ/N,"avg_cost":cost/N,"unsafe":unsafe/N,"per_tt":dict(per_tt)}
-
-# Method A: always frontier
-def route_frontier(req, tt, pred):
-    ps = TIER_STR[4]**(pred["difficulty"]*0.6)
-    return 4, rng.random()<ps, TIER_COST[4], False
-
-# Method B: heuristic
-def route_heuristic(req, tt, pred):
-    h = min(pred["difficulty"]+1,5)
-    h = max(h, TASK_FLOOR.get(tt,2))
-    ps = TIER_STR[h]**(pred["difficulty"]*0.6)
-    return h, rng.random()<ps, TIER_COST[h], (h < 4 and not rng.random()<ps)
-
-# Method C: v8 router (no feedback)
-def route_v8(req, tt, pred):
-    r = router.route(req, tt, pred["difficulty"], pred)
-    ps = TIER_STR[r.tier]**(r.dynamic_difficulty*0.6)
-    return r.tier, rng.random()<ps, TIER_COST[r.tier], r.escalated
-
-# Method D: v9 = v8 router + execution feedback cascade
-def route_v9(req, tt, pred):
-    r = router.route(req, tt, pred["difficulty"], pred)
-    initial_tier = r.tier
-    ps_initial = TIER_STR[initial_tier]**(r.dynamic_difficulty*0.6)
-    initial_success = rng.random() < ps_initial
-    lps = sim_logprobs(initial_tier, r.dynamic_difficulty, initial_success, rng)
-    signal = ef_router.analyze_output(lps, task_type=tt, current_tier=initial_tier)
-    if signal.should_escalate and initial_tier < 5:
-        final_tier = min(initial_tier + 1, 5)
-        final_tier = max(final_tier, TASK_FLOOR.get(tt, 1))
-        ps_final = TIER_STR[final_tier]**(r.dynamic_difficulty*0.6)
-        final_success = rng.random() < ps_final
-        total_cost = TIER_COST[initial_tier] + TIER_COST[final_tier]
-        return final_tier, final_success, total_cost, False
-    else:
-        return initial_tier, initial_success, TIER_COST[initial_tier], False
-
-# Method E: oracle
-def route_oracle(req, tt, pred):
-    for t in range(1,6):
-        ps = TIER_STR[t]**(pred["difficulty"]*0.6)
-        if rng.random() < ps:
-            return t, True, TIER_COST[t], False
-    return 5, False, TIER_COST[5], False
-
-# Method F: always cheap
-def route_cheap(req, tt, pred):
-    ps = TIER_STR[1]**(pred["difficulty"]*0.6)
-    return 1, rng.random()<ps, TIER_COST[1], (not rng.random()<ps)
-
-# Run all methods
-print(f"\n[1] Running {N} simulated traces per method...")
-results = {}
-for name, fn in [("always_frontier",route_frontier),("always_cheap",route_cheap),
-                  ("heuristic",route_heuristic),("v8_router",route_v8),
-                  ("v9_feedback",route_v9),("oracle",route_oracle)]:
-    rng_state = rng.getstate()
-    rng.seed(42)
-    results[name] = eval_method(name, fn)
-    rng.setstate(rng_state)
-
-# Print comparison
-print(f"\n\n{'Method':<20} {'Success':>10} {'AvgCost':>10} {'CostRed':>10} {'Unsafe':>10}")
-print("-"*60)
-fc = results["always_frontier"]["avg_cost"]
-for name in ["oracle","always_frontier","v9_feedback","v8_router","heuristic","always_cheap"]:
-    r = results[name]
-    cr = (1-r["avg_cost"]/fc)*100
-    print(f"{name:<20} {r['success']:>10.3f} {r['avg_cost']:>10.4f} {cr:>9.1f}% {r['unsafe']:>10.3f}")
-
-# Per-task comparison
-print(f"\n\n[2] Per-task success rate comparison:")
-print(f"{'Task':<20} {'Frontier':>10} {'v8':>10} {'v9_feedback':>12} {'CostRed_v9':>12}")
-print("-"*65)
-for tt in sorted(set(k for r in results.values() for k in r["per_tt"])):
-    f_r = results["always_frontier"]["per_tt"].get(tt,{"succ":0,"n":1,"cost":0})
-    v8_r = results["v8_router"]["per_tt"].get(tt,{"succ":0,"n":1,"cost":0})
-    v9_r = results["v9_feedback"]["per_tt"].get(tt,{"succ":0,"n":1,"cost":0})
-    f_sr = f_r["succ"]/max(f_r["n"],1)
-    v8_sr = v8_r["succ"]/max(v8_r["n"],1)
-    v9_sr = v9_r["succ"]/max(v9_r["n"],1)
-    f_c = f_r["cost"]/max(f_r["n"],1)
-    v9_c = v9_r["cost"]/max(v9_r["n"],1)
-    cr = (1-v9_c/f_c)*100 if f_c > 0 else 0
-    print(f"{tt:<20} {f_sr:>10.3f} {v8_sr:>10.3f} {v9_sr:>12.3f} {cr:>11.1f}%")
-
-# Cost-quality frontier
-print(f"\n\n[3] Cost-Quality Frontier:")
-for name in ["always_cheap","v8_router","v9_feedback","heuristic","always_frontier","oracle"]:
-    r = results[name]
-    cr = (1-r["avg_cost"]/fc)*100
-    print(f"  {name:<20} success={r['success']:.3f} cost={r['avg_cost']:.4f} costRed={cr:.1f}%")
-
-# Key metrics
-v9 = results["v9_feedback"]
-v8 = results["v8_router"]
-fr = results["always_frontier"]
-v9_cr = (1-v9["avg_cost"]/fr["avg_cost"])*100
-v8_cr = (1-v8["avg_cost"]/fr["avg_cost"])*100
-quality_gap_v9 = fr["success"] - v9["success"]
-quality_gap_v8 = fr["success"] - v8["success"]
-
-print(f"\n\n[4] KEY RESULTS:")
-print(f"  v8 success:     {v8['success']:.3f} (gap vs frontier: {quality_gap_v8:.3f})")
-print(f"  v9 success:     {v9['success']:.3f} (gap vs frontier: {quality_gap_v9:.3f})")
-print(f"  v8 costRed:     {v8_cr:.1f}%")
-print(f"  v9 costRed:     {v9_cr:.1f}%")
-print(f"  Quality gain v9 vs v8: {v9['success']-v8['success']:+.3f}")
-print(f"  Cost increase v9 vs v8: {v9['avg_cost']-v8['avg_cost']:+.4f}")
-
-# Save
-with open("/app/benchmark_v9_results.json","w") as f:
-    save_data = {}
-    for name, r in results.items():
-        save_data[name] = {"success":r["success"],"avg_cost":r["avg_cost"],"unsafe":r["unsafe"]}
-    json.dump(save_data, f, indent=2)
-print(f"\nSaved to /app/benchmark_v9_results.json")
-print("DONE!")

training/bfcl_eval.py

@@ -1,102 +0,0 @@
-#!/usr/bin/env python3
-"""BFCL v3 Function-Calling Benchmark: Evaluate ACO tool-use routing."""
-import sys,json
-from collections import defaultdict
-from datasets import load_dataset
-
-print("="*80)
-print("BFCL v3 FUNCTION-CALLING BENCHMARK")
-print("="*80)
-
-# Load BFCL
-print("\n[1] Loading BFCL v3 trajectories...")
-ds = load_dataset('bespokelabs/bfcl-v3-02-27-metrics-trajectories', split='train')
-print(f"  Loaded {len(ds)} rows")
-
-# Get unique models and their success rates
-model_stats = defaultdict(lambda: {"valid":0,"total":0,"tool_errors":0})
-for row in ds:
-    m = row['model_name']
-    model_stats[m]["total"] += 1
-    if row['valid']: model_stats[m]["valid"] += 1
-    model_stats[m]["tool_errors"] += row.get('num_tool_errors', 0)
-
-# Print model success rates
-print(f"\n[2] Model success rates (top 20 by volume):")
-sorted_models = sorted(model_stats.items(), key=lambda x: -x[1]["total"])[:20]
-print(f"{'Model':<40} {'Valid':>8} {'Total':>8} {'Rate':>8} {'ToolErr':>8}")
-print("-"*75)
-for m, s in sorted_models:
-    rate = s["valid"]/max(s["total"],1)
-    print(f"{m:<40} {s['valid']:>8} {s['total']:>8} {rate:>8.3f} {s['tool_errors']:>8}")
-
-# Group by task ID to find routing opportunities
-print(f"\n[3] Task-level routing analysis...")
-task_results = defaultdict(dict)
-for row in ds:
-    task_results[row['id']][row['model_name']] = {
-        'valid': row['valid'],
-        'tool_errors': row.get('num_tool_errors', 0),
-        'category': row.get('category', ''),
-    }
-
-# Classify models into tiers based on success rate
-all_model_rates = {m: s["valid"]/max(s["total"],1) for m, s in model_stats.items()}
-# Tier assignment based on success rate
-def get_tier(rate):
-    if rate >= 0.85: return 4  # frontier
-    if rate >= 0.70: return 3  # medium
-    if rate >= 0.50: return 2  # cheap
-    return 1  # tiny
-
-model_tiers = {m: get_tier(r) for m, r in all_model_rates.items()}
-
-# For each task: optimal tier = cheapest tier that succeeds
-opt_tier_dist = defaultdict(int)
-savings_opportunity = 0
-for tid, results in task_results.items():
-    successful_tiers = []
-    for m, r in results.items():
-        if r['valid']:
-            successful_tiers.append(model_tiers.get(m, 3))
-    if successful_tiers:
-        opt = min(successful_tiers)
-        opt_tier_dist[opt] += 1
-        if opt < 4:  # could have used cheaper than frontier
-            savings_opportunity += 1
-
-total_tasks = len(task_results)
-print(f"  Total unique tasks: {total_tasks}")
-print(f"  Tasks solvable cheaper than frontier: {savings_opportunity} ({savings_opportunity/total_tasks*100:.1f}%)")
-print(f"  Optimal tier distribution:")
-for tier in sorted(opt_tier_dist.keys()):
-    print(f"    Tier {tier}: {opt_tier_dist[tier]} ({opt_tier_dist[tier]/total_tasks*100:.1f}%)")
-
-# Tool error analysis
-print(f"\n[4] Tool-use cost analysis...")
-total_tool_errors = sum(s["tool_errors"] for s in model_stats.values())
-total_calls = sum(s["total"] for s in model_stats.values())
-print(f"  Total tool errors: {total_tool_errors}")
-print(f"  Tool error rate: {total_tool_errors/max(total_calls,1)*100:.2f}%")
-
-# Categories of errors
-error_cats = defaultdict(int)
-for row in ds:
-    if row.get('error_type'):
-        error_cats[row['error_type']] += 1
-print(f"\n  Error categories:")
-for cat, count in sorted(error_cats.items(), key=lambda x: -x[1])[:10]:
-    print(f"    {cat}: {count}")
-
-# Save results
-results = {
-    "total_tasks": total_tasks,
-    "savings_opportunity_pct": savings_opportunity/total_tasks*100,
-    "opt_tier_distribution": dict(opt_tier_dist),
-    "model_success_rates": {m: s["valid"]/max(s["total"],1) for m, s in sorted_models},
-    "tool_error_rate": total_tool_errors/max(total_calls,1),
-}
-with open("/app/bfcl_results.json", "w") as f:
-    json.dump(results, f, indent=2)
-print(f"\nSaved to /app/bfcl_results.json")
-print("DONE!")

training/train_router_real.py

@@ -1,435 +0,0 @@
-#!/usr/bin/env python3
-"""Train v10 router on REAL SWE-Router execution data.
-
-This is the big one: 500 tasks x 8 models = 4000 real outcomes.
-We learn which model succeeds on which task, at what cost.
-"""
-import sys, json, random, pickle, math
-from collections import defaultdict
-from datasets import load_dataset
-import numpy as np
-
-print("="*80)
-print("TRAINING v10 ROUTER ON REAL SWE-ROUTER DATA")
-print("="*80)
-
-# Load all SWE-Router traces
-MODELS = ['claude-opus-4.7','gpt-5-mini','gpt-5-nano','gpt-5.2',
-          'gemini-2.5-pro','gemini-3-pro','deepseek-v3.2','deepseek-v4-flash']
-
-MODEL_TIER = {
-    'deepseek-v4-flash': 1, 'gpt-5-nano': 1,
-    'gpt-5-mini': 2, 'deepseek-v3.2': 2,
-    'gemini-2.5-pro': 3,
-    'claude-opus-4.7': 4, 'gpt-5.2': 4,
-    'gemini-3-pro': 5,
-}
-
-TIER_COST = {1:0.01, 2:0.05, 3:0.15, 4:0.30, 5:0.50}
-
-print("\n[1] Loading SWE-Router traces...")
-traces = defaultdict(dict)
-for model in MODELS:
-    ds = load_dataset(f'SWE-Router/swebench-verified-{model}', split='test')
-    for row in ds:
-        iid = row['instance_id']
-        traces[iid][model] = {
-            'resolved': row['resolved'],
-            'cost': float(row['instance_cost']),
-            'api_calls': int(row['api_calls']),
-            'problem': row['problem_statement'],
-        }
-    print(f"  {model}: loaded")
-
-print(f"\n  Total tasks: {len(traces)}")
-print(f"  Total traces: {sum(len(v) for v in traces.values())}")
-
-# ─── Feature Engineering ────────────────────────────────────────────────
-print("\n[2] Engineering features from problem statements...")
-
-# Keyword sets for feature extraction
-CODE_KW = ["python","javascript","code","function","bug","debug","refactor","implement","test",
-           "compile","runtime","segfault","thread","async","class","module","import","error","traceback"]
-LEGAL_KW = ["contract","legal","compliance","gdpr","privacy","policy","regulatory","liability"]
-RESEARCH_KW = ["research","investigate","compare","analyze","survey","paper"]
-TOOL_KW = ["search","fetch","retrieve","query","api","database","scrape","aggregate"]
-CRITICAL_KW = ["critical","production","urgent","emergency","live","deployed","safety","security"]
-SIMPLE_KW = ["typo","simple","quick","brief","minor","small","easy","trivial","just"]
-LONG_KW = ["plan","project","roadmap","orchestrate","migrate","pipeline","deploy","architecture"]
-MATH_KW = ["calculate","compute","solve","equation","formula","optimize","probability"]
-
-def extract_features(problem_text):
-    r = problem_text.lower()
-    feats = {
-        'req_len': len(problem_text),
-        'num_words': len(problem_text.split()),
-        'has_code': int(any(k in r for k in CODE_KW)),
-        'n_code': sum(1 for k in CODE_KW if k in r),
-        'has_legal': int(any(k in r for k in LEGAL_KW)),
-        'has_research': int(any(k in r for k in RESEARCH_KW)),
-        'has_tool': int(any(k in r for k in TOOL_KW)),
-        'has_critical': int(any(k in r for k in CRITICAL_KW)),
-        'has_simple': int(any(k in r for k in SIMPLE_KW)),
-        'has_long': int(any(k in r for k in LONG_KW)),
-        'has_math': int(any(k in r for k in MATH_KW)),
-        'has_error_msg': int('error' in r or 'traceback' in r or 'exception' in r),
-        'has_file_path': int('/' in r and ('.' in r.split('/')[0] if '/' in r else False)),
-        'n_lines': problem_text.count('\n') + 1,
-        'has_version': int('version' in r or 'update' in r or 'upgrade' in r),
-        'has_add': int('add' in r or 'new' in r or 'create' in r),
-        'has_fix': int('fix' in r or 'bug' in r or 'issue' in r or 'broken' in r),
-        'has_change': int('change' in r or 'modify' in r or 'update' in r),
-        'has_remove': int('remove' in r or 'delete' in r or 'drop' in r),
-        'has_test': int('test' in r or 'spec' in r or 'assert' in r),
-        'has_doc': int('doc' in r or 'readme' in r or 'comment' in r),
-        # SWE-specific features
-        'has_see_also': int('see also' in r or 'related' in r),
-        'has_steps_to_reproduce': int('steps to reproduce' in r or 'reproduce' in r),
-    }
-    return feats
-
-# ─── Build Training Data ────────────────────────────────────────────────
-print("\n[3] Building training data...")
-
-# For each task, we know which models succeeded.
-# Ground truth: optimal_tier = cheapest tier where at least one model succeeded
-# Features: extracted from problem statement
-
-all_feat_keys = None
-training_data = []
-tier_labels = {1:[],2:[],3:[],4:[],5:[]}
-cost_labels = []
-
-for iid, model_results in traces.items():
-    problem = next(iter(model_results.values()))['problem']
-    feats = extract_features(problem)
-    
-    if all_feat_keys is None:
-        all_feat_keys = sorted(feats.keys())
-    
-    feat_vec = [float(feats.get(k, 0.0)) for k in all_feat_keys]
-    
-    # Determine ground truth: which tiers succeeded?
-    tier_success = {}
-    for model, result in model_results.items():
-        tier = MODEL_TIER[model]
-        if tier not in tier_success:
-            tier_success[tier] = False
-        if result['resolved']:
-            tier_success[tier] = True
-    
-    # Optimal tier = cheapest that succeeded
-    optimal_tier = 5
-    for t in range(1, 6):
-        if tier_success.get(t, False):
-            optimal_tier = t
-            break
-    
-    # Per-tier success labels
-    for t in range(1, 6):
-        tier_labels[t].append(int(tier_success.get(t, False)))
-    
-    training_data.append({
-        'features': feat_vec,
-        'optimal_tier': optimal_tier,
-        'tier_success': tier_success,
-        'cost': min(r['cost'] for r in model_results.values()),
-    })
-
-print(f"  Training samples: {len(training_data)}")
-print(f"  Features: {len(all_feat_keys)}")
-print(f"  Optimal tier distribution:")
-opt_dist = defaultdict(int)
-for t in training_data:
-    opt_dist[t['optimal_tier']] += 1
-for tier in sorted(opt_dist.keys()):
-    print(f"    Tier {tier}: {opt_dist[tier]} ({opt_dist[tier]/len(training_data)*100:.1f}%)")
-print(f"  Per-tier success rates:")
-for t in range(1,6):
-    s = sum(tier_labels[t])
-    print(f"    Tier {t}: {s}/{len(training_data)} = {s/len(training_data)*100:.1f}%")
-
-# ─── Train XGBoost Models ────────────────────────────────────────────────
-print("\n[4] Training XGBoost per-tier success predictors...")
-
-from xgboost import XGBClassifier
-from sklearn.calibration import IsotonicRegression
-from sklearn.model_selection import cross_val_score
-import warnings
-warnings.filterwarnings('ignore')
-
-X = np.array([t['features'] for t in training_data], dtype=np.float32)
-y_tier = {t: np.array(tier_labels[t]) for t in range(1,6)}
-y_optimal = np.array([t['optimal_tier'] for t in training_data])
-
-tier_clfs = {}
-tier_calibs = {}
-tier_cv_scores = {}
-
-for t in range(1, 6):
-    y = y_tier[t]
-    n_pos = y.sum()
-    n_neg = len(y) - n_pos
-    
-    # Scale pos weight for imbalanced data
-    spw = max(1, n_neg / max(n_pos, 1))
-    
-    clf = XGBClassifier(
-        n_estimators=200, max_depth=5, learning_rate=0.05,
-        subsample=0.8, colsample_bytree=0.8,
-        scale_pos_weight=spw,
-        eval_metric='logloss', use_label_encoder=False,
-        random_state=42,
-    )
-    
-    # CV score
-    try:
-        scores = cross_val_score(clf, X, y, cv=5, scoring='f1')
-        tier_cv_scores[t] = scores.mean()
-    except:
-        tier_cv_scores[t] = 0.0
-    
-    clf.fit(X, y)
-    
-    # Calibrate
-    p_raw = clf.predict_proba(X)[:, 1]
-    cal = IsotonicRegression(out_of_bounds='clip')
-    cal.fit(p_raw, y)
-    
-    tier_clfs[t] = clf
-    tier_calibs[t] = cal
-    
-    p_cal = cal.transform(p_raw)
-    brier = np.mean((p_cal - y) ** 2)
-    
-    print(f"  Tier {t}: n_pos={n_pos}, CV_f1={tier_cv_scores[t]:.3f}, Brier={brier:.4f}")
-
-# ─── Train Direct Optimal-Tier Predictor ────────────────────────────────
-print("\n[5] Training direct optimal-tier predictor...")
-
-from xgboost import XGBRegressor
-
-opt_clf = XGBClassifier(
-    n_estimators=300, max_depth=6, learning_rate=0.05,
-    subsample=0.8, colsample_bytree=0.8,
-    eval_metric='mlogloss', use_label_encoder=False,
-    random_state=42, num_class=5,
-)
-opt_clf.fit(X, y_optimal - 1)  # 0-indexed
-opt_pred = opt_clf.predict(X) + 1
-opt_acc = np.mean(opt_pred == y_optimal)
-print(f"  Direct optimal-tier accuracy: {opt_acc:.3f}")
-print(f"  Confusion (predicted vs actual):")
-from collections import Counter
-for actual_tier in range(1, 6):
-    mask = y_optimal == actual_tier
-    if mask.sum() > 0:
-        pred_dist = Counter(opt_pred[mask].tolist())
-        print(f"    Actual tier {actual_tier}: {dict(pred_dist)}")
-
-# ─── Evaluate on SWE-Router data ────────────────────────────────────────
-print("\n[6] Evaluating routing policies on SWE-Router...")
-
-from aco.classifier import TaskCostClassifier
-classifier = TaskCostClassifier()
-
-def route_v10(problem_text):
-    """v10: Real-data trained router."""
-    feats = extract_features(problem_text)
-    feat_vec = np.array([float(feats.get(k, 0.0)) for k in all_feat_keys], dtype=np.float32).reshape(1,-1)
-    
-    # Method 1: Direct optimal tier prediction
-    predicted_tier = int(opt_clf.predict(feat_vec)[0]) + 1
-    
-    # Method 2: Per-tier P(success) cascade
-    tier_probs = {}
-    for t in range(1, 6):
-        p_raw = tier_clfs[t].predict_proba(feat_vec)[0, 1]
-        p_cal = float(tier_calibs[t].transform([p_raw])[0])
-        tier_probs[t] = p_cal
-    
-    # Find cheapest tier with P(success) > threshold
-    for t in range(1, 6):
-        if tier_probs[t] >= 0.5:  # 50% success threshold
-            cascade_tier = t
-            break
-    else:
-        cascade_tier = 5
-    
-    return predicted_tier, cascade_tier, tier_probs
-
-# Evaluate
-TIER_TO_SWE = {
-    1: 'deepseek-v4-flash', 2: 'gpt-5-mini',
-    3: 'gemini-2.5-pro', 4: 'claude-opus-4.7', 5: 'gemini-3-pro',
-}
-
-policies = defaultdict(lambda: {"success":0,"cost":0.0,"n":0})
-
-for iid, model_results in traces.items():
-    problem = next(iter(model_results.values()))['problem']
-    
-    # Oracle
-    resolved = [(m, r) for m, r in model_results.items() if r['resolved']]
-    if resolved:
-        cheapest = min(resolved, key=lambda x: TIER_COST.get(MODEL_TIER[x[0]], 1.0))
-        policies['oracle']['success'] += 1
-        policies['oracle']['cost'] += cheapest[1]['cost']
-    else:
-        policies['oracle']['cost'] += min(r['cost'] for r in model_results.values())
-    policies['oracle']['n'] += 1
-    
-    # Always frontier (tier 4)
-    f_model = 'claude-opus-4.7'
-    if f_model in model_results:
-        policies['frontier']['success'] += int(model_results[f_model]['resolved'])
-        policies['frontier']['cost'] += model_results[f_model]['cost']
-    policies['frontier']['n'] += 1
-    
-    # v8 (old synthetic-trained router)
-    pred = classifier.classify(problem)
-    from aco.router import ModelCascadeRouter
-    old_router = ModelCascadeRouter(model_path="/app/router_models/router_bundle_v8.pkl")
-    r8 = old_router.route(problem, "coding", pred["difficulty"], pred)
-    m8 = TIER_TO_SWE.get(r8.tier, 'claude-opus-4.7')
-    if m8 in model_results:
-        policies['v8_synthetic']['success'] += int(model_results[m8]['resolved'])
-        policies['v8_synthetic']['cost'] += model_results[m8]['cost']
-    policies['v8_synthetic']['n'] += 1
-    
-    # v10 direct optimal-tier
-    predicted_tier, cascade_tier, tier_probs = route_v10(problem)
-    m10 = TIER_TO_SWE.get(predicted_tier, 'claude-opus-4.7')
-    if m10 in model_results:
-        policies['v10_direct']['success'] += int(model_results[m10]['resolved'])
-        policies['v10_direct']['cost'] += model_results[m10]['cost']
-    else:
-        # Fallback to frontier
-        policies['v10_direct']['success'] += int(model_results.get('claude-opus-4.7',{}).get('resolved',0))
-        policies['v10_direct']['cost'] += model_results.get('claude-opus-4.7',{}).get('cost',0.3)
-    policies['v10_direct']['n'] += 1
-    
-    # v10 cascade (per-tier P(success) > 0.5)
-    m10c = TIER_TO_SWE.get(cascade_tier, 'claude-opus-4.7')
-    if m10c in model_results:
-        policies['v10_cascade']['success'] += int(model_results[m10c]['resolved'])
-        policies['v10_cascade']['cost'] += model_results[m10c]['cost']
-    else:
-        policies['v10_cascade']['success'] += int(model_results.get('claude-opus-4.7',{}).get('resolved',0))
-        policies['v10_cascade']['cost'] += model_results.get('claude-opus-4.7',{}).get('cost',0.3)
-    policies['v10_cascade']['n'] += 1
-    
-    # Always cheap (tier 1)
-    c_model = 'deepseek-v4-flash'
-    if c_model in model_results:
-        policies['always_cheap']['success'] += int(model_results[c_model]['resolved'])
-        policies['always_cheap']['cost'] += model_results[c_model]['cost']
-    policies['always_cheap']['n'] += 1
-
-# Print results
-print(f"\n\n{'='*80}")
-print("REAL SWE-BENCH RESULTS WITH v10 REAL-DATA ROUTER")
-print(f"{'='*80}")
-
-fr_cost = policies['frontier']['cost'] / policies['frontier']['n']
-print(f"\n{'Policy':<20} {'Success':>10} {'AvgCost':>10} {'CostRed':>10}")
-print("-"*50)
-for name in ['oracle','v10_direct','v10_cascade','v8_synthetic','frontier','always_cheap']:
-    r = policies[name]
-    sr = r['success']/r['n']
-    ac = r['cost']/r['n']
-    cr = (1 - ac/fr_cost)*100
-    print(f"{name:<20} {sr:>10.3f} {ac:>10.4f} {cr:>9.1f}%")
-
-# Also try hybrid v10 + feedback
-# v10 routes to cheap model; if it fails, escalate
-print("\n\n[7] v10 + feedback cascade...")
-policies_hybrid = defaultdict(lambda: {"success":0,"cost":0.0,"n":0})
-
-for iid, model_results in traces.items():
-    problem = next(iter(model_results.values()))['problem']
-    predicted_tier, cascade_tier, tier_probs = route_v10(problem)
-    
-    # Start with cascade_tier (more conservative than direct)
-    m_cascade = TIER_TO_SWE.get(cascade_tier, 'claude-opus-4.7')
-    
-    if m_cascade in model_results and model_results[m_cascade]['resolved']:
-        # Initial model succeeded
-        policies_hybrid['v10_feedback']['success'] += 1
-        policies_hybrid['v10_feedback']['cost'] += model_results[m_cascade]['cost']
-    elif cascade_tier < 5:
-        # Failed: escalate
-        up_tier = min(cascade_tier + 1, 5)
-        up_model = TIER_TO_SWE.get(up_tier, 'claude-opus-4.7')
-        if up_model in model_results and model_results[up_model]['resolved']:
-            policies_hybrid['v10_feedback']['success'] += 1
-            policies_hybrid['v10_feedback']['cost'] += model_results[m_cascade]['cost']
-            policies_hybrid['v10_feedback']['cost'] += model_results[up_model]['cost']
-        else:
-            # Try tier 4 (frontier) as last resort
-            f_model = 'claude-opus-4.7'
-            if f_model in model_results and model_results[f_model]['resolved']:
-                policies_hybrid['v10_feedback']['success'] += 1
-                policies_hybrid['v10_feedback']['cost'] += model_results[m_cascade]['cost']
-                policies_hybrid['v10_feedback']['cost'] += model_results[f_model]['cost']
-            else:
-                policies_hybrid['v10_feedback']['cost'] += model_results[m_cascade]['cost']
-    else:
-        policies_hybrid['v10_feedback']['cost'] += model_results.get(m_cascade, {}).get('cost', 0.3)
-    policies_hybrid['v10_feedback']['n'] += 1
-    
-    # Also track oracle
-    resolved = [(m, r) for m, r in model_results.items() if r['resolved']]
-    if resolved:
-        cheapest = min(resolved, key=lambda x: TIER_COST.get(MODEL_TIER[x[0]], 1.0))
-        policies_hybrid['oracle']['success'] += 1
-        policies_hybrid['oracle']['cost'] += cheapest[1]['cost']
-    policies_hybrid['oracle']['n'] += 1
-
-    # Frontier
-    f_model = 'claude-opus-4.7'
-    policies_hybrid['frontier']['success'] += int(model_results[f_model]['resolved'])
-    policies_hybrid['frontier']['cost'] += model_results[f_model]['cost']
-    policies_hybrid['frontier']['n'] += 1
-
-fr_cost_h = policies_hybrid['frontier']['cost'] / policies_hybrid['frontier']['n']
-print(f"\n{'Policy':<20} {'Success':>10} {'AvgCost':>10} {'CostRed':>10}")
-print("-"*50)
-for name in ['oracle','v10_feedback','frontier']:
-    r = policies_hybrid[name]
-    sr = r['success']/r['n']
-    ac = r['cost']/r['n']
-    cr = (1-ac/fr_cost_h)*100
-    print(f"{name:<20} {sr:>10.3f} {ac:>10.4f} {cr:>9.1f}%")
-
-# Save v10 bundle
-v10_bundle = {
-    'tier_clfs': {str(k):v for k,v in tier_clfs.items()},
-    'tier_calibrators': {str(k):v for k,v in tier_calibs.items()},
-    'opt_clf': opt_clf,
-    'feat_keys': all_feat_keys,
-    'tier_config': {str(k):v for k,v in TIER_COST.items()},
-    'version': '10.0',
-    'description': 'ACO v10: Trained on REAL SWE-Router execution data (500 tasks x 8 models)',
-    'training_data': 'SWE-Router/swebench-verified-*',
-    'n_training': len(training_data),
-    'n_features': len(all_feat_keys),
-}
-with open('/app/router_models/router_bundle_v10.pkl', 'wb') as f:
-    pickle.dump(v10_bundle, f)
-print(f"\nSaved router_bundle_v10.pkl ({os.path.getsize('/app/router_models/router_bundle_v10.pkl')/1024:.0f} KB)")
-
-# Save results
-all_results = {}
-for name, r in policies.items():
-    all_results[name] = {"success":r['success']/r['n'],"avg_cost":r['cost']/r['n']}
-for name, r in policies_hybrid.items():
-    all_results[f"hybrid_{name}"] = {"success":r['success']/r['n'],"avg_cost":r['cost']/r['n']}
-all_results['v10_cv_scores'] = tier_cv_scores
-all_results['v10_opt_acc'] = opt_acc
-all_results['feat_keys'] = all_feat_keys
-with open('/app/swe_v10_results.json', 'w') as f:
-    json.dump(all_results, f, indent=2, default=str)
-
-print(f"\nSaved swe_v10_results.json")
-print("DONE!")