Upload VERIFIED_METRICS.md

VERIFIED_METRICS.md

@@ -1,79 +1,62 @@
 # Verified Metrics — Agent Cost Optimizer v2 Specialists
 
-**Generated 2026-06-03 by independent verification (verify_v2.py)**
-**Job: 6a207319e6aa50b87b9e8d26 on a10g-large**
+**Generated 2026-06-03 by independent verification job `6a207319e6aa50b87b9e8d26` on a10g-large.**
+**Raw results: `v2_verification_results.json` in this repo.**
 
-## Summary
+## v2 vs v1 Comparison
 
-| Model | Architecture | Params | Task | v2 Accuracy | v2 F1 (macro) | v1 Accuracy | v1 F1 (macro) | Delta F1 | Collapsed? |
-|---|---|---|---|---|---|---|---|---|---|
-| tier-router-v2 | ModernBERT-base | 149.6M | 3-class difficulty | *PENDING* | *PENDING* | *PENDING* | *PENDING* | *PENDING* | — |
-| tool-gater-v2 | ModernBERT-base | 149.6M | binary tool need | *PENDING* | *PENDING* | *PENDING* | *PENDING* | *PENDING* | — |
-| verifier-gater-v2 | ModernBERT-base | 149.6M | binary patch verification | *PENDING* | *PENDING* | *PENDING* | *PENDING* | *PENDING* | — |
+| Model | Arch | Params | v1 F1 | v2 F1 | Delta | v1 Acc | v2 Acc | Collapsed? |
+|---|---|---|---|---|---|---|---|---|
+| tier-router | ModernBERT/DistilBERT | 149M/67M | 0.670 | 0.677 | +0.007 | 70.1% | 69.2% | No |
+| tool-gater | ModernBERT/DistilBERT | 149M/67M | 0.920 | 0.724 | **-0.196** | 98.2% | 91.7% | No |
+| verifier-gater | ModernBERT/DistilBERT | 149M/67M | 0.654 | 0.667 | +0.013 | 65.5% | 66.7% | No |
 
-*Results will be updated when job 6a207319e6aa50b87b9e8d26 completes.*
+## Per-Class Breakdown
 
-## Training Data
-
-| Split | Source | Train Rows | Test Rows | Classes |
+### tier-router-v2 (3-class, 1,260 test samples)
+| Class | Precision | Recall | F1 | Support |
 |---|---|---|---|---|
-| tool_gater | lockon/ToolACE | 9,605 | 1,695 | 5.6% pos |
-| tier_router | RouteWorks/RouterArena | 7,140 | 1,260 | 3-class balanced |
-| verifier_gater | R2E-Gym/R2EGym-Verifier-Trajectories | 4,887 | 863 | 47.0% pos |
+| easy (0) | 0.738 | 0.799 | 0.768 | 593 |
+| medium (1) | 0.534 | 0.554 | 0.544 | 359 |
+| hard (2) | 0.812 | 0.646 | 0.720 | 308 |
 
-Full trace dataset: https://huggingface.co/datasets/narcolepticchicken/aco-traces
+### tool-gater-v2 (binary, 1,695 test, 5.6% pos)
+| Class | Precision | Recall | F1 | Support |
+|---|---|---|---|---|
+| no_tool (0) | 0.977 | 0.934 | 0.955 | 1,592 |
+| call_tool (1) | 0.393 | 0.660 | 0.493 | 103 |
 
-## Training Configuration (all v2 models)
+**Note: v1 tool-gater achieves F1=0.920 (pos P=0.884, R=0.816). Use v1.**
 
-- Base: ModernBERT-base (answerdotai/ModernBERT-base)
-- Loss: Focal loss (γ=2.0, α from class frequencies)
-- Optimizer: AdamW, lr=2e-5
-- Epochs: 10 with early stopping (patience=4)
-- Context: 2,048 tokens
-- Dropout: 0.1 on attention, MLP, embedding, classifier
-- Threshold calibration: grid search on validation set
+### verifier-gater-v2 (binary, 863 test, 44.6% pos)
+| Class | Precision | Recall | F1 | Support |
+|---|---|---|---|---|
+| fail (0) | 0.778 | 0.559 | 0.650 | 478 |
+| pass (1) | 0.594 | 0.803 | 0.683 | 385 |
 
-## Known Limitations
+Threshold: 0.52 (calibrated). Not collapsed — predicts both classes.
 
-### verifier-gater-v2
-The R2E-Gym feature extractor captures only ~600 chars of github issue + 
-~600 chars of agent action summary + ~400 chars of patch. This is
-insufficient to distinguish successful from failed patches. The
-verifier-gater alpha values [0.95, 1.05] indicate near-zero discriminative 
-signal in these feature snippets.
+## Verdict
 
-**Root cause:** The training features lack the full execution trace
-(test pass/fail, error messages) that would enable verifier-gater to
-discriminate successful from failed patches.
+- **tier-router-v2**: No improvement. Use v1 or either interchangeably.
+- **tool-gater-v2**: **REGRESSION.** ModernBERT overfits on 5.6% minority class. **Use v1** (`narcolepticchicken/aco-specialists-tool-gater`, F1=0.920).
+- **verifier-gater-v2**: Marginal improvement (+1.3pp). Neither model is production-ready. Features are too sparse.
 
-**Fix needed:** Enrich features with execution results, test outputs,
-and error traces before retraining.
+## Training Data (published)
 
-### tool-gater-v2
-5.6% positive rate — extreme imbalance. Focal loss with class-alpha 
-helps but the model may still struggle more than the ~87% F1 target.
+All preprocessing identical to training:
+- `narcolepticchicken/aco-traces/data/tier_router/{train,test}.parquet`
+- `narcolepticchicken/aco-traces/data/tool_gater/{train,test}.parquet`  
+- `narcolepticchicken/aco-traces/data/verifier_gater/{train,test}.parquet`
 
-### tier-router-v2
-Most promising model. RouterArena Difficulty labels provide clean signal.
-Previous run showed acc=69.2%, F1=0.677 on 3-class. The config has
-alpha values [0.641, 1.044, 1.315] indicating class imbalance was
-handled.
+Source: lockon/ToolACE, RouteWorks/RouterArena, R2E-Gym/R2EGym-Verifier-Trajectories.
 
 ## Reproducibility
 
-All training data and preprocessing are published:
-1. Source datasets: lockon/ToolACE, RouteWorks/RouterArena, R2E-Gym/R2EGym-Verifier-Trajectories
-2. Preprocessed splits: narcolepticchicken/aco-traces
-3. Preprocessing code: verify_v2.py loaders (identical to training)
-4. Verification script: verify_v2.py
-
-To reproduce training:
-```python
-# 1. Get the data
-from datasets import load_dataset
-ds = load_dataset("narcolepticchicken/aco-traces", "tier_router")
-
-# 2. Train with focal loss
-from transformers import AutoModelForSequenceClassification, Trainer, TrainingArguments
-# ... (see training/simple_strategies.py for full recipe)
+```bash
+# Download preprocessed data
+git clone https://huggingface.co/datasets/narcolepticchicken/aco-traces
+
+# Run verification (requires GPU for ModernBERT)
+uv run --with transformers,torch,datasets,scikit-learn https://huggingface.co/narcolepticchicken/agent-cost-optimizer/resolve/main/verify_v2.py
 ```