# Verified Metrics — Agent Cost Optimizer v2 Specialists

**Generated 2026-06-03 by independent verification job `6a207319e6aa50b87b9e8d26` on a10g-large.**
**Raw results: `v2_verification_results.json` in this repo.**

## v2 vs v1 Comparison

| Model | Arch | Params | v1 F1 | v2 F1 | Delta | v1 Acc | v2 Acc | Collapsed? |
|---|---|---|---|---|---|---|---|---|
| tier-router | ModernBERT/DistilBERT | 149M/67M | 0.670 | 0.677 | +0.007 | 70.1% | 69.2% | No |
| tool-gater | ModernBERT/DistilBERT | 149M/67M | 0.920 | 0.724 | **-0.196** | 98.2% | 91.7% | No |
| verifier-gater | ModernBERT/DistilBERT | 149M/67M | 0.654 | 0.667 | +0.013 | 65.5% | 66.7% | No |

## Per-Class Breakdown

### tier-router-v2 (3-class, 1,260 test samples)
| Class | Precision | Recall | F1 | Support |
|---|---|---|---|---|
| easy (0) | 0.738 | 0.799 | 0.768 | 593 |
| medium (1) | 0.534 | 0.554 | 0.544 | 359 |
| hard (2) | 0.812 | 0.646 | 0.720 | 308 |

### tool-gater-v2 (binary, 1,695 test, 5.6% pos)
| Class | Precision | Recall | F1 | Support |
|---|---|---|---|---|
| no_tool (0) | 0.977 | 0.934 | 0.955 | 1,592 |
| call_tool (1) | 0.393 | 0.660 | 0.493 | 103 |

**Note: v1 tool-gater achieves F1=0.920 (pos P=0.884, R=0.816). Use v1.**

### verifier-gater-v2 (binary, 863 test, 44.6% pos)
| Class | Precision | Recall | F1 | Support |
|---|---|---|---|---|
| fail (0) | 0.778 | 0.559 | 0.650 | 478 |
| pass (1) | 0.594 | 0.803 | 0.683 | 385 |

Threshold: 0.52 (calibrated). Not collapsed — predicts both classes.

## Verdict

- **tier-router-v2**: No improvement. Use v1 or either interchangeably.
- **tool-gater-v2**: **REGRESSION.** ModernBERT overfits on 5.6% minority class. **Use v1** (`narcolepticchicken/aco-specialists-tool-gater`, F1=0.920).
- **verifier-gater-v2**: Marginal improvement (+1.3pp). Neither model is production-ready. Features are too sparse.

## Training Data (published)

All preprocessing identical to training:
- `narcolepticchicken/aco-traces/data/tier_router/{train,test}.parquet`
- `narcolepticchicken/aco-traces/data/tool_gater/{train,test}.parquet`  
- `narcolepticchicken/aco-traces/data/verifier_gater/{train,test}.parquet`

Source: lockon/ToolACE, RouteWorks/RouterArena, R2E-Gym/R2EGym-Verifier-Trajectories.

## Reproducibility

```bash
# Download preprocessed data
git clone https://huggingface.co/datasets/narcolepticchicken/aco-traces

# Run verification (requires GPU for ModernBERT)
uv run --with transformers,torch,datasets,scikit-learn https://huggingface.co/narcolepticchicken/agent-cost-optimizer/resolve/main/verify_v2.py
```