# ACO Deployment Guide

## What ACO Is

ACO (Agent Cost Optimizer) is a middleware proxy that sits between your agent and LLM providers. It reduces cost by:

1. **Model routing** — cheapest adequate model per request
2. **Tool gating** — suppresses unnecessary tool calls (DistilBERT F1=0.92)
3. **Context compression** — trims verbose traces/error logs
4. **Cache-aware layout** — reorders prompts for provider prefix-cache discounts
5. **Telemetry** — live dashboard + JSON API for cost tracking

## Quick Start

### Option 1: Run as a proxy (zero agent code changes)

```bash
pip install aco-proxy
# Or: pip install fastapi uvicorn httpx openai

# Set provider API keys
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export DEEPSEEK_API_KEY=sk-...
export GOOGLE_API_KEY=AI...

# Start the proxy
aco-proxy --port 8080
```

Then point your agent at the proxy:
```python
import openai
client = openai.OpenAI(base_url="http://localhost:8080/v1", api_key="your-key")
```

That's it. All LLM calls now go through ACO optimizations.

### Option 2: Run via Docker

```bash
docker run -p 8080:8080 \
  -e OPENAI_API_KEY=sk-... \
  -e ANTHROPIC_API_KEY=sk-ant-... \
  ghcr.io/narcolepticchicken/aco-proxy:latest
```

### Option 3: Run as HF Space

Deploy the proxy as a Hugging Face Space for a hosted dashboard:

```bash
# Create a Space with Docker SDK
huggingface-cli repo create aco-proxy --type space --sdk docker
```

## Configuration

### Model Registry

ACO's model registry maps models to cost tiers. Edit `aco/proxy.py` to add/remove models:

```python
MODEL_REGISTRY = {
    "deepseek-v4-flash":  {"tier": 1, "cost_in": 0.14,  "cost_out": 0.28,  "ctx": 128000},
    "gpt-5-mini":          {"tier": 2, "cost_in": 0.15,  "cost_out": 0.60,  "ctx": 128000},
    "gemini-2.5-pro":      {"tier": 3, "cost_in": 1.25,  "cost_out": 10.00, "ctx": 1048576},
    # Add your models here
}
```

### Provider Endpoints

```python
PROVIDER_ENDPOINTS = {
    "openai":    "https://api.openai.com/v1",
    "anthropic": "https://api.anthropic.com/v1",
    "google":    "https://generativelanguage.googleapis.com/v1beta",
    "deepseek":  "https://api.deepseek.com/v1",
}
```

### Environment Variables

| Variable | Purpose | Default |
|---|---|---|
| `OPENAI_API_KEY` | OpenAI provider key | Required |
| `ANTHROPIC_API_KEY` | Anthropic provider key | Optional |
| `DEEPSEEK_API_KEY` | DeepSeek provider key | Optional |
| `GOOGLE_API_KEY` | Google provider key | Optional |
| `OPENAI_BASE_URL` | Custom OpenAI endpoint | `https://api.openai.com/v1` |
| `ACO_PORT` | Proxy port | `8080` |
| `ACO_HOST` | Proxy host | `0.0.0.0` |

## API Endpoints

### `POST /v1/chat/completions`
OpenAI-compatible. Pass any model from the registry; ACO routes to the cheapest adequate model.

### `GET /dashboard`
Live HTML dashboard showing:
- Total calls, cost, success rate
- Per-model call distribution
- Cache hit rate, tool gates, model reroutes
- Recent request log (last 30 calls)

### `GET /telemetry`
JSON telemetry for programmatic consumption:
```json
{
  "total_calls": 42,
  "total_cost": 0.0234,
  "calls": [{"model": "gpt-5-mini", "tier": 2, "cost": 0.0005, ...}]
}
```

### `GET /telemetry/reset`
Clear all telemetry data.

### `GET /health`
Health check endpoint.

### `GET /v1/models`
List available models in OpenAI format.

## How Routing Works

```
Request comes in with model="gemini-2.5-pro" (tier 3)
                    │
                    ▼
         ┌─────────────────────┐
         │  Extract user text  │
         └─────────┬───────────┘
                   │
                   ▼
         ┌─────────────────────┐
         │  Is text < 300 chars │─── Yes ──→ Route to tier 1 (deepseek-v4-flash)
         │  AND tier >= 3?      │
         └─────────┬───────────┘
                   │ No
                   ▼
         ┌─────────────────────┐
         │  Coding keywords?    │─── Yes ──→ Keep tier 2 minimum
         │  (def, class, fix)   │
         └─────────┬───────────┘
                   │ No
                   ▼
              Pass through (no routing change)
```

## How Tool Gating Works

The tool-gater is a DistilBERT classifier (F1=0.92) trained on ToolACE + RouterArena data.

```
Request has tools=[search, calculator, ...]
                    │
                    ▼
         ┌─────────────────────┐
         │  Check conversation  │
         │  history for prior   │─── Has tool history ──→ Don't gate
         │  tool calls          │
         └─────────┬───────────┘
                   │ No prior tools
                   ▼
         ┌─────────────────────┐
         │  Run DistilBERT     │
         │  classifier on      │
         │  user query         │
         └─────────┬───────────┘
                   │
                   ▼
         P(skip_tool) > P(call_tool)?
                   │
              Yes ─┴─ No
              │       │
              ▼       ▼
        Gate tools  Keep tools
        (remove     (pass to
         tools param) upstream)
```

## Integration Examples

### With LangChain

```python
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="gpt-5-mini",
    openai_api_base="http://localhost:8080/v1",
    openai_api_key="your-key"
)
```

### With CrewAI

```python
from crewai import Agent

agent = Agent(
    llm=ChatOpenAI(
        model="gpt-5-mini",
        base_url="http://localhost:8080/v1",
        api_key="your-key"
    )
)
```

### With AutoGen

```python
from autogen import ConversableAgent

agent = ConversableAgent(
    "assistant",
    llm_config={
        "model": "gpt-5-mini",
        "api_base": "http://localhost:8080/v1",
        "api_key": "your-key"
    }
)
```

### With raw HTTP

```bash
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5-mini",
    "messages": [{"role": "user", "content": "What is 2+2?"}]
  }'
```

## Supported Providers

| Provider | Models | Key Env Var |
|---|---|---|
| OpenAI | gpt-5-mini, gpt-5.2, gpt-5-nano | `OPENAI_API_KEY` |
| Anthropic | claude-opus-4.7 | `ANTHROPIC_API_KEY` |
| Google | gemini-2.5-flash, gemini-2.5-pro, gemini-3-pro | `GOOGLE_API_KEY` |
| DeepSeek | deepseek-v4-flash, deepseek-v3.2 | `DEEPSEEK_API_KEY` |

## Supported Agent Types

- **Coding agents** (SWE-bench style) — routing floor at tier 2
- **Research agents** — routing to tier 2-3 with retrieval
- **RAG agents** — context compression + cache layout
- **Tool-use agents** — ML tool gating
- **Legal/security agents** — verifier always on for high-risk
- **Personal assistants** — aggressive cost reduction

## Performance

Benchmarked on 100 simulated tasks across 5 domains:

| Config | Success | Cost | Savings |
|---|---|---|---|
| Always frontier | 89% | $10.79 | baseline |
| Always cheap | 61% | $0.11 | 99% (but -28pp quality) |
| **Full ACO** | **91%** | **$1.56** | **85.5%** |

Full ACO achieves **iso-quality** (actually +2pp better) at 85.5% cost reduction.

## Modules That Matter (Ablation Results)

| Module | Impact if Removed | Verdict |
|---|---|---|
| Model router | -13pp quality | CRITICAL |
| Verifier budgeter | -8pp quality | CRITICAL |
| Retry optimizer | -8pp quality | CRITICAL |
| Cache layout | +1.8% cost | SAVES MONEY |
| Tool gate | +2.5% cost | SAVES MONEY |
| Context budgeter | +0.9% cost, +2pp quality | MARGINAL |
| Meta-tools | -1pp quality | MARGINAL |

## Limitations

1. **Tool-gater is the only production-ready specialist** (F1=0.92). Tier-router (F1=0.67) and verifier-gater (F1=0.65) are too weak to deploy.
2. **Proxy is untested against live LLM APIs** — all validation is simulated.
3. **Model prices are hardcoded** — update `MODEL_REGISTRY` when providers change pricing.
4. **No streaming tool-gate** — tool gating is skipped for streaming requests.
5. **No multi-turn cascade** — retry cascade only escalates once.

## Troubleshooting

### "No module named 'transformers'"
The tool-gater needs transformers + torch. Install:
```bash
pip install transformers torch
```
Or run without ML gating (heuristic fallback works).

### "Upstream 401: Unauthorized"
Check that your API keys are set:
```bash
echo $OPENAI_API_KEY
```

### "Upstream 404: Model not found"
The model name in your request must match the provider's API. Check `MODEL_REGISTRY` in `aco/proxy.py`.

### Dashboard shows no data
Send at least one request to `/v1/chat/completions` first. The dashboard auto-refreshes every 3 seconds.

## Files

| File | Purpose |
|---|---|
| `aco/proxy.py` | Main proxy server (FastAPI) |
| `aco/router.py` | Model cascade router |
| `aco/cascade.py` | Cascade router with fallback |
| `aco/compression.py` | Context compression |
| `aco/telemetry.py` | Telemetry collector |
| `aco/tool_gater.py` | ML tool-gating classifier |
| `benchmark_suite.py` | Simulated benchmark (9 configs × 100 tasks) |
| `ablation_study.py` | Ablation + frontier report |
| `smoke_test_proxy.py` | End-to-end proxy smoke test |
| `train_aco.py` | Reproducible training pipeline |
| `config.yaml` | Policy configuration |

## Links

- **Hub repo**: https://huggingface.co/narcolepticchicken/agent-cost-optimizer
- **Tool-gater model**: https://huggingface.co/narcolepticchicken/aco-specialists-tool-gater
- **Training data**: https://huggingface.co/datasets/narcolepticchicken/aco-traces
- **Truth document**: https://huggingface.co/narcolepticchicken/agent-cost-optimizer/blob/main/TRUTH.md