Upload DEPLOYMENT.md

DEPLOYMENT.md

@@ -0,0 +1,330 @@
+# ACO Deployment Guide
+
+## What ACO Is
+
+ACO (Agent Cost Optimizer) is a middleware proxy that sits between your agent and LLM providers. It reduces cost by:
+
+1. **Model routing** — cheapest adequate model per request
+2. **Tool gating** — suppresses unnecessary tool calls (DistilBERT F1=0.92)
+3. **Context compression** — trims verbose traces/error logs
+4. **Cache-aware layout** — reorders prompts for provider prefix-cache discounts
+5. **Telemetry** — live dashboard + JSON API for cost tracking
+
+## Quick Start
+
+### Option 1: Run as a proxy (zero agent code changes)
+
+```bash
+pip install aco-proxy
+# Or: pip install fastapi uvicorn httpx openai
+
+# Set provider API keys
+export OPENAI_API_KEY=sk-...
+export ANTHROPIC_API_KEY=sk-ant-...
+export DEEPSEEK_API_KEY=sk-...
+export GOOGLE_API_KEY=AI...
+
+# Start the proxy
+aco-proxy --port 8080
+```
+
+Then point your agent at the proxy:
+```python
+import openai
+client = openai.OpenAI(base_url="http://localhost:8080/v1", api_key="your-key")
+```
+
+That's it. All LLM calls now go through ACO optimizations.
+
+### Option 2: Run via Docker
+
+```bash
+docker run -p 8080:8080 \
+  -e OPENAI_API_KEY=sk-... \
+  -e ANTHROPIC_API_KEY=sk-ant-... \
+  ghcr.io/narcolepticchicken/aco-proxy:latest
+```
+
+### Option 3: Run as HF Space
+
+Deploy the proxy as a Hugging Face Space for a hosted dashboard:
+
+```bash
+# Create a Space with Docker SDK
+huggingface-cli repo create aco-proxy --type space --sdk docker
+```
+
+## Configuration
+
+### Model Registry
+
+ACO's model registry maps models to cost tiers. Edit `aco/proxy.py` to add/remove models:
+
+```python
+MODEL_REGISTRY = {
+    "deepseek-v4-flash":  {"tier": 1, "cost_in": 0.14,  "cost_out": 0.28,  "ctx": 128000},
+    "gpt-5-mini":          {"tier": 2, "cost_in": 0.15,  "cost_out": 0.60,  "ctx": 128000},
+    "gemini-2.5-pro":      {"tier": 3, "cost_in": 1.25,  "cost_out": 10.00, "ctx": 1048576},
+    # Add your models here
+}
+```
+
+### Provider Endpoints
+
+```python
+PROVIDER_ENDPOINTS = {
+    "openai":    "https://api.openai.com/v1",
+    "anthropic": "https://api.anthropic.com/v1",
+    "google":    "https://generativelanguage.googleapis.com/v1beta",
+    "deepseek":  "https://api.deepseek.com/v1",
+}
+```
+
+### Environment Variables
+
+| Variable | Purpose | Default |
+|---|---|---|
+| `OPENAI_API_KEY` | OpenAI provider key | Required |
+| `ANTHROPIC_API_KEY` | Anthropic provider key | Optional |
+| `DEEPSEEK_API_KEY` | DeepSeek provider key | Optional |
+| `GOOGLE_API_KEY` | Google provider key | Optional |
+| `OPENAI_BASE_URL` | Custom OpenAI endpoint | `https://api.openai.com/v1` |
+| `ACO_PORT` | Proxy port | `8080` |
+| `ACO_HOST` | Proxy host | `0.0.0.0` |
+
+## API Endpoints
+
+### `POST /v1/chat/completions`
+OpenAI-compatible. Pass any model from the registry; ACO routes to the cheapest adequate model.
+
+### `GET /dashboard`
+Live HTML dashboard showing:
+- Total calls, cost, success rate
+- Per-model call distribution
+- Cache hit rate, tool gates, model reroutes
+- Recent request log (last 30 calls)
+
+### `GET /telemetry`
+JSON telemetry for programmatic consumption:
+```json
+{
+  "total_calls": 42,
+  "total_cost": 0.0234,
+  "calls": [{"model": "gpt-5-mini", "tier": 2, "cost": 0.0005, ...}]
+}
+```
+
+### `GET /telemetry/reset`
+Clear all telemetry data.
+
+### `GET /health`
+Health check endpoint.
+
+### `GET /v1/models`
+List available models in OpenAI format.
+
+## How Routing Works
+
+```
+Request comes in with model="gemini-2.5-pro" (tier 3)
+                    │
+                    ▼
+         ┌─────────────────────┐
+         │  Extract user text  │
+         └─────────┬───────────┘
+                   │
+                   ▼
+         ┌─────────────────────┐
+         │  Is text < 300 chars │─── Yes ──→ Route to tier 1 (deepseek-v4-flash)
+         │  AND tier >= 3?      │
+         └─────────┬───────────┘
+                   │ No
+                   ▼
+         ┌─────────────────────┐
+         │  Coding keywords?    │─── Yes ──→ Keep tier 2 minimum
+         │  (def, class, fix)   │
+         └─────────┬───────────┘
+                   │ No
+                   ▼
+              Pass through (no routing change)
+```
+
+## How Tool Gating Works
+
+The tool-gater is a DistilBERT classifier (F1=0.92) trained on ToolACE + RouterArena data.
+
+```
+Request has tools=[search, calculator, ...]
+                    │
+                    ▼
+         ┌─────────────────────┐
+         │  Check conversation  │
+         │  history for prior   │─── Has tool history ──→ Don't gate
+         │  tool calls          │
+         └─────────┬───���───────┘
+                   │ No prior tools
+                   ▼
+         ┌─────────────────────┐
+         │  Run DistilBERT     │
+         │  classifier on      │
+         │  user query         │
+         └─────────┬───────────┘
+                   │
+                   ▼
+         P(skip_tool) > P(call_tool)?
+                   │
+              Yes ─┴─ No
+              │       │
+              ▼       ▼
+        Gate tools  Keep tools
+        (remove     (pass to
+         tools param) upstream)
+```
+
+## Integration Examples
+
+### With LangChain
+
+```python
+from langchain_openai import ChatOpenAI
+
+llm = ChatOpenAI(
+    model="gpt-5-mini",
+    openai_api_base="http://localhost:8080/v1",
+    openai_api_key="your-key"
+)
+```
+
+### With CrewAI
+
+```python
+from crewai import Agent
+
+agent = Agent(
+    llm=ChatOpenAI(
+        model="gpt-5-mini",
+        base_url="http://localhost:8080/v1",
+        api_key="your-key"
+    )
+)
+```
+
+### With AutoGen
+
+```python
+from autogen import ConversableAgent
+
+agent = ConversableAgent(
+    "assistant",
+    llm_config={
+        "model": "gpt-5-mini",
+        "api_base": "http://localhost:8080/v1",
+        "api_key": "your-key"
+    }
+)
+```
+
+### With raw HTTP
+
+```bash
+curl http://localhost:8080/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "gpt-5-mini",
+    "messages": [{"role": "user", "content": "What is 2+2?"}]
+  }'
+```
+
+## Supported Providers
+
+| Provider | Models | Key Env Var |
+|---|---|---|
+| OpenAI | gpt-5-mini, gpt-5.2, gpt-5-nano | `OPENAI_API_KEY` |
+| Anthropic | claude-opus-4.7 | `ANTHROPIC_API_KEY` |
+| Google | gemini-2.5-flash, gemini-2.5-pro, gemini-3-pro | `GOOGLE_API_KEY` |
+| DeepSeek | deepseek-v4-flash, deepseek-v3.2 | `DEEPSEEK_API_KEY` |
+
+## Supported Agent Types
+
+- **Coding agents** (SWE-bench style) — routing floor at tier 2
+- **Research agents** — routing to tier 2-3 with retrieval
+- **RAG agents** — context compression + cache layout
+- **Tool-use agents** — ML tool gating
+- **Legal/security agents** — verifier always on for high-risk
+- **Personal assistants** — aggressive cost reduction
+
+## Performance
+
+Benchmarked on 100 simulated tasks across 5 domains:
+
+| Config | Success | Cost | Savings |
+|---|---|---|---|
+| Always frontier | 89% | $10.79 | baseline |
+| Always cheap | 61% | $0.11 | 99% (but -28pp quality) |
+| **Full ACO** | **91%** | **$1.56** | **85.5%** |
+
+Full ACO achieves **iso-quality** (actually +2pp better) at 85.5% cost reduction.
+
+## Modules That Matter (Ablation Results)
+
+| Module | Impact if Removed | Verdict |
+|---|---|---|
+| Model router | -13pp quality | CRITICAL |
+| Verifier budgeter | -8pp quality | CRITICAL |
+| Retry optimizer | -8pp quality | CRITICAL |
+| Cache layout | +1.8% cost | SAVES MONEY |
+| Tool gate | +2.5% cost | SAVES MONEY |
+| Context budgeter | +0.9% cost, +2pp quality | MARGINAL |
+| Meta-tools | -1pp quality | MARGINAL |
+
+## Limitations
+
+1. **Tool-gater is the only production-ready specialist** (F1=0.92). Tier-router (F1=0.67) and verifier-gater (F1=0.65) are too weak to deploy.
+2. **Proxy is untested against live LLM APIs** — all validation is simulated.
+3. **Model prices are hardcoded** — update `MODEL_REGISTRY` when providers change pricing.
+4. **No streaming tool-gate** — tool gating is skipped for streaming requests.
+5. **No multi-turn cascade** — retry cascade only escalates once.
+
+## Troubleshooting
+
+### "No module named 'transformers'"
+The tool-gater needs transformers + torch. Install:
+```bash
+pip install transformers torch
+```
+Or run without ML gating (heuristic fallback works).
+
+### "Upstream 401: Unauthorized"
+Check that your API keys are set:
+```bash
+echo $OPENAI_API_KEY
+```
+
+### "Upstream 404: Model not found"
+The model name in your request must match the provider's API. Check `MODEL_REGISTRY` in `aco/proxy.py`.
+
+### Dashboard shows no data
+Send at least one request to `/v1/chat/completions` first. The dashboard auto-refreshes every 3 seconds.
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `aco/proxy.py` | Main proxy server (FastAPI) |
+| `aco/router.py` | Model cascade router |
+| `aco/cascade.py` | Cascade router with fallback |
+| `aco/compression.py` | Context compression |
+| `aco/telemetry.py` | Telemetry collector |
+| `aco/tool_gater.py` | ML tool-gating classifier |
+| `benchmark_suite.py` | Simulated benchmark (9 configs × 100 tasks) |
+| `ablation_study.py` | Ablation + frontier report |
+| `smoke_test_proxy.py` | End-to-end proxy smoke test |
+| `train_aco.py` | Reproducible training pipeline |
+| `config.yaml` | Policy configuration |
+
+## Links
+
+- **Hub repo**: https://huggingface.co/narcolepticchicken/agent-cost-optimizer
+- **Tool-gater model**: https://huggingface.co/narcolepticchicken/aco-specialists-tool-gater
+- **Training data**: https://huggingface.co/datasets/narcolepticchicken/aco-traces
+- **Truth document**: https://huggingface.co/narcolepticchicken/agent-cost-optimizer/blob/main/TRUTH.md