# AUDIT.md — Independent Verification of ACO Claims

**Date: 2026-06-15** · Auditor: ML Intern (independent re-evaluation, not training logs)

This document records an independent audit of the claims in the ACO project. Method:
re-load artifacts from the Hub, recreate test splits from scratch
(`train_test_split(test_size=0.15, seed=42)` on the original source datasets), run fresh
inference, and recompute metrics with sklearn — deliberately **not** trusting any logged
training metric.

---

## Part A — Specialist classifier metrics: VERIFIED ✅

All three v1 DistilBERT specialists were re-evaluated on freshly recreated held-out splits.
GPU verification job: `6a1f651eb291489980136ba6` (COMPLETED). Tool-gater re-run cross-checked
against `v2_verification_results.json` (exact match).

| Model | Metric | Claimed (old) | Verified (re-run) | Verdict |
|-------|--------|---------------|-------------------|---------|
| Tool-Use Gater | Accuracy | 0.9503 | **0.9823** | ✅ Better than claimed |
| Tool-Use Gater | F1 Macro | 0.8649 | **0.9195** | ✅ Better than claimed |
| Tier Router | Accuracy | 0.7008 | **0.7016** | ✅ Matches |
| Tier Router | F1 Macro | 0.6730 | **0.6738** | ✅ Matches |
| Verifier Gater | Accuracy | 0.6559 | **0.6547** | ✅ Matches |
| Verifier Gater | F1 Macro | 0.6551 | **0.6539** | ✅ Matches |

Test set sizes: Tool-Use Gater N=1695, Tier Router N=1260, Verifier Gater N=863.

Majority-class baselines (proves real learning, not majority voting):

| Model | Majority baseline | Verified accuracy | Lift |
|-------|-------------------|-------------------|------|
| Tool-Use Gater | 93.9% | 98.2% | +4.3 pp |
| Tier Router | 47.1% | 70.2% | +23.1 pp |
| Verifier Gater | 55.4% | 65.5% | +10.1 pp |

**Correction applied:** the tool-gater model card and project summary previously quoted
~0.935 acc / 0.60 F1-macro (an early epoch-1 checkpoint, pre-oversampling). The pushed model
is the final checkpoint and is **better** than documented. Cards updated 2026-06-15.

**Architecture claims:** confirmed via `config.json` — all three are
`DistilBertForSequenceClassification`, n_layers=6, dim=768, vocab=30522, 67.0M params,
`model.safetensors` = 255.4 MB each. ✅

### v2 ModernBERT: regression confirmed ❌

`v2_verification_results.json` (job `6a207319`) shows the v2 upgrade did **not** help:

| Model | v1 F1 | v2 F1 | Delta | Verdict |
|-------|-------|-------|-------|---------|
| tier-router | 0.670 | 0.677 | +0.007 | flat (noise) |
| tool-gater | 0.920 | 0.724 | **−0.196** | **regression** |
| verifier-gater | 0.654 | 0.667 | +0.013 | flat (noise) |

Recommendation stands: **use v1**.

---

## Part B — Cost-savings, cascade, and Django claims

### B1. "Cascade ≈ 56% cost reduction at iso-quality" — ⚠️ SIMULATED, not test-verified

- **Source of the number:** a **trace-based simulation**. Costs come from a hardcoded pricing
  table and per-tier success-rate model (`aco/benchmark.py`), and from replaying SWE-Router
  dataset traces. There is **no live Docker test execution** behind the headline figure.
- **The repo already corrected its own overclaim.** `CORRECTED_REPORT.md` (Fix #1) shows that
  against a *fair* baseline (frontier-with-retry, equal retry budget), frontier-retry actually
  **solves 4 MORE instances** (420 vs 416), not fewer. The solve-rate difference is **not**
  statistically significant ([−2.8pp, +1.0pp], 1000-bootstrap). Cascade's real, defensible win
  is **cost** (~56% cheaper at statistically-equivalent quality), **not** higher solve rate.
- A prior "$585 provider-routing savings" figure was a **unit error**, corrected to $18.16 in
  `CORRECTED_REPORT.md` (Fix #3).
- **Verdict:** The cost-reduction claim is internally consistent and honestly caveated *within*
  the repo's own docs, but it is a **simulation result**. It should not be cited as a measured,
  live-evaluated SWE-bench result. The README has been updated to label it "simulated."

### B2. "4/4 Django instances solved by free-tier Llama-3.1-8B" — ⚠️ MISLEADING (true but weaker than it sounds)

- From `BATCH_VALIDATION_RESULTS.md` (job `6a04d3a3`, status ERROR — crashed at instance 5).
- What is true: 4 Django instances each produced a patch that passed **`git apply --check`**
  (i.e. syntactically valid diffs) from T1 in 3–6 turns.
- What is **not** true: these were **not** test-verified. The same doc states plainly:
  *"No test verification... The patches are syntactically valid diffs but may not fix the bug
  correctly."* FAIL_TO_PASS pytest was never run on them.
- **Verdict:** "4/4 valid patches" = 4/4 *apply-able* diffs, **not** 4/4 *resolved* SWE-bench
  instances. Do not read it as a 100% solve rate.

### B3. Live SWE-bench resolution by the cascade agent — ❌ NOT DEMONSTRATED

- The validation scripts (`validate_cascade.py`, `quick_validate.py`) contain *correct*
  end-to-end logic: clone repo → conda env → cascade agent (T1→T2) → apply model patch +
  test_patch → run FAIL_TO_PASS and PASS_TO_PASS via pytest.
- However, **no completed results artifact** with test-verified `resolved=True` exists in the
  repo (no populated `validation_results.jsonl` / `batch_results.jsonl`). `CORRECTED_REPORT.md`
  (Fix #4) confirms Docker/test verification was only "PARTIAL" and blocked by lack of a Docker
  daemon on HF infra.
- **Verdict:** The cascade is proven *in simulation* and the agent *produces* patches, but a
  genuine live SWE-bench *resolution* rate is **not** established anywhere in the repo.

### B4. Dataset existence — ✅ VERIFIED

- `narcolepticchicken/aco-traces` exists with `tool_gater`, `tier_router`, `verifier_gater`
  configs (train/test parquet). Preprocessing in `build_traces.py` is identical to the
  training/verification loaders. The earlier "aco-traces is empty" claim is **false now**.

---

## Bottom line

- **Classifier metrics (Part A): all TRUE and verified.** Tool-gater was *understated* and has
  been corrected upward. v2 ModernBERT is a confirmed regression — v1 is the production set.
- **Cost-savings (Part B): honest but must be labelled "simulated."** The 56% figure is a trace
  simulation, the "4/4 Django" is apply-check-only (not test-verified), and no live SWE-bench
  resolution rate exists. The repo's own TRUTH.md / CORRECTED_REPORT.md already disclose these
  caveats; the stale overclaims were in the README files, now corrected.

**Nothing fabricated. The main risk was stale/over-optimistic README framing, now fixed.**