# package-courier — agentic serving benchmark for opencoti-llamafile

A multi-agent, tool-calling benchmark that measures what actually matters
when an opencoti-llamafile engine serves a fleet of LangGraph agents:
end-to-end task throughput, per-package latency, speed vs context growth,
and the effect of **PolyKV shared-prefix pools** and the **tps-floor
worker scheduler** under real concurrency.

## The workload

A local **work server** generates P packages, each requiring a chain of
N verification steps. A worker agent must acquire a package
(`get_package`), walk the chain with attribute-lookup tools + `next_step`
(every step's result is verified server-side — no shortcuts), and
finally `deliver_package` with the correct final answer. Wrong answers,
lost packages, and timeouts all cost score. Each package is one
multi-turn tool-calling conversation, so context grows turn by turn —
exactly the agentic traffic shape (long shared system+tools prefix,
divergent per-session tails) that PolyKV pools target.

Everything is verified and scored by the work server, so the benchmark
is *correctness-gated*: an engine change that breaks tool-calling or
corrupts context shows up as lost score, not just different tok/s.

## Components

| file | role |
|---|---|
| `work_server.py` | package generator + verifier + scorer (HTTP, seeded, deterministic) |
| `agent_app.py` | LangGraph worker fleet: PolyKV pool pinning, dynamic tps-floor scheduler, per-turn telemetry |
| `run.py` | launcher: boots engine + work server + agents, collects the report |
| `report_html.py` | single-file HTML report (inline SVG, zero external refs — safe to attach anywhere) |

## Modes

- **pooled** (default): the chat-templated system+tools prefix (~1.5k
  tokens) is pinned once as a PolyKV pool; every session attaches and
  reuses it (`cache_n >= prefix_len` on every turn). Worker count is
  dynamic when `--tps-floor` is set: the scheduler grows the fleet
  while per-session tps holds above the floor and shrinks it when the
  engine saturates below it (live worker count and target are both
  plotted in the report).
- **naive** (`--no-polykv`): fixed worker fleet, no pool — every turn
  re-prefills its full prompt. This is the baseline that shows what
  prefix sharing is worth.

## Running

```bash
# pooled leg, dynamic workers, 100 packages x 10 steps
python run.py --engine-url http://127.0.0.1:8231 \
    --packages 100 --steps 10 --seed 1 \
    --workers 4 --max-workers 8 --tps-floor 15 --html \
    --out runs/pooled

# naive baseline: same workload, fixed 8 workers, no pools
python run.py --engine-url http://127.0.0.1:8231 \
    --packages 100 --steps 10 --seed 1 \
    --no-polykv --workers 8 --max-workers 8 --html \
    --out runs/naive
```

`run.py` can also boot the engine itself (`--model <gguf>` plus
pass-through `--ctx/--parallel/--pools/--ctk/--ctv`). Speculative
decode is first-class: `--spec-type draft-mtp` (NextN self-spec) or
`--spec-type draft-assistant --mtp-head <drafter.gguf>`, with
`--draft-n N` for the per-cycle draft depth. Anything else goes through
`--engine-args`.

Re-render the report from existing run data without re-benching:

```bash
python run.py --out runs/pooled --report-only
# or: python report_html.py runs/pooled/run
```

## Report

`report.html` is one self-contained file: score/latency/config tiles
plus six charts — speed vs chain step, speed vs context size, engine
timeline (mean active tps + sessions), cumulative deliveries, delivery
time per package, and worker target vs **live** workers. Landscape
layout, prints on A4.

## Benchmark sets published with opencoti-llamafile

1. **Pooled vs naive A/B** — same seed/workload/engine, 8-way
   concurrency: the value of PolyKV prefix sharing on agentic traffic.
2. **Stress ladder** — automatic worker count at tps floors 15 / 5 / 1,
   up to maximum PolyKV fan-out: how the engine degrades (and the
   scheduler adapts) as concurrency is pushed past saturation.