docs/features/package_courier_harness.md

# Package-courier harness — concurrency test + agentic performance benchmark

*Spec agreed 2026-07-16 (user design + AskUserQuestion decisions). Tasks
#739 (work server) / #740 (LangGraph app) / #741 (launcher + reporting).
Sibling of the #712 PolyKV control-plane API; consumes `opencoti-langgraph`.*

## Purpose

One harness that is simultaneously:

1. a **concurrency test** for the PolyKV control-plane (pools, fork,
   admission, capacity, tps) under real multi-session load;
2. a **live tool-calling exercise** of the engine's native
   `/v1/chat/completions` tool-call path (the coverage gap from the C4
   audit), multi-turn;
3. a repeatable **agentic performance benchmark** with a single scalar
   score (dollars) plus timing/throughput stats for graphs;
4. the **reference LangGraph implementation** for the
   `opencoti_langgraph` integration package (updated 2026-07-16,
   bug-2220): each worker episode is a compiled `langgraph.StateGraph`
   — `START → capacity_gate → agent ⇄ tools → END` with a bounded
   `hold` backoff loop — where `capacity_gate` is the packaged
   `make_capacity_gate` node, `agent` is one `OpencotiChatModel` call,
   and routing is `Command`-based over `add_messages` state. Offline
   routing tests: `bench/package_courier/test_agent_graph.py`. Live
   gates 2026-07-16 (Qwen3-4B, 3090): pooled 8/8 delivered 100%
   (cache_n>0 on 64/64 turns, scheduler 2→5 workers), naive 6/6 100%.

## The workload

A Python **work server** (runs in the `opencoti` conda env, part of the
harness) owns the ground truth:

- At start it **procedurally generates** `P` packages (default **100**),
  each with a chain of `n` dependent steps (default **10**). Every chain
  is unique in tool sequence and content mix. Steps are built from a
  vocabulary of fake attribute tools (`haircolor`, `cartype`, `petname`,
  …): step *k*'s instruction names the tool to call and embeds the
  answer of step *k−1* as input. Content is credible-but-nonsense —
  plausible enough that a model executes it without wondering why.
- Tools exposed to the agents (as LangChain tools hitting the work
  server's REST API; the model emits **native OpenAI tool_calls** via
  `bind_tools`/ToolNode):
  - `get_package()` → `package_id` + step-1 instructions (acquisition
    is the queue pull);
  - `<attribute>(package_id, n_step)` → the step's answer; the server
    **records every call and verifies** it is the right tool with the
    right args for that package/step;
  - `next_step(package_id, n_step, result)` → verifies the result
    (it is the input of the next step) and returns step *k+1*;
  - `deliver_package(package_id, final_answer)` → verified against the
    chain's final answer; correct delivery pays **$1**;
  - a completion signal endpoint the framework calls when all agents
    are done.
- **Timeout**: server ends the run if no package is delivered for
  15 min (default) and no completion signal arrived.

## The agent side (opencoti-langgraph app)

- Root **pool** = system prompt + tool instructions (chat-templated
  block). Workers attach with `pool_id` + per-worker `session_id`.
- **Naive mode** (`--no-polykv`): identical workload on plain sessions;
  worker count fixed at `K == --parallel`, FIFO package pull. This is
  the only shape naive supports — it is the A/B baseline.
- **PolyKV mode** (default): additionally supports a **dynamic worker
  target driven by `--tps-floor`** — the framework must oblige the
  floor: spawn additional workers only while
  `GET /capacity` admits (`can_admit`, `expected_tokens` arm) and
  `mean_active_tps` stays ≥ floor; back off when the floor is
  threatened. Enforced admission (`429 + Retry-After` →
  `PoolSaturatedError`) is honored as the hard backstop.
- The framework handles package acquisition, chain execution
  (each step = a new turn with a tool call), retries/prioritization
  (oldest-acquired first), monitors deliveries, and signals completion.

## Launcher & flags

The launcher boots all three processes: the opencoti-llamafile server
(model, `-c`, `--parallel`, `--polykv-max-pools`, KV types, MTP flags —
all pass-through), the work server, and the agent app.

Defaults (updated 2026-07-16, user decision + bug-2222): 100 packages ×
10 steps; engine `--parallel 8`; pooled worker fleet driven by
`--tps-floor 15` (fixed worker counts are only for the naive A/B leg,
where the floor defaults to 0); work-server stall timeout auto-scales
`max(900 s, steps × 60 s)` — the stall timer resets only on delivery,
so long chains (e.g. 55 steps) need the scaled window; model =
**Qwen3.6-27B-Omnimerge-v4 (bs2)** — hybrid GDN + NextN MTP, i.e. the
serving target and the hardest pool path; everything overridable.

## Scoring & report

- Score: delivered-correct × $1 (100 packages → $100 = 100%).
- Report: packages delivered / not delivered / **lost in transit** /
  failed-verification (and **at which step**), total wall time, delivery
  time mean/percentiles, per-step verification log.
- Engine stats: the harness polls `/polykv/tps` (per-session tps + ctx
  budget), `/polykv/pools/*/capacity` (headroom, `compaction_pressure`),
  `/slots`, `/metrics` on an interval → timeseries for graphs; the
  framework also reports its own API stats at the end.
- **Speed curves (the HF question — how does agent speed behave as turns
  and context grow):** every model turn is sampled with its chain step
  `n`, prompt size, `cache_n`, generation tps and prompt tps; the report
  aggregates **tps-vs-step** (mean tps at chain step 1..n across all
  packages) and **tps-vs-context** (512-token buckets). Pooled vs naive
  runs give the same curves side by side — i.e. whether/where per-turn
  speed decays with accumulated context, and what the pool buys back.
- Every run doubles as a **pooled-vs-naive A/B benchmark** (same seed,
  same packages) — memory, aggregate tps, deliveries/minute.

## Non-goals (v1)

- No compaction exercise (chains are short; CompactionNode has its own
  gate) — candidate for v2 with long chains.
- No multi-model routing; one engine per run.