docs/features/package_courier_harness.md

Package-courier harness — concurrency test + agentic performance benchmark

Spec agreed 2026-07-16 (user design + AskUserQuestion decisions). Tasks #739 (work server) / #740 (LangGraph app) / #741 (launcher + reporting). Sibling of the #712 PolyKV control-plane API; consumes opencoti-langgraph.

Purpose

One harness that is simultaneously:

1. a concurrency test for the PolyKV control-plane (pools, fork, admission, capacity, tps) under real multi-session load;
2. a live tool-calling exercise of the engine's native /v1/chat/completions tool-call path (the coverage gap from the C4 audit), multi-turn;
3. a repeatable agentic performance benchmark with a single scalar score (dollars) plus timing/throughput stats for graphs;
4. the reference LangGraph implementation for the opencoti_langgraph integration package (updated 2026-07-16, bug-2220): each worker episode is a compiled langgraph.StateGraph — START → capacity_gate → agent ⇄ tools → END with a bounded hold backoff loop — where capacity_gate is the packaged make_capacity_gate node, agent is one OpencotiChatModel call, and routing is Command-based over add_messages state. Offline routing tests: bench/package_courier/test_agent_graph.py. Live gates 2026-07-16 (Qwen3-4B, 3090): pooled 8/8 delivered 100% (cache_n>0 on 64/64 turns, scheduler 2→5 workers), naive 6/6 100%.

The workload

A Python work server (runs in the opencoti conda env, part of the harness) owns the ground truth:

• At start it procedurally generates P packages (default 100), each with a chain of n dependent steps (default 10). Every chain is unique in tool sequence and content mix. Steps are built from a vocabulary of fake attribute tools (haircolor, cartype, petname, …): step k's instruction names the tool to call and embeds the answer of step k−1 as input. Content is credible-but-nonsense — plausible enough that a model executes it without wondering why.
• Tools exposed to the agents (as LangChain tools hitting the work server's REST API; the model emits native OpenAI tool_calls via bind_tools/ToolNode):
  • get_package() → package_id + step-1 instructions (acquisition is the queue pull);
  • <attribute>(package_id, n_step) → the step's answer; the server records every call and verifies it is the right tool with the right args for that package/step;
  • next_step(package_id, n_step, result) → verifies the result (it is the input of the next step) and returns step k+1;
  • deliver_package(package_id, final_answer) → verified against the chain's final answer; correct delivery pays $1;
  • a completion signal endpoint the framework calls when all agents are done.
• Timeout: server ends the run if no package is delivered for 15 min (default) and no completion signal arrived.

The agent side (opencoti-langgraph app)

• Root pool = system prompt + tool instructions (chat-templated block). Workers attach with pool_id + per-worker session_id.
• Naive mode (--no-polykv): identical workload on plain sessions; worker count fixed at K == --parallel, FIFO package pull. This is the only shape naive supports — it is the A/B baseline.
• PolyKV mode (default): additionally supports a dynamic worker target driven by --tps-floor — the framework must oblige the floor: spawn additional workers only while GET /capacity admits (can_admit, expected_tokens arm) and mean_active_tps stays ≥ floor; back off when the floor is threatened. Enforced admission (429 + Retry-After → PoolSaturatedError) is honored as the hard backstop.
• The framework handles package acquisition, chain execution (each step = a new turn with a tool call), retries/prioritization (oldest-acquired first), monitors deliveries, and signals completion.

Launcher & flags

The launcher boots all three processes: the opencoti-llamafile server (model, -c, --parallel, --polykv-max-pools, KV types, MTP flags — all pass-through), the work server, and the agent app.

Defaults (updated 2026-07-16, user decision + bug-2222): 100 packages × 10 steps; engine --parallel 8; pooled worker fleet driven by --tps-floor 15 (fixed worker counts are only for the naive A/B leg, where the floor defaults to 0); work-server stall timeout auto-scales max(900 s, steps × 60 s) — the stall timer resets only on delivery, so long chains (e.g. 55 steps) need the scaled window; model = Qwen3.6-27B-Omnimerge-v4 (bs2) — hybrid GDN + NextN MTP, i.e. the serving target and the hardest pool path; everything overridable.

Scoring & report

• Score: delivered-correct × $1 (100 packages → $100 = 100%).
• Report: packages delivered / not delivered / lost in transit / failed-verification (and at which step), total wall time, delivery time mean/percentiles, per-step verification log.
• Engine stats: the harness polls /polykv/tps (per-session tps + ctx budget), /polykv/pools/*/capacity (headroom, compaction_pressure), /slots, /metrics on an interval → timeseries for graphs; the framework also reports its own API stats at the end.
• Speed curves (the HF question — how does agent speed behave as turns and context grow): every model turn is sampled with its chain step n, prompt size, cache_n, generation tps and prompt tps; the report aggregates tps-vs-step (mean tps at chain step 1..n across all packages) and tps-vs-context (512-token buckets). Pooled vs naive runs give the same curves side by side — i.e. whether/where per-turn speed decays with accumulated context, and what the pool buys back.
• Every run doubles as a pooled-vs-naive A/B benchmark (same seed, same packages) — memory, aggregate tps, deliveries/minute.

Non-goals (v1)

• No compaction exercise (chains are short; CompactionNode has its own gate) — candidate for v2 with long chains.
• No multi-model routing; one engine per run.