# PolyKV Control-Plane API — Design (F5 M6-S5 / #712) **Status:** design (2026-07-16). Implementation is phased (§10); nothing here is shipped yet. **Inputs:** [polykv_api_research.md](polykv_api_research.md) (design-space survey), [poly_kv.md](poly_kv.md) (the KV mechanism this sits on top of), [llamafile-usage.md](../llamafile-usage.md) §3.4 (SharedKVPool user surface). **Single rule:** additive / opt-in. Every endpoint and field lives under the existing `opencoti` JSON namespace or new `/polykv/*` routes; no upstream behavior changes when PolyKV control is unused. --- ## 1. Goals A rich, opt-in control plane that satisfies every multi-agent serving journey on top of the existing SharedKVPool (`shared_pool_slot` + `shared_prefix_n_tokens`, server-context.cpp:3085). Four requirements, plus two the requirements imply: 1. **Full status** of pools + the sessions attached to each. 2. **tps reporting** — aggregate system throughput *and* per-session generation tps. (All-new: upstream llama.cpp reports per-*response* `timings` but has no per-session/slot running instrumentation.) 3. **Autonomous capacity/admission loop** — a main session sets a per-session gen-tps target (e.g. 5 tps); when the average across active sessions approaches the floor the orchestrator holds off spawning sub-agents. Behavioral knobs: reject a new session vs. warn. 4. **LangGraph extensions** — a custom `BaseChatModel`, the pool-control API exposed as LangGraph tools, and PolyKV-aware agent-state routing. 5. **Nesting** (implied by 3): a sub-agent that discovers fan-out work can become a pool parent for its own sub-agents, so parallelizable tasks run in parallel instead of being forced sequential. 6. **Fork / branch** (COW): a sub-agent group that shares an *ancestor* prefix but then diverges (different toolset/context) gets its own shared pool branched at the divergence point. ## 2. Decisions locked (2026-07-16) | # | Decision | Choice | |---|----------|--------| | API home | where the control plane lives | **Self-contained in the C++ engine** — extends `/props`, `/slots`, adds `/polykv/*`. Works standalone; no `opencoti-server` in the path. | | Admission | enforce the tps floor or advise | **Both** — advisory (report) by default; optional per-pool enforced mode with `reject`\|`warn`\|`queue`. | | Pool model | how pools are created | **Explicit first-class objects** — `create`/`fork`/`attach`/`release`, immutable from birth (§3.3). Contiguous-prefix contract checked at fork. Boilerplate hidden by the opencoti plugin. | | Scope | deliverable shape | **Design the full surface now; implement in gated phases** (§10). | | Fork | divergent child prefix | **First-class in v1** (copy-free COW, §3.2 / §7) — *not* deferred. | **Defaulted (override any):** transport = HTTP on the engine + SSE for the live tps stream (poll fallback); tenancy = single trust domain (no per-pool ACL); tps = per-session *experienced* decode tps (EWMA) + aggregate decode throughput, floor = mean of active sessions' experienced tps (prefill excluded); nesting depth uncapped; interior pools pinned while any descendant is attached + grace TTL; seq-ids for pools+leaves sized at boot. --- ## 3. Data model ### 3.1 A pool is a tree node ``` Pool { id : int32 // seq_id in the unified cache parent : int32 | ROOT // -1 for a root pool branch_pos D : int32 // shared-ancestor length inherited from parent own_range : [D, L) // tokens this pool materialized itself prefix_len L : int32 // total shared prefix = D + |own_range| tokens : [llama_token] // full prefix token array [0, L), host-side (auto-P + fork validation) prefix_hash : [u64] // cumulative rolling hash per position (validates any D; ~8 B/token host RAM) pin : bool // resident-pinned (auto while descendants attached) children : [Pool] // interior pools branched/extended from this one sessions : [Session] // leaf sharers attached at this pool's prefix last_access : ts } Session { // a leaf = an ordinary decoding slot id : int32 // seq_id pool : int32 // the pool it attached to suffix_range : [L, L+S) // its own private, mutable continuation tps_ewma : f32 // experienced decode tps ttft_ms, state, ... } ``` The **root** pool (`parent = ROOT`, `D = 0`) is what today's `shared_pool_slot` request already produces. Everything else is new. ### 3.2 Three edge types, one mechanism A child pool inherits `[0, D)` from its parent and owns `[D, L)`. `D` selects the edge type: - **extend** — `D = parent.L`. Child = parent prefix **+** appended shared tokens. This is *nesting*: a sub-agent adds its own shared context and hands it to its sub-agents. - **branch (COW fork)** — `D < parent.L`. Child shares the common ancestor `[0, D)` and diverges with its own `[D, L)`. This is the fork: same system prefix, different toolset after `D`. - **root** — `parent = ROOT`, `D = 0`. **Why the fork is copy-free.** `seq_cp` in the unified cache is additive membership: for each cell where `seq_has(cell, parent)` in `[0, D)` it calls `seq_add(cell, child)` (llama-kv-cache.cpp:2007-2044). No cell is duplicated — the ancestor cells simply gain the child in their sequence set. Because prefixes are **immutable** (FROZEN, §3.3), the shared range is never mutated, so the "copy" a classic COW would eventually pay never happens. The child only computes new KV for `[D, L)`, prefilled at RoPE position `D` attending to the shared `[0, D)` already in cache — exactly how attaching with a shorter `shared_prefix_n_tokens` works today. **Extend and branch are the same code path parameterized by `D`.** ### 3.3 Immutability & versioning **Pools are immutable from birth.** A pool's prefix is fully materialized at `create`/`fork` and never changes afterwards — there is no OPEN/extend-in-place state. "Extending" a pool *always* means creating a child (`fork` with `D = parent.L`). This is simpler than a freeze-on-first-attach state machine and loses nothing: the "build context incrementally" journey is covered by the main agent decoding in its **own session** and then forking `from_session` (§4.1), which snapshots its current context as a new pool in one call — no in-place mutation ever needed. This makes "add a tool to the shared prefix after agents attached" well-defined — it is a *new versioned pool* (a fork), and the old one keeps serving its sharers unchanged. (This is the precise answer to the HuggingFace question: the prefix is static per pool; a tool/MCP added later is a new pool version, valid only for sessions that attach to it — never mutated under sessions already on the old one.) ### 3.4 Lifetime Cell freeing is implicit-refcounted: a cell is reclaimed only when its sequence set empties, so an ancestor's cells physically survive as long as *any* descendant references them — even if the ancestor's own slot is torn down. On top of that: - **Pin policy.** Interior pools are auto-pinned (`--no-clear-idle` semantics, per-node) while any descendant is attached. Without this a new leaf attaching to an idle-cleared interior pool hits the residency guard (server-context.cpp:3116) and silently falls back to full reprocess — correct but slow. A **grace TTL** delays reclaim after the last descendant detaches (cheap re-attach for bursty fan-out). - **seq-id budget.** Every pool *and* every leaf consumes one `n_seq_max` slot. A deep/wide tree spends seq-ids on interior nodes too; the boot-time reserve accounts for both. Exposed in `/capacity`. ### 3.5 Materialization — the slot/seq decoupling (P2's real work) Today a "pool" is just an ordinary server **slot** that happened to serve a request: `shared_pool_slot` names a slot index, the pool's tokens live in `slots[pool].prompt.tokens`, and the pool seq **is** the slot's seq. First-class pools break that coupling, and this is the largest single implementation lift in the milestone: - **seq-ids beyond slots.** Pool seq-ids come from a boot-time reserve above the slot range: `n_seq_max = n_parallel + --polykv-max-pools` (default proposed: 16). A pool occupies a seq-id but **no slot** once materialized. - **Prefill-only internal task.** `create`/`fork` materializes `[D, L)` via an internal task that runs through the normal batching path (so it composes with every KV feature) but emits **no logits and no generation**; it transiently borrows a slot for the prefill, then the KV stays on the pool's seq and the slot is returned. `seq_cp(parent → pool, 0, D)` runs first for fork/extend. - **Host-side pool state.** The Pool object (tokens, cumulative hash, tree links, pin/TTL) lives in server memory beside the slot table — *not* in a slot — and is what `/pools` reports and attach validates against. - **Unified KV required.** The split-stream `seq_cp` overload hard-asserts `is_full` (llama-kv-cache.cpp:2063); pool sharing is only defined under `--kv-unified` — already the SharedKVPool prerequisite, now stated as a boot-time check when `--polykv-max-pools > 0`. --- ## 4. API surface All additive. Reuses the `opencoti` JSON namespace (`get_opencoti_props()` :3787) and the `/slots` accumulator precedent (:183). Endpoint shapes follow research §6. ### 4.1 Pool lifecycle (imperative, explicit) | Verb | Endpoint | Body → result | |------|----------|---------------| | create root | `POST /polykv/pools` | `{tokens? \| from_session?}` → `{pool_id, prefix_len}`. `from_session` snapshots a live session's current context as the pool prefix — the **primary** creation path for agents (no token round-trip). | | fork (extend or COW branch) | `POST /polykv/pools/{id}/fork` | `{branch_pos D, tokens[D:L] \| from_session?}` → new child `{pool_id, parent:id, branch_pos, prefix_len}`. `D == parent.L` ⇒ extend (nesting); `D < parent.L` ⇒ COW branch. | | attach | *(request path, unchanged mechanism)* | per-request `pool_id` (legacy `shared_pool_slot` kept). **`shared_prefix_n_tokens` becomes optional**: the server auto-computes `P` as the longest hash-match between the request prompt and the pool's stored tokens (**auto-P**) — killing the mis-set-P footgun and the HF-question confusion. An explicit `P` is still accepted and validated. | | pin/evict/flush | `POST /polykv/pools/{id}/{pin\|evict\|flush}` | retention control (research §6 Req 1) | | release | `DELETE /polykv/pools/{id}` | detach + reclaim when refcount hits 0 (respects grace TTL) | Contiguous-prefix contract is checked at `fork`: the child's declared `tokens[0:D]` must hash-match the parent's `[0:D)` (cumulative `prefix_hash` at `D`), else `409 Conflict` — the caller then either re-roots or full-prefills. All routes live under a `/polykv/` prefix so a future upstream llamafile/llama.cpp bump can never collide with an unprefixed `/pools`. ### 4.2 Status (research §6 Req 1) - `GET /polykv/pools` → `{pools:[{pool_id, parent, branch_pos, prefix_len, own_kv_bytes, subtree_kv_bytes, residency, pinned, last_access_ts, children:[pool_id], sessions:[{session_id, suffix_n_tokens, marginal_kv_bytes, tps_ewma, ttft_ms, state}]}], tree_depth, seq_ids_used, seq_ids_max}`. The `parent`/`children` fields make the **tree** explicit. **KV attribution is by ownership, never by membership**: a shared cell belongs to many seqs, so summing per-seq bytes would double-count — a pool reports `own_kv_bytes` = its `[D, L)` range only, plus a `subtree_kv_bytes` rollup (own + all descendants + attached sessions' suffixes). Σ own over the tree == physical bytes, by construction. - `GET /polykv/pools/{id}` → one subtree. - `/slots` gains `pool_id` + `tps_ewma` per slot (extends #677). - `/props` `opencoti` block gains a `polykv` sub-object (config: floor, admission mode, pin/TTL policy). ### 4.3 tps (research §6 Req 2) - **Per-response** (already partly present): `timings{predicted_per_second, prompt_per_second, cache_n}`. - **Headers** (TGI model): `x-session-tps`, `x-cached-prefix-tokens`, `x-pool-id`. **Deferred to P2+**: HTTP headers must be emitted before the body, but the slot (and thus the live tps value) is assigned only after the task is queued — a header set at request time would always be stale, and trailers aren't portable. The per-response `timings` JSON already carries the authoritative per-response rate; headers add value only once pool attach happens at request routing (P2), where `x-pool-id`/`x-cached-prefix-tokens` are known up front. - **Live stream:** `GET /polykv/tps` (SSE) — periodic `{ts, aggregate_tps, sessions:[{session_id, pool_id, tps_ewma}], mean_active_tps}`. This is the signal the admission loop subscribes to. - **`/metrics`** (Prometheus, low-cardinality per research §4): gauges `opencoti_polykv:gen_throughput_tps{pool}`, `pool_kv_bytes{pool}`, `pool_sessions{pool}`; counters `prefix_cache_hits_total{pool}`, `queries_total{pool}`. Label = pool only (never session — cardinality). ### 4.4 Admission (research §6 Req 3; decision: both modes) - `POST /polykv/pools/{id}/admission` `{target_tps_per_session, mode:"advisory"| "enforced", on_saturation:"reject"|"warn"|"queue"}` — per-pool policy. - `GET /polykv/pools/{id}/capacity` → `{can_admit, projected_mean_tps_if_admitted, headroom_sessions, kv_headroom_pct, reason}`. `headroom_sessions` is computed against **marginal** per-session KV (`S_i·bpt`), not full context — the basis of the ~7× reduction (research §5). - **Advisory (default):** endpoints report; the orchestrator decides. Nothing is rejected server-side. - **Enforced (opt-in):** on a new attach with projected `mean_active_tps < floor` OR `kv_headroom` exhausted → `reject` = `429`/`503` + `Retry-After` (RFC 6585/9110); `warn` = admit + `X-Sessions-Remaining` header; `queue` = hold until headroom returns. - **Throttle-toward-floor is explicitly NOT v1.** Andes' insight (over-floor generation is wasted GPU → throttle fast sessions back toward the floor to reclaim capacity) requires per-slot rate control inside the continuous- batching decode loop — i.e. skipping a slot in some decode batches. That is scheduler surgery with its own correctness/perf gates, not an API feature; it is banked as a P4-follow candidate, and v1 `warn` is headers-only. --- ## 5. tps instrumentation (all-new, C++ server-side) Upstream has no per-session running tps. Design: - Extend the `/slots` lifetime accumulators (server-context.cpp:183) with a per-slot decode-token counter + wall-clock, updated at each decode-batch completion for the tokens attributable to that slot. - **Experienced tps** per session = EWMA over a rolling window (default: last `W` decoded tokens or `T` ms, whichever first) of that slot's decode throughput. This is the quantity that *falls as concurrency rises* even while aggregate climbs (attention doesn't batch-amortize — research §5), so it is the correct SLO signal. - **Aggregate** = system decode throughput (all slots' decode tokens / wall). - **`mean_active_tps`** = mean of experienced tps across currently-decoding sessions — the value the floor compares against (matches "average sessions gen tps"). - Prefill tokens are excluded from "gen tps" (counted separately as `prompt_per_second`). - Cost: a handful of integer counters + one EWMA update per slot per batch — negligible; gated behind the PolyKV block so off-path is byte-identical. --- ## 6. Admission control Two coupled signals (research §5): 1. **Soft / low-lag:** `mean_active_tps` vs the floor. Directly measures the SLO; leads `num_requests_waiting` (which lags). 2. **Hard wall:** pool KV-occupancy %. Reject *before* this forces preempt-oldest eviction. `projected_mean_tps_if_admitted` estimates the post-admit floor from the `B_sat` knee model (below the knee admitting is ~free; above it every admit costs everyone's TPOT ~linearly). On high-VRAM cards (RTX 6000 96 GB) the tps knee is hit **before** the KV wall, so signal (1) dominates on opencoti's hardware — which is exactly why the floor, not occupancy, is the headline. Framing: "5 tok/s/session" = **TPOT SLO 200 ms/token; maximize goodput s.t. TPOT ≥ floor** (a floor, not a maximize — Andes). The default is advisory: the LangGraph capacity-gate node (§8) reads `/capacity` and decides. Enforced mode is the same math with server-side `reject`/`warn`/`queue`. --- ## 7. Nesting & COW-fork — worked journeys **Journey A — sub-agent parallelization (nesting / extend).** Main agent M runs on root pool `P0 = [system+tools]` (len N0, FROZEN). M builds task context and `POST /pools/P0/fork {branch_pos:N0, tokens:[N0:N1]}` → interior pool `P1 = [system+tools+task]`. M spawns k sub-agents that each attach to `P1` and decode their own suffix in parallel. Physically, `P1`'s `[0:N0)` cells are P0's cells (now in seq-set {P0,P1,leaf…}); `[N0:N1)` is prefilled once. The k sub-agents cost only their marginal suffixes. The fan-out work that used to run sequentially now runs concurrently under one shared branch. **Journey B — divergent toolset (COW fork).** Two sub-agent groups share the system preamble `[0:D)` but need different toolsets. Group B does `POST /pools/P0/fork {branch_pos:D, tokens:[D:D']}` (D < N0) → `P1' = [system+toolsetB]`, sharing `[0:D)` zero-copy, prefilling `[D:D')` fresh. Its sub-agents attach to `P1'`. No physical copy of the shared preamble. **Invariants enforced.** (a) contiguous-prefix hash check at `fork`; (b) immutability from birth — a pool's prefix is fully materialized before any descendant can attach (§3.3), so no attach ever races a prefix change; (c) recursive pin — interior pools stay resident while descendants attached; (d) seq-id budget counts interior nodes. Divergence that is *not* a clean prefix of any existing pool → new root (or full prefill) — reported, never silently wrong. **Hybrid/recurrent guard (bug-2203) is recursive.** Every attach/fork on a recurrent or hybrid model still hits the exact-full-state guard (server-context.cpp:3116) at every level. --- ## 8. LangGraph integration (research §6 Req 4) Package home: a new **top-level `opencoti/` non-bun root** (per the repo's "new non-bun roots" rule) — Python package `opencoti-langgraph` (pip). Kept out of the bun workspace. Pin `BaseChatModel` signatures to a specific `langchain-core` version tag (the API is mid-migration — research caveat). - **`OpencotiChatModel(BaseChatModel)`** — implement `_generate` + `_llm_type` (required), `_stream`/`_agenerate`/`_astream`; override `bind_tools`. Populate `usage_metadata` and `response_metadata{pool_id, session_tps, backpressure}` from the headers/§4.3. - **Pool-control tools** — `create_pool`, `fork_pool`, `pool_status`, `session_tps`, `capacity` exposed as LangChain tools via `bind_tools` / `ToolNode`, so an agent can manage its own pool tree. - **Capacity-gate node** — reads `/capacity`; returns `Command(update={"capacity_ok":ok, "headroom_sessions":n}, goto=...)` to route the graph (spawn vs. hold). `interrupt()` for block-until-free. `InMemoryRateLimiter` is explicitly insufficient — it can't see per-session tps or KV headroom. - **Agent-state routing** — a reducer surfaces `headroom_sessions` / `mean_active_tps` into graph state so conditional edges branch on live PolyKV capacity. --- ## 9. Composition with the KV stack PolyKV control is orthogonal to the KV *mechanics* and must not perturb them: - **rolling-KV / DCA / turbo / TCQ / quant-KV** — pool sharing is `seq_cp` membership on whatever cell type is configured; the tps/admission plane is read-only over the existing decode loop. Off-path byte-identical (gate). - **MTP / speculative** — per-session tps counts *accepted* tokens (the user- visible generation rate), not draft steps. - **`--parallel ≥ 2` / `--kv-unified`** — the multi-slot requirement is already the SharedKVPool baseline; the tree just adds interior seq-ids. - **hybrid/recurrent** — bug-2203 guard, recursive (§7). ## 10. Phased implementation plan (gated) Full surface designed here; implementation lands in gated phases, each with a correctness + off-path-identity gate before the next: - **P1 — tps instrumentation** (§5). Per-slot counters + EWMA + `/slots`/`/props` exposure + `/polykv/tps` SSE + `/metrics`. Prereq for everything. **✅ IMPLEMENTED 2026-07-16** (live in vendored tree, patch capture pending): gates green — sustained EWMA −3.2% vs `timings` ground truth; 2-session concurrency shows correct `aggregate_tps` ≈ Σ sessions + `mean_active_tps`; `/metrics` 3 series render. bug-2205: aggregate MUST come from Σ `n_decoded_cum` (per-token), not `n_tokens_predicted_total` (completion-time only). - **P2 — pool objects + status API** (§3, §4.1-4.2). The slot/seq decoupling (§3.5): seq-id reserve, prefill-only materialization task, host-side Pool state, **auto-P attach**; `GET /polykv/pools`. Root pools only. **✅ IMPLEMENTED 2026-07-16** (live in vendored tree, patch capture pending): gate 23/23 on Qwen3-4B — boot check (`--polykv-max-pools` without `--kv-unified` fails clean; NOTE `--parallel` must be explicit, unset auto-sets `kv_unified=true`), create-from-prompt prefill-only P=95, auto-P exact attach `cache_n=95`, divergence attach `cache_n=42<95`, `from_session` zero-copy `seq_cp` snapshot (affinity is evict-on-reuse — only the most-recent session per slot resolves), pin/unpin/GET-one, release + released-id clean fallback, disabled path inert, pooled vs unpooled greedy text IDENTICAL (bit-shared cells). Mutations only on the server thread via `SERVER_TASK_TYPE_POLYKV`; recurrent guard per bug-2203. - **P3 — nesting & COW-fork** (§7). `fork` with arbitrary `D` (extend + COW branch), cumulative-hash contract check, recursive pin, seq-id accounting. **✅ IMPLEMENTED 2026-07-16** (live in vendored tree, patch capture pending): gate 18/18 substantive on Qwen3-4B — extend fork (D=parent.L, own=22), COW branch (D=47<95, zero-copy `seq_cp` of `[0,D)` + fresh prefill), contract violation → clean 4xx, auto-P attach to a child (cache_n=117), tree reporting (parent/children/branch_pos/own_len/tree_depth), release ordering enforced (parent-with-children refused, leaves-first drains), seq exhaustion at max_pools, `from_session` fork. The §4.1 prefix contract is enforced **token-exact** (`std::equal` against the parent's stored tokens — strictly subsumes the cumulative-hash check; `prefix_hash` kept as identity fingerprint). Pooled-vs-unpooled greedy divergence on the branch was classified benign by first-token probe (same top-5 ranking, max |Δlogprob| 0.112 — batch-shape numerics, #495 class). Recurrent bug-2203 guard is recursive: hybrid/recurrent forks must extend at D == parent.L. - **P4 — admission** (§6). `/capacity`, advisory metrics, then enforced mode (`reject`/`warn`/`queue`). **✅ IMPLEMENTED 2026-07-16** (live in vendored tree, patch capture pending; `queue` deferred per §11 default): gate 18/18 on Qwen3-4B — /capacity shape (idle: can_admit, projected null, kv cells by OWNERSHIP: pools' own ranges + slots' marginal suffixes via new `slot.n_pool_shared`), advisory floor never rejects, **enforced+reject → 429 + Retry-After** under an active decoding session (tps-floor arm needs n_active ≥ 1), **enforced+warn → 200 + X-Sessions-Remaining**, off-path (no pool_id / unknown pool_id) untouched. Projection = conservative fully-saturated knee: `mean_active · n/(n+1)` (never over-admits below the real B_sat knee). Fast path: an atomic enforced-pool count — ordinary traffic pays zero; the gate itself is a high-priority OP_CAPACITY control task from `handle_completions_impl` (same pattern as /slots' METRICS task). - **P5 — LangGraph package** (§8). `opencoti-langgraph` under top-level `opencoti/`. Runs in the first-class **`opencoti` conda env** (py3.11, langchain-core 1.4.x + langgraph 1.2.x pinned; the same env will later host the consultants council alongside opencoti-server). Includes the P6d `CompactionNode` (§13). **P5 ✅ IMPLEMENTED 2026-07-16**: offline 21/21 (pytest, httpx MockTransport FakeServer), live gate **19/19** on Qwen3-4B `:8231 --parallel 2 --kv-unified --polykv-max-pools 4` — pool tree (token-path fork D=65), `OpencotiChatModel` invoke + streaming with pool attach proven (`cache_n=68 ≥ root prefix 65`), `session_tps`/usage metadata, capacity context arm + gate-node routing, enforced 429 → `PoolSaturatedError(retry_after)` → advisory recovery, **P6c compact-by-refork e2e** (old pool released, migrated session coherent on the new pool, `cache_n=71`), tps `ctx_*` fields. Contract lessons (bug-2208): fork `prompt`/`tokens` = the child's **FULL prefix [0,L)** token-exact vs the parent (never suffix-only — build via `/tokenize`: parent tokens + suffix `add_special=false`); a pool prefix meant for `/v1/chat/completions` attach must be the **chat-templated** system block (raw text shares 0 tokens → `cache_n=0`); Qwen3 thinking models need `chat_template_kwargs: {enable_thinking: false}` for content-based gates. Client accordingly ships `tokenize()` + `compact_by_refork(compacted_tokens=…)`. - **P6 — context exhaustion & compaction** (§13). **P6a (ctx-shift pool guard) + P6b (context visibility + /capacity context arm) land BEFORE P5** per user decision 2026-07-16; P6c/P6d ship with P5. **P6a+P6b ✅ IMPLEMENTED 2026-07-16** (live in vendored tree, patch capture pending): gate 23/23 on Qwen3-4B at `-c 4096 --parallel 1 --context-shift` (the exact §13.1-regime-3 configuration) — boot WARN; pooled session forced past n_ctx = graceful `truncated=true` stop at 4021 tokens (position wall fires one token before unified-cell exhaustion when the prefix is pool-shared), NO shift, pool provably coherent afterwards (fresh attach cache_n=60, needle answered); non-pooled ctx-shift still engages and completes after pools released (off-path intact); `?expected_tokens` context arm rejects with `"context headroom exhausted"`; `compaction_pressure` 0 at idle, ∈[0,1]; `orphaned_pin` true for a pinned never-attached leaf; `/polykv/tps` sessions carry `ctx_used/ctx_total/ctx_headroom_tokens`. Guard bookkeeping: `slot.n_pool_shared` is clamped to the retained prefix at prompt-cache decision time (`min(n_pool_shared, n_past)`) — NOT blind-reset, because cache reuse keeps pool-shared cells across later non-pool requests on the same session. NOTE the geometry lesson: with `--kv-unified` the slot n_ctx is the FULL `-c` (not divided by `--parallel`), and multi-slot cell exhaustion precedes the positional wall — gates must use `--parallel 1`. ## 11. Open sub-decisions (defaulted; flag to change) - EWMA window `W`/`T` defaults (proposed: 128 tokens or 2 s). - Grace TTL for interior-pool reclaim (proposed: reuse `--slot-shrink-idle-ms`). - `/metrics` opt-in flag name + default-off. - Whether `queue` (admission) is v1 or P4-follow (proposed: `reject`+`warn` v1, `queue` follow). - `--polykv-max-pools` default (proposed: 16; each reserved seq-id costs KV bookkeeping but no cells until materialized). - Hash algorithm for the cumulative `prefix_hash` (proposed: xxhash64 rolling; 8 B/token host RAM — ~800 KB for a 100k-token prefix, negligible). - SSE tps-stream cadence (proposed: 500 ms, configurable; stream is opt-in so off-path cost is zero). ## 12. Verification - Off-path identity: PolyKV-control-unused boot is byte-identical (DSO + decode). - tps accuracy: instrumented per-session tps vs. an external wall-clock token-rate measurement, ±few %. - Fork correctness: a branched pool's decode is logit-equivalent to the same session full-prefilled (real_frac=0 / KLD), never greedy-needle. - Tree lifetime: ancestor cells survive descendant-only references; reclaim on refcount 0; recursive pin prevents the residency-miss fallback. - Admission: projected vs. realized `mean_active_tps` under a spawn ramp; `reject`/`warn` behavior + `Retry-After`/`X-Sessions-Remaining` headers. - Composition gates: rolling-KV / DCA / turbo / MTP / `--parallel 2` unchanged. ## 13. Context exhaustion & compaction orchestration (P6) *Added 2026-07-16 on user review: the design above admits sessions by tps floor and marginal KV, but had no story for what happens when the shared context actually fills — and continuous agentic frameworks WILL fill it. Decision: P6a+P6b land BEFORE P5 (the LangGraph capacity gate builds on the context arm); P6c/P6d ship with/after P5.* ### 13.1 What the engine does today (verified, three regimes) 1. **Prompt too long at admission** → clean `ERROR_TYPE_EXCEED_CONTEXT_SIZE`. 2. **Generation reaches `n_ctx`, `--ctx-shift` off (default)** → clean stop with `truncated=true` (`server-context.cpp:1898`). Not corrupting, but silent mid-thought truncation is exactly the "fails badly" mode for agents: no advance warning, no orchestration signal. 3. **`--ctx-shift` on + pool-attached session → CORRUPTION.** The shift does `seq_rm` + `seq_add(…, -n_discard)` (`server-context.cpp:3261-3262`); in the unified cache `seq_add` mutates the **per-cell position**, and pool-shared cells are members of many seqs — the shift re-RoPEs the pool's prefix for every other session and every descendant pool. Upstream guards this for parent/child shared prompts (`server-context.cpp:3192`) but a pool attach is neither, so the guard does not fire. The rest-kv-eviction variant uses the same primitives and has the same hazard. The "safe" naive alternative — COW-copy shared cells before shifting — allocates a duplicate of the prefix at the exact moment the cache is full. Both intuited failure modes (corruption / usage inflation) are real. ### 13.2 Design principle: compaction is a prompt REWRITE, never an in-place KV op LangGraph (and every agentic framework) compacts at the prompt-assembly level: `trim_messages`, `RemoveMessage` reducers, or a summarization `pre_model_hook` that replaces old messages with a running summary. The compacted context is **new text** — its tokens cannot match the old cells, so cell-level "compaction" is not even meaningful. Therefore the server must never mutate pooled cells; the correct primitive already exists in the tree: > **Compaction = re-root via fork.** The system prompt + tool definitions > survive compaction verbatim — that is the stable ancestor pool > `[0, D_sys)`. Compacted context = `fork(ancestor, D=D_sys, > tokens = summary + recent tail)` — a new sibling branch, prefilled once. > Sessions migrate to the new pool; the old working subtree is `release`d. > Pool immutability is preserved by construction, no cell position is ever > touched, and net cells go **down** (old subtree reclaimed) instead of up > (COW duplication). Pinned pools under this model are a **leak hazard, not a corruption hazard**: a pin left on the abandoned subtree blocks reclaim. Discipline is unpin-after-migrate; the server reports orphans (pinned + zero sessions + zero children) so the orchestrator can sweep. ### 13.3 Phases - **P6a — corruption guard (before P5).** Extend the ctx-shift refusal to pool-attached slots (`slot.n_pool_shared > 0` ⇒ clean error, same as the `:3192` shared-prompt refusal) + boot WARN when `--ctx-shift` is combined with `--polykv-max-pools`. Rest-kv-eviction path included. Off-path inert. - **P6b — visibility + spawn-gate parity (before P5).** Per-session `ctx_used`/`ctx_headroom_tokens` in `/polykv/tps` + `/slots`; `/capacity` gains `?expected_tokens=N` and a **context arm** to `can_admit` (reason `"context headroom exhausted"`) so sub-agent spawns are gated on context exactly like the tps floor; plus a `compaction_pressure` field (0..1, driven by free-cell fraction and largest-session share) so the orchestrator compacts BEFORE the wall. - **P6c — compact-by-refork protocol (with P5).** Documented two-step: summarize via normal completion → `fork` the stable ancestor at `D_sys` with the compacted suffix → migrate sessions → `release` old subtree. Orphaned-pin flag in `/polykv/pools`. Atomic convenience endpoint only if the two-step proves racy in practice. - **P6d — LangGraph side (inside opencoti-langgraph).** `CompactionNode` (pre_model_hook pattern: watch `response_metadata` headroom / `compaction_pressure` → summarize → re-root → swap `pool_id` in graph state); the capacity-gate node checks the context arm from day 1. ### 13.4 Non-goals - In-place KV compaction/merging of pooled cells (meaningless under rewrite semantics; corrupting under sharing). - Server-side self-summarization (the orchestrator owns the summary — it has the conversation semantics; the server only has tokens). - Growing `n_ctx` at runtime (allocation is boot-time; capacity planning is the admission plane's job).