ManniX-ITA commited on
Commit
3b7efe6
·
verified ·
1 Parent(s): 5e37ae2

docs: surface PolyKV by name (umbrella def, §3.4 retitle, per-pool tiering semantics) — sync from git fbc32c1f3a

Browse files
Files changed (1) hide show
  1. USAGE.md +29 -5
USAGE.md CHANGED
@@ -152,7 +152,7 @@ to `.exe` (Windows needs the extension) can.
152
  | KV quantization (scalar) | f16 | `-ctk` / `-ctv` | capacity |
153
  | TurboQuant / TCQ KV tiers | off | `-ctk`/`-ctv turbo*` | capacity |
154
  | Auto KV-tier policy | **off** | `OPENCOTI_KV_AUTO_TIER=1` | capacity (policy) |
155
- | SharedKVPool | off (per request) | `shared_pool_slot` JSON field | multi-agent capacity |
156
  | Rolling-KV window / spill | **auto** (engages only under pressure) | `--vram-target`, `--kv-residency-mode` | capacity |
157
  | Mixed-KV spilled tail | off | `-ctkt` / `-ctvt` | capacity |
158
  | DCA long-context | **off** | `--dca on` | context extension |
@@ -170,7 +170,14 @@ Every boot flag also has an env twin
170
 
171
  ---
172
 
173
- ## 3. KV capacity stack
 
 
 
 
 
 
 
174
 
175
  These four features share one goal — **fit more context / more
176
  sessions in fixed VRAM** — and are designed to stack. Recommended
@@ -253,7 +260,7 @@ prompt-cache-reuse are guarded off (requests bounded at `n_ctx`);
253
  the compute-buffer reserve for long contexts is measured
254
  automatically at boot (two-pass reserve — no knob).
255
 
256
- ### 3.4 SharedKVPool (multi-agent shared prefix)
257
 
258
  N agents attending **one physical copy** of a common prefix (system
259
  prompt + tool defs). Per-request JSON, no CLI flag:
@@ -271,6 +278,23 @@ Composes with KV quantization (the pool holds quantized cells) and
271
  with session KV-reuse. The pool is read-only for consumers; each
272
  agent's divergent suffix is private.
273
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
274
  ---
275
 
276
  ## 4. Long context
@@ -505,12 +529,12 @@ currently appear only in the server log.
505
 
506
  ## 8. Composition matrix
507
 
508
- | | quant-KV | auto-tier | rolling-KV | SharedKVPool | DCA | sparse-attn | MTP | RYS |
509
  |---|---|---|---|---|---|---|---|---|
510
  | **quant-KV** | — | K-only honors | ✅ (tiles dequant-on-lift) | ✅ | ✅ | ✅ (the win case) | ✅ (turbo+MTP is the top decode combo) | ✅ |
511
  | **auto-tier** | | — | ✅ (it *manages* spill) | ✅ | ✅ (probes in DCA state) | ✅ | ✅ | ✅ (sizing is eff-plan-aware) |
512
  | **rolling-KV** | | | — | ✅ | ✅ | ✅ | ✅ | ✅ (validated: window spill × RYS on hybrid) |
513
- | **SharedKVPool** | | | | — | ✅ | ✅ | ✅ | untested |
514
  | **DCA** | | | | | — | ✅ | ✅ (dual-ctx) | ✅ (eff→src mapped) |
515
  | **sparse-attn** | | | | | | — | ✅ | ✅ |
516
  | **MTP** | | | | | | | — | ✅ (draft runs base stack; target keeps RYS) |
 
152
  | KV quantization (scalar) | f16 | `-ctk` / `-ctv` | capacity |
153
  | TurboQuant / TCQ KV tiers | off | `-ctk`/`-ctv turbo*` | capacity |
154
  | Auto KV-tier policy | **off** | `OPENCOTI_KV_AUTO_TIER=1` | capacity (policy) |
155
+ | PolyKV pool (SharedKVPool) | off (per request) | `shared_pool_slot` JSON field | multi-agent capacity |
156
  | Rolling-KV window / spill | **auto** (engages only under pressure) | `--vram-target`, `--kv-residency-mode` | capacity |
157
  | Mixed-KV spilled tail | off | `-ctkt` / `-ctvt` | capacity |
158
  | DCA long-context | **off** | `--dca on` | context extension |
 
170
 
171
  ---
172
 
173
+ ## 3. KV capacity stack — PolyKV
174
+
175
+ **PolyKV** is the umbrella name for this whole stack: the compressed
176
+ shared KV pool. Concretely it is the KV quantization tiers of §3.1
177
+ plus the multi-agent SharedKVPool of §3.4, stacking with the
178
+ auto-tier policy (§3.2) and the rolling-KV window (§3.3). If you
179
+ arrived here looking for "PolyKV" from an announcement: §3.4 is the
180
+ shared-prefix pool itself; §3.1 is what the pooled cells are made of.
181
 
182
  These four features share one goal — **fit more context / more
183
  sessions in fixed VRAM** — and are designed to stack. Recommended
 
260
  the compute-buffer reserve for long contexts is measured
261
  automatically at boot (two-pass reserve — no knob).
262
 
263
+ ### 3.4 PolyKV SharedKVPool (multi-agent shared prefix)
264
 
265
  N agents attending **one physical copy** of a common prefix (system
266
  prompt + tool defs). Per-request JSON, no CLI flag:
 
278
  with session KV-reuse. The pool is read-only for consumers; each
279
  agent's divergent suffix is private.
280
 
281
+ **Tiering is pinned per-pool, never per-session.** The K/V tiers —
282
+ including the mixed-KV recent-window ⊕ compressed-tail pair — are
283
+ properties of the boot-allocated cache tensors, chosen once at boot
284
+ (by you or by auto-tier) before any session exists. The window/tail
285
+ boundary is a per-layer residency budget over the *physical cell
286
+ axis*, so a prefix cell is in the resident window or evicted (and
287
+ quantized exactly once, on eviction) for **all** sequences
288
+ simultaneously. Sharing itself is not copy-on-write: a sharer joins
289
+ the prefix by adding its sequence bit to the existing cells, and a
290
+ diverging session just appends private suffix cells — there is no
291
+ per-session copy that could be re-quantized, and no way for two
292
+ sessions to see the same prefix at different tiers. The flip side:
293
+ you cannot give one session a higher-precision read of a shared
294
+ prefix than another; that would require forking the prefix into a
295
+ private copy, which is exactly the O(N) memory cost the pool exists
296
+ to avoid.
297
+
298
  ---
299
 
300
  ## 4. Long context
 
529
 
530
  ## 8. Composition matrix
531
 
532
+ | | quant-KV | auto-tier | rolling-KV | PolyKV pool | DCA | sparse-attn | MTP | RYS |
533
  |---|---|---|---|---|---|---|---|---|
534
  | **quant-KV** | — | K-only honors | ✅ (tiles dequant-on-lift) | ✅ | ✅ | ✅ (the win case) | ✅ (turbo+MTP is the top decode combo) | ✅ |
535
  | **auto-tier** | | — | ✅ (it *manages* spill) | ✅ | ✅ (probes in DCA state) | ✅ | ✅ | ✅ (sizing is eff-plan-aware) |
536
  | **rolling-KV** | | | — | ✅ | ✅ | ✅ | ✅ | ✅ (validated: window spill × RYS on hybrid) |
537
+ | **PolyKV pool (SharedKVPool)** | | | | — | ✅ | ✅ | ✅ | untested |
538
  | **DCA** | | | | | — | ✅ | ✅ (dual-ctx) | ✅ (eff→src mapped) |
539
  | **sparse-attn** | | | | | | — | ✅ | ✅ |
540
  | **MTP** | | | | | | | — | ✅ (draft runs base stack; target keeps RYS) |