docs: surface PolyKV by name (umbrella def, §3.4 retitle, per-pool tiering semantics) — sync from git fbc32c1f3a
Browse files
USAGE.md
CHANGED
|
@@ -152,7 +152,7 @@ to `.exe` (Windows needs the extension) can.
|
|
| 152 |
| KV quantization (scalar) | f16 | `-ctk` / `-ctv` | capacity |
|
| 153 |
| TurboQuant / TCQ KV tiers | off | `-ctk`/`-ctv turbo*` | capacity |
|
| 154 |
| Auto KV-tier policy | **off** | `OPENCOTI_KV_AUTO_TIER=1` | capacity (policy) |
|
| 155 |
-
| SharedKVPool | off (per request) | `shared_pool_slot` JSON field | multi-agent capacity |
|
| 156 |
| Rolling-KV window / spill | **auto** (engages only under pressure) | `--vram-target`, `--kv-residency-mode` | capacity |
|
| 157 |
| Mixed-KV spilled tail | off | `-ctkt` / `-ctvt` | capacity |
|
| 158 |
| DCA long-context | **off** | `--dca on` | context extension |
|
|
@@ -170,7 +170,14 @@ Every boot flag also has an env twin
|
|
| 170 |
|
| 171 |
---
|
| 172 |
|
| 173 |
-
## 3. KV capacity stack
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 174 |
|
| 175 |
These four features share one goal — **fit more context / more
|
| 176 |
sessions in fixed VRAM** — and are designed to stack. Recommended
|
|
@@ -253,7 +260,7 @@ prompt-cache-reuse are guarded off (requests bounded at `n_ctx`);
|
|
| 253 |
the compute-buffer reserve for long contexts is measured
|
| 254 |
automatically at boot (two-pass reserve — no knob).
|
| 255 |
|
| 256 |
-
### 3.4 SharedKVPool (multi-agent shared prefix)
|
| 257 |
|
| 258 |
N agents attending **one physical copy** of a common prefix (system
|
| 259 |
prompt + tool defs). Per-request JSON, no CLI flag:
|
|
@@ -271,6 +278,23 @@ Composes with KV quantization (the pool holds quantized cells) and
|
|
| 271 |
with session KV-reuse. The pool is read-only for consumers; each
|
| 272 |
agent's divergent suffix is private.
|
| 273 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 274 |
---
|
| 275 |
|
| 276 |
## 4. Long context
|
|
@@ -505,12 +529,12 @@ currently appear only in the server log.
|
|
| 505 |
|
| 506 |
## 8. Composition matrix
|
| 507 |
|
| 508 |
-
| | quant-KV | auto-tier | rolling-KV |
|
| 509 |
|---|---|---|---|---|---|---|---|---|
|
| 510 |
| **quant-KV** | — | K-only honors | ✅ (tiles dequant-on-lift) | ✅ | ✅ | ✅ (the win case) | ✅ (turbo+MTP is the top decode combo) | ✅ |
|
| 511 |
| **auto-tier** | | — | ✅ (it *manages* spill) | ✅ | ✅ (probes in DCA state) | ✅ | ✅ | ✅ (sizing is eff-plan-aware) |
|
| 512 |
| **rolling-KV** | | | — | ✅ | ✅ | ✅ | ✅ | ✅ (validated: window spill × RYS on hybrid) |
|
| 513 |
-
| **SharedKVPool** | | | | — | ✅ | ✅ | ✅ | untested |
|
| 514 |
| **DCA** | | | | | — | ✅ | ✅ (dual-ctx) | ✅ (eff→src mapped) |
|
| 515 |
| **sparse-attn** | | | | | | — | ✅ | ✅ |
|
| 516 |
| **MTP** | | | | | | | — | ✅ (draft runs base stack; target keeps RYS) |
|
|
|
|
| 152 |
| KV quantization (scalar) | f16 | `-ctk` / `-ctv` | capacity |
|
| 153 |
| TurboQuant / TCQ KV tiers | off | `-ctk`/`-ctv turbo*` | capacity |
|
| 154 |
| Auto KV-tier policy | **off** | `OPENCOTI_KV_AUTO_TIER=1` | capacity (policy) |
|
| 155 |
+
| PolyKV pool (SharedKVPool) | off (per request) | `shared_pool_slot` JSON field | multi-agent capacity |
|
| 156 |
| Rolling-KV window / spill | **auto** (engages only under pressure) | `--vram-target`, `--kv-residency-mode` | capacity |
|
| 157 |
| Mixed-KV spilled tail | off | `-ctkt` / `-ctvt` | capacity |
|
| 158 |
| DCA long-context | **off** | `--dca on` | context extension |
|
|
|
|
| 170 |
|
| 171 |
---
|
| 172 |
|
| 173 |
+
## 3. KV capacity stack — PolyKV
|
| 174 |
+
|
| 175 |
+
**PolyKV** is the umbrella name for this whole stack: the compressed
|
| 176 |
+
shared KV pool. Concretely it is the KV quantization tiers of §3.1
|
| 177 |
+
plus the multi-agent SharedKVPool of §3.4, stacking with the
|
| 178 |
+
auto-tier policy (§3.2) and the rolling-KV window (§3.3). If you
|
| 179 |
+
arrived here looking for "PolyKV" from an announcement: §3.4 is the
|
| 180 |
+
shared-prefix pool itself; §3.1 is what the pooled cells are made of.
|
| 181 |
|
| 182 |
These four features share one goal — **fit more context / more
|
| 183 |
sessions in fixed VRAM** — and are designed to stack. Recommended
|
|
|
|
| 260 |
the compute-buffer reserve for long contexts is measured
|
| 261 |
automatically at boot (two-pass reserve — no knob).
|
| 262 |
|
| 263 |
+
### 3.4 PolyKV SharedKVPool (multi-agent shared prefix)
|
| 264 |
|
| 265 |
N agents attending **one physical copy** of a common prefix (system
|
| 266 |
prompt + tool defs). Per-request JSON, no CLI flag:
|
|
|
|
| 278 |
with session KV-reuse. The pool is read-only for consumers; each
|
| 279 |
agent's divergent suffix is private.
|
| 280 |
|
| 281 |
+
**Tiering is pinned per-pool, never per-session.** The K/V tiers —
|
| 282 |
+
including the mixed-KV recent-window ⊕ compressed-tail pair — are
|
| 283 |
+
properties of the boot-allocated cache tensors, chosen once at boot
|
| 284 |
+
(by you or by auto-tier) before any session exists. The window/tail
|
| 285 |
+
boundary is a per-layer residency budget over the *physical cell
|
| 286 |
+
axis*, so a prefix cell is in the resident window or evicted (and
|
| 287 |
+
quantized exactly once, on eviction) for **all** sequences
|
| 288 |
+
simultaneously. Sharing itself is not copy-on-write: a sharer joins
|
| 289 |
+
the prefix by adding its sequence bit to the existing cells, and a
|
| 290 |
+
diverging session just appends private suffix cells — there is no
|
| 291 |
+
per-session copy that could be re-quantized, and no way for two
|
| 292 |
+
sessions to see the same prefix at different tiers. The flip side:
|
| 293 |
+
you cannot give one session a higher-precision read of a shared
|
| 294 |
+
prefix than another; that would require forking the prefix into a
|
| 295 |
+
private copy, which is exactly the O(N) memory cost the pool exists
|
| 296 |
+
to avoid.
|
| 297 |
+
|
| 298 |
---
|
| 299 |
|
| 300 |
## 4. Long context
|
|
|
|
| 529 |
|
| 530 |
## 8. Composition matrix
|
| 531 |
|
| 532 |
+
| | quant-KV | auto-tier | rolling-KV | PolyKV pool | DCA | sparse-attn | MTP | RYS |
|
| 533 |
|---|---|---|---|---|---|---|---|---|
|
| 534 |
| **quant-KV** | — | K-only honors | ✅ (tiles dequant-on-lift) | ✅ | ✅ | ✅ (the win case) | ✅ (turbo+MTP is the top decode combo) | ✅ |
|
| 535 |
| **auto-tier** | | — | ✅ (it *manages* spill) | ✅ | ✅ (probes in DCA state) | ✅ | ✅ | ✅ (sizing is eff-plan-aware) |
|
| 536 |
| **rolling-KV** | | | — | ✅ | ✅ | ✅ | ✅ | ✅ (validated: window spill × RYS on hybrid) |
|
| 537 |
+
| **PolyKV pool (SharedKVPool)** | | | | — | ✅ | ✅ | ✅ | untested |
|
| 538 |
| **DCA** | | | | | — | ✅ | ✅ (dual-ctx) | ✅ (eff→src mapped) |
|
| 539 |
| **sparse-attn** | | | | | | — | ✅ | ✅ |
|
| 540 |
| **MTP** | | | | | | | — | ✅ (draft runs base stack; target keeps RYS) |
|