# T\*-aware auto rolling-KV spill (HAL #582 P1+, #621 follow-on)

Status: **T1–T5 VALIDATED on both hosts, 2026-07-11. T5 quality/tps gate PASS.** Opt-in under
`OPENCOTI_KV_AUTO_TIER=1` (same switch as the 0116 auto-tier). Builds on patch 0116
(`opencoti_auto_select_kv_tier`) and the rolling-KV window (0070/POSITION_WINDOW). **Host-only
— the FA microbench dispatches to the existing CUDA DSO, so no DSO rebuild.** Byte-identical
when the env is unset (opt-in gate returns before any measurement). Patch 0118 capture +
final commit are the only remaining step (held for explicit go) — see "T5 verdict" below.

## T5 verdict (2026-07-11, bs2 GPU0, Qwen2.5-14B-1M Q8_0, ctx 65536, DCA-off)

The physics reframes the gate: T\*(0.20) is only ~14–17 MiB at this config (small `t_resident`
at 65k, `c_engage ~1 ms` eats most of `0.20·t_resident`) while f16 spills ~2 GB, so the SHIP
question is not "does auto KEEP f16" (a razor-thin band) but **"is T\*'s quantize-vs-keep
verdict CORRECT?"**. Four uniform RULER-niah cells (`--vram-target 27000` forces the spill):

| cell | niah | decode tps |
|---|---|---|
| f16_resident (vram 0) | 100.0 | 50.2 |
| f16_spill (~2 GB, window) | 100.0 | **10.81** |
| q8_resident (fits) | 100.0 | 35.26 |
| auto (`OPENCOTI_KV_AUTO_TIER=1`) | 100.0 | 35.28 |

**PASS — all four checks:** (1) **auto → q8_0/q8_0**: boot WARN shows *"top tier f16 SPILLS by
1630 MiB … T\* tolerable spill 14 MiB (bw_eff=10.3 host-RAM-bound, target_drop=0.20) → auto KV
tier = q8_0/q8_0 (fits)"* — the drop-model rejects f16 because 1630 ≫ 14 MiB. (2) **f16-spill tps
drop = 78.5%** (50.2→10.81) ≫ 20% target — keeping f16 would cost far more than `target_drop`,
so rejecting it was justified (host-RAM-bound 10.3 GB/s link craters even a 2 GB spill). (3) **q8
niah = f16 niah = 100.0** — quantizing lossless-enough; f16_spill niah 100.0 also confirms the
streaming window is itself lossless. (4) **q8 tps 35.26 ≥ f16_spill 10.81** — the fallback
strictly dominates (3.3× faster AND lossless). Consistency: **auto 35.28 ≈ q8_resident 35.26**
(auto's end-to-end matches the tier it selected). Correctness via RULER niah, never greedy needle.

## Validation (2026-07-11, Qwen2.5-14B-Instruct-1M, both hosts)

Measured terms confirm T2 (bandwidth probes) + T3 (c_engage microbench) fire and T4 (drop model)
responds monotonically to `target_drop`:

| host | pcie | host_ram (1-thread memcpy) | **bw_eff = min** | binding | fa_ms/cell | c_engage |
|---|---|---|---|---|---|---|
| solidPC (3090, PCIe4, DDR4) | 6.7 | 11.5 | **6.7** | PCIe-bound | 6.2e-4 | 0.80 ms |
| bs2 (RTX 6000, PCIe5, DDR5) | 51.2 | 9.3 | **9.3** | **host-RAM-bound** | 2.6e-4 | 0.82 ms |

**req #3 demonstrated:** the binding bandwidth is host-dependent — PCIe on solidPC, host-RAM on
bs2 (fast PCIe5 makes DDR5 single-stream read the bottleneck). `min()` picks correctly on each;
without the host-RAM probe bw_eff would be 5× too optimistic on bs2. T\* drop-response (solidPC,
ctx 106 k, f16 spilling): tolerable spill **0 → 62 → 157 → 235 MiB** as drop 0.02 → 0.30 → 0.70
→ 0.99 (jitter-immune, same-boot). `T*=0` at drop 0.02 (c_engage > 0.02·t_resident) confirms the
safe degradation to the T1 "fully resident" gate. Byte-identical when env unset: 0 auto-tier WARN
lines. **Caveat:** on the 3090's 6.7 GB/s link T\* is a few tens–hundreds of MiB, so a tier *flip*
via T\* is only reachable at long ctx on high-bandwidth hosts (bs2); on the 3090 the model
correctly almost always quantizes. `host_ram` is a single-thread `memcpy` proxy (conservative
under-estimate of DMA-engine host read) — bug-2151 fixed a dead-store-elimination defect that had
made it return garbage (~2e7 GB/s). Diagnostics are WARN-level (`budget…top tier`, `T* drop-model
— …tolerable spill`) mirroring the existing `auto KV tier` WARN, for operator auditability.

## Problem

0116 auto-tier picks the least-compressing scalar KV pair that keeps the cache **fully
resident**. But a *higher* KV tier (better quality) with a *small* host-tail spill can beat a
harder-compressed fully-resident tier — **when** the spill tail is within a "negligible
tps-drop" band. This spec adds that decision: prefer `f16-window + small spill` over
quantizing, but only when the spill is provably cheap on *this* host + model + context.

## The decision (fit-condition reformulation)

The 0116 ladder walk changes from "first tier that fully fits" to "first (least-compressing)
tier whose **spill** ≤ T\*":

```
choose first tier i (least→most compression) s.t.  kv_bytes(i) ≤ budget + T*_bytes
```

`budget` is the existing resident budget (free − already-used − compute-reserve, capped by
`--vram-target`), identical to `opencoti_compute_resident_window_cells`. When even the floor
tier spills past T\*, pick the floor (rolling-KV spills the rest, as today). If f16's spill ≤
T\*, f16 is chosen and the window spills it — **Policy C: prefer f16-window+spill over
quantizing** (user, 2026-07-11).

## T\* — the negligible-drop tail band

Target drop = **20%** (user raised from 5% — at 5% it essentially never engaged because the
fixed streaming-engagement step already costs ~15%). T\* solved from a per-token drop model:

```
drop(tail) = ( c_engage + tail_bytes / bw_eff ) / t_resident
T*_bytes   = ( 0.20 · t_resident − c_engage ) · bw_eff          // ≤ 0  ⇒ never spill → quantize
```

Terms and how each user requirement maps in:

- **`t_resident`** = `(W_active + KV_resident_bytes) / vram_bw` — per-token time with the cache
  resident. `KV_resident` grows with context, so short-ctx ⇒ tiny `t_resident` ⇒ `c_engage`
  dominates ⇒ `T*≤0` ⇒ quantize (never spill a short-ctx cache); long-ctx ⇒ spill viable.
  **This is the ctx-value gate (req #2) — it falls out of the formula, no separate threshold.**
  `W_active`/`KV_resident` are summed from the model's real per-layer KV-head structure
  (`n_embd_k_gqa`/`n_embd_v_gqa`, global vs local heads) — **req #4 (attention structure)**.
- **`bw_eff`** = `min(pcie_bw_gbps, host_ram_bw_gbps)` — **req #3: RAM→GPU bandwidth matters,
  not just PCIe.** On solidPC (PCIe4 ~6.5 + DDR4) PCIe binds; on bs2 (PCIe5 ~50 + DDR5) they
  are comparable so `min` bites. `pcie_bw_gbps` exists (`common/pcie-profile.cpp`);
  `host_ram_bw_gbps` is NEW (boot memcpy probe or dmidecode DDR type/speed).
- **`vram_bw`** = GPU memory bandwidth (**req E**), host-side from `nvidia-smi`/device props.
- **`c_engage`** = fixed streaming-path engagement cost (the −15% first-spill step, §5c ≈
  ~1.4 ms). **Boot micro-measured (user chose option B).** It is model- and DCA-dependent, so a
  static constant is unreliable (§5c 256k shows −15% first-spill; the 12k clean p639 run showed
  <1% — same host). See "c_engage probe" below.
- **DCA (req #5)**: DCA-on vs DCA-off have different knees (§5c: DCA-off linear
  bandwidth-bound; DCA-on flat-then-sharp-cliff, post bug-2144). The probe runs in the actual
  `cparams.dca_enabled` state, so `c_engage` and the post-knee ceiling are captured for the
  live DCA setting automatically. A DCA-on validity ceiling clamps T\* to the pre-cliff band
  (~800 MiB observed on bs2).

### Anchors (sanity)
- 3090 (§4/#588): `t ≈ 23.6 ms + tail/6.5 GB/s` ⇒ T\*(20%) ≈ a few tens of MiB — essentially
  "quantize" on the slow link. ✓
- bs2 256k (§5c): −15% plateau holds to ~680 MiB then cliffs at ~1080 ⇒ T\*(20%) ≈ up to the
  ~680 MiB plateau edge once `t_resident` is large. ✓

## c_engage boot probe (req B — boot micro-measure)

The tier is chosen in the kv-cache ctor *before* allocation, but timing `c_engage` needs a
decode-capable path. Two viable mechanisms (pick at implementation):
1. **Standalone synthetic streaming-FA microbench** — build a tiny ggml graph on the CUDA
   backend: resident FA over a small K vs windowed FA (window + one-stride tail) over the same
   shapes; time both; `c_engage = t_windowed − t_resident`. Self-contained, no model decode,
   runs at ctor time. Preferred.
2. **Post-context two-phase** — build resident, run a few real decode steps resident vs
   forced-tiny-tail, measure, then (only if the tier would change) rebuild. Most faithful but
   double-allocates.
Cache the measured `c_engage` per `(device, dca_state, head_dim class)` in `.opencoti/` so
only the first boot pays the probe.

## Requirement #4 — honor an explicit `-ctk` as a block

**Simplification (2026-07-11): no new cparams fields / no cross-struct plumbing.** Auto-tier
runs in the ctor BEFORE it mutates `type_k`/`type_v`, and nothing else makes them non-f16 at
that point — so the *incoming* type already encodes explicitness. In
`opencoti_auto_select_kv_tier` (replacing the blanket `type_k != F16 || type_v != F16 →
return`):
- incoming `type_v != F16` → user set `-ctv` → **disable auto-tier entirely** (only rolling-KV
  window stays auto). User.
- incoming `type_k != F16` (and `type_v == F16`) → user set `-ctk` → **hold K fixed**, walk
  only the V axis (V-ladder `{f16, q8_0, q4_0}` under the pinned K).
- both F16 → full behavior above.

## Bandwidth acquisition — also no plumbing

- `pcie_bw_gbps` is ALREADY a `llama_kv_cache` ctor param (`cparams.pcie_bw_gbps`) — pass it to
  the helper (one-arg change).
- `vram_bw_gbps` and `host_ram_bw_gbps` are measured **inside the helper**, host-side
  (`nvidia-smi --query-gpu=…` bus/clock for VRAM bw; a one-shot `memcpy` bench for host DDR),
  cached in `.opencoti/` — no new cparams fields, no arg.cpp / llama_context_params threading.

## File surface (all additive, opt-in, byte-identical when env unset)

**Contained to ONE file** (`src/llama-kv-cache.cpp`), plus one ctor-arg addition (pass the
existing `pcie_bw_gbps` to the helper). No `cparams.h` / `arg.cpp` /
`llama_context_params` changes — see the two simplifications above. `common/pcie-profile.*` may
optionally host the host-RAM memcpy probe for reuse, but it can also live in the helper.

- `src/llama-kv-cache.cpp` — `opencoti_auto_select_kv_tier`: explicit-axis handling (T1), a
  host-side `vram_bw`/`host_ram_bw` probe (T2), the `c_engage` synthetic microbench (T3), the
  T\* drop-model fit-condition + DCA branch (T4). One extra ctor arg: `pcie_bw_gbps`.

## Increment plan (all in `llama-kv-cache.cpp` unless noted)

1. **T1 ✅** — explicit-axis handling in the helper (req #4): replace the blanket override guard
   with the V-disables / K-holds / both-walks logic. No plumbing (incoming types encode
   explicitness). Verified both hosts (bs2 + solidPC).
2. **T2 ✅** — `host_ram_bw` host-side probe (req #3), static-cached (env
   `OPENCOTI_KV_HOST_RAM_GBPS` override). `pcie_bw_gbps` passed from the ctor. Logs both +
   `bw_eff = min(pcie, host_ram)`. **VRAM bw dropped — subsumed by the FA microbench** (which
   reads KV from VRAM), so no separate device-memcpy probe. bug-2151: dead-store elision fixed.
3. **T3 ✅** — `c_engage` boot probe (req B) — reused the existing FA compute microbench
   (`opencoti_fa_compute_probe_ms`, #351) at two cell counts (8192 win, 256 floor) →
   `fa_ms_cell` + `c_engage`. **HOST-ONLY** (dispatches to the existing DSO — no CUDA rebuild).
4. **T4 ✅** — T\* drop-model fit-condition in the helper (20% default target, env
   `OPENCOTI_KV_TSTAR_DROP`; cliff `OPENCOTI_KV_TSTAR_MAX_SPILL_MIB` default 800). Fast-path
   guard keeps the common "f16 fits" boot byte-identical. `dca_enabled` plumbing deliberately
   avoided — universal 800 MiB cliff instead (conservative DCA-on pre-cliff band).
5. **T5 (PENDING)** — RULER-niah quality gate: verify auto picks f16-window+spill at long-ctx
   where it beats quant (niah + decode-tps ≥ the quant alternative). Needs a bs2 long-ctx cell
   where T\* is genuinely large enough to KEEP f16 (on the 3090 T\* is too small to flip). Then
   capture patch 0118 + bookkeeping. Build on both hosts already done.

## Standing gates
Correctness via RULER niah / logit-equiv, never greedy needle. Byte-identical no-op when
`OPENCOTI_KV_AUTO_TIER` unset. bs2 build (host-mostly; the synthetic probe may touch CUDA — if
so, DSO rebuild + dual-restamp + `nm -D`). Commit only when the user asks.