docs/features/tstar_auto_spill.md

T*-aware auto rolling-KV spill (HAL #582 P1+, #621 follow-on)

Status: T1–T5 VALIDATED on both hosts, 2026-07-11. T5 quality/tps gate PASS. Opt-in under OPENCOTI_KV_AUTO_TIER=1 (same switch as the 0116 auto-tier). Builds on patch 0116 (opencoti_auto_select_kv_tier) and the rolling-KV window (0070/POSITION_WINDOW). Host-only — the FA microbench dispatches to the existing CUDA DSO, so no DSO rebuild. Byte-identical when the env is unset (opt-in gate returns before any measurement). Patch 0118 capture + final commit are the only remaining step (held for explicit go) — see "T5 verdict" below.

T5 verdict (2026-07-11, bs2 GPU0, Qwen2.5-14B-1M Q8_0, ctx 65536, DCA-off)

The physics reframes the gate: T*(0.20) is only ~14–17 MiB at this config (small t_resident at 65k, c_engage ~1 ms eats most of 0.20·t_resident) while f16 spills ~2 GB, so the SHIP question is not "does auto KEEP f16" (a razor-thin band) but "is T*'s quantize-vs-keep verdict CORRECT?". Four uniform RULER-niah cells (--vram-target 27000 forces the spill):

cell	niah	decode tps
f16_resident (vram 0)	100.0	50.2
f16_spill (~2 GB, window)	100.0	10.81
q8_resident (fits)	100.0	35.26
auto (OPENCOTI_KV_AUTO_TIER=1)	100.0	35.28

PASS — all four checks: (1) auto → q8_0/q8_0: boot WARN shows "top tier f16 SPILLS by 1630 MiB … T* tolerable spill 14 MiB (bw_eff=10.3 host-RAM-bound, target_drop=0.20) → auto KV tier = q8_0/q8_0 (fits)" — the drop-model rejects f16 because 1630 ≫ 14 MiB. (2) f16-spill tps drop = 78.5% (50.2→10.81) ≫ 20% target — keeping f16 would cost far more than target_drop, so rejecting it was justified (host-RAM-bound 10.3 GB/s link craters even a 2 GB spill). (3) q8 niah = f16 niah = 100.0 — quantizing lossless-enough; f16_spill niah 100.0 also confirms the streaming window is itself lossless. (4) q8 tps 35.26 ≥ f16_spill 10.81 — the fallback strictly dominates (3.3× faster AND lossless). Consistency: auto 35.28 ≈ q8_resident 35.26 (auto's end-to-end matches the tier it selected). Correctness via RULER niah, never greedy needle.

Validation (2026-07-11, Qwen2.5-14B-Instruct-1M, both hosts)

Measured terms confirm T2 (bandwidth probes) + T3 (c_engage microbench) fire and T4 (drop model) responds monotonically to target_drop:

host	pcie	host_ram (1-thread memcpy)	bw_eff = min	binding	fa_ms/cell	c_engage
solidPC (3090, PCIe4, DDR4)	6.7	11.5	6.7	PCIe-bound	6.2e-4	0.80 ms
bs2 (RTX 6000, PCIe5, DDR5)	51.2	9.3	9.3	host-RAM-bound	2.6e-4	0.82 ms

req #3 demonstrated: the binding bandwidth is host-dependent — PCIe on solidPC, host-RAM on bs2 (fast PCIe5 makes DDR5 single-stream read the bottleneck). min() picks correctly on each; without the host-RAM probe bw_eff would be 5× too optimistic on bs2. T* drop-response (solidPC, ctx 106 k, f16 spilling): tolerable spill 0 → 62 → 157 → 235 MiB as drop 0.02 → 0.30 → 0.70 → 0.99 (jitter-immune, same-boot). T*=0 at drop 0.02 (c_engage > 0.02·t_resident) confirms the safe degradation to the T1 "fully resident" gate. Byte-identical when env unset: 0 auto-tier WARN lines. Caveat: on the 3090's 6.7 GB/s link T* is a few tens–hundreds of MiB, so a tier flip via T* is only reachable at long ctx on high-bandwidth hosts (bs2); on the 3090 the model correctly almost always quantizes. host_ram is a single-thread memcpy proxy (conservative under-estimate of DMA-engine host read) — bug-2151 fixed a dead-store-elimination defect that had made it return garbage (~2e7 GB/s). Diagnostics are WARN-level (budget…top tier, T* drop-model — …tolerable spill) mirroring the existing auto KV tier WARN, for operator auditability.

Problem

0116 auto-tier picks the least-compressing scalar KV pair that keeps the cache fully resident. But a higher KV tier (better quality) with a small host-tail spill can beat a harder-compressed fully-resident tier — when the spill tail is within a "negligible tps-drop" band. This spec adds that decision: prefer f16-window + small spill over quantizing, but only when the spill is provably cheap on this host + model + context.

The decision (fit-condition reformulation)

The 0116 ladder walk changes from "first tier that fully fits" to "first (least-compressing) tier whose spill ≤ T*":

choose first tier i (least→most compression) s.t.  kv_bytes(i) ≤ budget + T*_bytes

budget is the existing resident budget (free − already-used − compute-reserve, capped by --vram-target), identical to opencoti_compute_resident_window_cells. When even the floor tier spills past T*, pick the floor (rolling-KV spills the rest, as today). If f16's spill ≤ T*, f16 is chosen and the window spills it — Policy C: prefer f16-window+spill over quantizing (user, 2026-07-11).

T* — the negligible-drop tail band

Target drop = 20% (user raised from 5% — at 5% it essentially never engaged because the fixed streaming-engagement step already costs ~15%). T* solved from a per-token drop model:

drop(tail) = ( c_engage + tail_bytes / bw_eff ) / t_resident
T*_bytes   = ( 0.20 · t_resident − c_engage ) · bw_eff          // ≤ 0  ⇒ never spill → quantize

Terms and how each user requirement maps in:

• t_resident = (W_active + KV_resident_bytes) / vram_bw — per-token time with the cache resident. KV_resident grows with context, so short-ctx ⇒ tiny t_resident ⇒ c_engage dominates ⇒ T*≤0 ⇒ quantize (never spill a short-ctx cache); long-ctx ⇒ spill viable. This is the ctx-value gate (req #2) — it falls out of the formula, no separate threshold. W_active/KV_resident are summed from the model's real per-layer KV-head structure (n_embd_k_gqa/n_embd_v_gqa, global vs local heads) — req #4 (attention structure).
• bw_eff = min(pcie_bw_gbps, host_ram_bw_gbps) — req #3: RAM→GPU bandwidth matters, not just PCIe. On solidPC (PCIe4 ~6.5 + DDR4) PCIe binds; on bs2 (PCIe5 ~50 + DDR5) they are comparable so min bites. pcie_bw_gbps exists (common/pcie-profile.cpp); host_ram_bw_gbps is NEW (boot memcpy probe or dmidecode DDR type/speed).
• vram_bw = GPU memory bandwidth (req E), host-side from nvidia-smi/device props.
• c_engage = fixed streaming-path engagement cost (the −15% first-spill step, §5c ≈ ~1.4 ms). Boot micro-measured (user chose option B). It is model- and DCA-dependent, so a static constant is unreliable (§5c 256k shows −15% first-spill; the 12k clean p639 run showed <1% — same host). See "c_engage probe" below.
• DCA (req #5): DCA-on vs DCA-off have different knees (§5c: DCA-off linear bandwidth-bound; DCA-on flat-then-sharp-cliff, post bug-2144). The probe runs in the actual cparams.dca_enabled state, so c_engage and the post-knee ceiling are captured for the live DCA setting automatically. A DCA-on validity ceiling clamps T* to the pre-cliff band (~800 MiB observed on bs2).

Anchors (sanity)

• 3090 (§4/#588): t ≈ 23.6 ms + tail/6.5 GB/s ⇒ T*(20%) ≈ a few tens of MiB — essentially "quantize" on the slow link. ✓
• bs2 256k (§5c): −15% plateau holds to ~680 MiB then cliffs at ~1080 ⇒ T*(20%) ≈ up to the ~680 MiB plateau edge once t_resident is large. ✓

c_engage boot probe (req B — boot micro-measure)

The tier is chosen in the kv-cache ctor before allocation, but timing c_engage needs a decode-capable path. Two viable mechanisms (pick at implementation):

1. Standalone synthetic streaming-FA microbench — build a tiny ggml graph on the CUDA backend: resident FA over a small K vs windowed FA (window + one-stride tail) over the same shapes; time both; c_engage = t_windowed − t_resident. Self-contained, no model decode, runs at ctor time. Preferred.
2. Post-context two-phase — build resident, run a few real decode steps resident vs forced-tiny-tail, measure, then (only if the tier would change) rebuild. Most faithful but double-allocates. Cache the measured c_engage per (device, dca_state, head_dim class) in .opencoti/ so only the first boot pays the probe.

Requirement #4 — honor an explicit -ctk as a block

Simplification (2026-07-11): no new cparams fields / no cross-struct plumbing. Auto-tier runs in the ctor BEFORE it mutates type_k/type_v, and nothing else makes them non-f16 at that point — so the incoming type already encodes explicitness. In opencoti_auto_select_kv_tier (replacing the blanket type_k != F16 || type_v != F16 → return):

• incoming type_v != F16 → user set -ctv → disable auto-tier entirely (only rolling-KV window stays auto). User.
• incoming type_k != F16 (and type_v == F16) → user set -ctk → hold K fixed, walk only the V axis (V-ladder {f16, q8_0, q4_0} under the pinned K).
• both F16 → full behavior above.

Bandwidth acquisition — also no plumbing

• pcie_bw_gbps is ALREADY a llama_kv_cache ctor param (cparams.pcie_bw_gbps) — pass it to the helper (one-arg change).
• vram_bw_gbps and host_ram_bw_gbps are measured inside the helper, host-side (nvidia-smi --query-gpu=… bus/clock for VRAM bw; a one-shot memcpy bench for host DDR), cached in .opencoti/ — no new cparams fields, no arg.cpp / llama_context_params threading.

File surface (all additive, opt-in, byte-identical when env unset)

Contained to ONE file (src/llama-kv-cache.cpp), plus one ctor-arg addition (pass the existing pcie_bw_gbps to the helper). No cparams.h / arg.cpp / llama_context_params changes — see the two simplifications above. common/pcie-profile.* may optionally host the host-RAM memcpy probe for reuse, but it can also live in the helper.

• src/llama-kv-cache.cpp — opencoti_auto_select_kv_tier: explicit-axis handling (T1), a host-side vram_bw/host_ram_bw probe (T2), the c_engage synthetic microbench (T3), the T* drop-model fit-condition + DCA branch (T4). One extra ctor arg: pcie_bw_gbps.

Increment plan (all in llama-kv-cache.cpp unless noted)

1. T1 ✅ — explicit-axis handling in the helper (req #4): replace the blanket override guard with the V-disables / K-holds / both-walks logic. No plumbing (incoming types encode explicitness). Verified both hosts (bs2 + solidPC).
2. T2 ✅ — host_ram_bw host-side probe (req #3), static-cached (env OPENCOTI_KV_HOST_RAM_GBPS override). pcie_bw_gbps passed from the ctor. Logs both + bw_eff = min(pcie, host_ram). VRAM bw dropped — subsumed by the FA microbench (which reads KV from VRAM), so no separate device-memcpy probe. bug-2151: dead-store elision fixed.
3. T3 ✅ — c_engage boot probe (req B) — reused the existing FA compute microbench (opencoti_fa_compute_probe_ms, #351) at two cell counts (8192 win, 256 floor) → fa_ms_cell + c_engage. HOST-ONLY (dispatches to the existing DSO — no CUDA rebuild).
4. T4 ✅ — T* drop-model fit-condition in the helper (20% default target, env OPENCOTI_KV_TSTAR_DROP; cliff OPENCOTI_KV_TSTAR_MAX_SPILL_MIB default 800). Fast-path guard keeps the common "f16 fits" boot byte-identical. dca_enabled plumbing deliberately avoided — universal 800 MiB cliff instead (conservative DCA-on pre-cliff band).
5. T5 (PENDING) — RULER-niah quality gate: verify auto picks f16-window+spill at long-ctx where it beats quant (niah + decode-tps ≥ the quant alternative). Needs a bs2 long-ctx cell where T* is genuinely large enough to KEEP f16 (on the 3090 T* is too small to flip). Then capture patch 0118 + bookkeeping. Build on both hosts already done.

Standing gates

Correctness via RULER niah / logit-equiv, never greedy needle. Byte-identical no-op when OPENCOTI_KV_AUTO_TIER unset. bs2 build (host-mostly; the synthetic probe may touch CUDA — if so, DSO rebuild + dual-restamp + nm -D). Commit only when the user asks.