# Sparse Attention — Quest-style block-selector for long-context decode (#551)

> Status: BUILT + CHARACTERIZED on the 3090 (2026-06-27); 1M validation on bs2
> pending. The real 1M **decode** lever — companion to the DCA + quant-KV
> **prefill** work (#444/#445/#554). Default-OFF, byte-identical when off. Develops
> on the 3090 (A4B / Qwen3-4B / 27B-Q4 @ ≤256k); 1M validation on bs2.
>
> **Headline result (§S4 results):** the sparse/dense **decode** ratio is
> *context-driven, not batch-driven*. Single-stream (B1), Qwen3-4B, f16 KV, 25%
> block coverage on the 3090, the ratio climbs monotonically and the gap to dense
> closes 9.2% → 0.8% from 16K to the model's 40K ceiling (0.908 → 0.992). Crossover
> (>1.0, a real decode win) extrapolates just past 40K — the 256k/1M regime on bs2.
> Batch is *not* our lever (unlike TheTom — see §S4 results for why).

## Why this, and why now

The 1M serving path has two halves. **Prefill** is handled by DCA's analytical
band + quantized KV (q5_0/q4_0 — f16 KV at 1M is 206 GB, impossible), and #554
removed the whole-cache-lift prefill cliff. **Decode is still O(n_kv) per token**:
at 1M every generated token attends to ~1M KV entries, memory-bound and slow. The
only thing that breaks that is *sparse attention* — attend to a small, query-chosen
subset of KV blocks instead of all of them.

## Method: Quest-style min/max block bounds (decided 2026-06-25)

Reference scouted: **TheTom/turboquant_plus** `docs/papers/block-selector-sparse-attention.md`.
Decisive finding for us: that work is **Apple Silicon / MLX**, where TheTom found
custom sparse kernels with *real* K-skipping LOSE (14× at B=8) to the tuned
`sdpa_vector_2pass`, so he uses a float mask that still streams the full K/V (saves
compute, **not bandwidth**). He explicitly notes the literature wins (Quest, NSA,
SeerAttention, DuoAttention) "all target **CUDA, where FlashAttention's sparse kernel
is mature**." **We are CUDA.** Our FA-VEC already has the `KV_max`/`KV_min` tile-trim
(`fattn-vec.cuh:363`) that lets us skip **whole KV blocks before the QK dot** — real
K-read + QK + V savings — which TheTom's Metal stack could not. So we build the thing
he couldn't, using the cleaner CUDA ancestor:

**Quest** (Tang 2024). Parameter-free, exact upper-bound:
- **Per-(KV-head, block) min/max key bounds.** Block size `B_SEL` (start 64; tune).
  Store `kmin[head_dim]`, `kmax[head_dim]` per block — a tiny side-cache
  (`n_blocks × head_dim × 2 × f16`). For quant-KV, compute min/max from the f16
  values **before** quantizing (composes with q5_0/q4_0).
- **Per-query upper bound.** For query `q`, the max possible `q·k` over a block is,
  channel-wise, `Σ_d max(q_d·kmin_d, q_d·kmax_d)` (use kmax where q_d>0, kmin where
  q_d<0). O(n_blocks·head_dim) — negligible vs O(n_kv·head_dim) dense attention.
- **Top-K block selection.** Keep the top-`K_SEL` blocks by upper bound **PLUS** the
  recent window + attention-sink block (streaming-LLM safety so recency/retrieval is
  never silently dropped). `K_SEL` and window are knobs; tune on RULER-niah.
- **Kernel skip.** Extend the FA-VEC tile loop to `continue` past unselected blocks —
  the real K+QK+V skip.

Budget policy: **fixed top-K + recent/sink** (decided). TheTom found adaptive top-K
"never beat union top-K" and converged loosely at long ctx; fixed-K is predictable in
VRAM/compute and easy to gate.

## Compose (three sparsity levers, three pipeline stages)

1. **Block-selector (this work)** — skip K blocks *before* QK (saves K-read + QK + V).
2. **DCA analytical band** — the selector narrows *within* each INTRA/SUCC/INTER band.
3. **sparse-V (#546)** — within selected blocks, skip V dequant for negligible-weight
   positions (the residual, post-softmax).
Plus **quant-KV** (min/max from dequant K at write). The three are independent gates at
the block / band / position granularities.

## Where it lives (insertion points, from the Explore map)

- **Write path (S1):** per-block min/max side-cache, filled as K is written
  (KV-cache write in `llama-graph.cpp` / the cpy path). Additive tensor; off ⇒ not allocated.
- **Selector (S2):** host/graph pre-pass computing the per-query block mask from the
  bounds; carried to the kernel by **extending the existing `KV_max`/`KV_min`
  mechanism** (`fattn-common.cuh:27-28`, `fattn-vec.cuh:352-353`) with a block-select
  bitmask/priority.
- **Kernel (S3):** block-skip `continue` in the FA-VEC tile loop at `fattn-vec.cuh:363`
  (the long-ctx **decode** hot path is FA-VEC, not MMA).

## S1 integration anchors (mapped 2026-06-25, read-only)

All file:line in `vendors/sources/llamafile/llama.cpp/`:
- **K write (scatter):** `src/llama-graph.cpp:2771` — `mctx_cur->cpy_k(ctx0, k_cur, k_idxs, il)` (impl in `llama-kv-cache.cpp`). The per-block min/max must be derived from `k_cur` here (pre-quant) or right after, per write.
- **KV tensor alloc (add the side-tensor here):** `src/llama-kv-cache.cpp:637` (GPU `ggml_new_tensor_3d(ctx_s, type_k, n_embd_k_gpu, wc, 1)`) + `:686` (host); stored in the `layers` vector (`llama_kv_cache_layer`, pushed `:738`). Add a parallel per-(KV-head, block) `kmin/kmax` tensor alongside `k_l`.
- **Host-fill model:** `set_input_dca` `src/llama-kv-cache.cpp:3899-4040` (dispatch `:5250`) — the idiom for filling a per-forward input tensor (direct `tensor->data` writes / `ggml_backend_tensor_set`); mirror for the selector's per-block mask in S2.
- **Flag plumbing (mirror `--dca` for `--sparse-attn*`):** `common/arg.cpp:1516,1528` (add_opt) · `src/llama-cparams.h:56-57` (cparams fields) · `common/common.cpp:1649-1650` (params→cparams) · `tools/server/server-context.cpp:1102` (server gate).
- **Sizing accessors:** `llama-hparams.h:268` `n_head_kv(il)` · `:283` `n_embd_head_k(il)` · `cparams.n_ctx` (graph ctor `llama-graph.cpp:1009`).
- **S1 sequence:** vendor-backup WHOLE tree FIRST → add `--sparse-attn{,-topk,-block-size}` flags (default off) → alloc the kmin/kmax side-tensor (only when enabled) → fill it at/after the `cpy_k` write → build host + verify default-off path is byte-identical (no new tensor allocated, no graph change). Selector + kernel skip are S2/S3.

## Correctness gate (NEVER greedy needle — bug-263/270/354)

- **RULER-niah @256k** on the 3090 (A4B + Qwen3-4B): the needle's block MUST survive
  selection — niah recall is the primary signal that block-selection didn't drop the
  answer.
- **Logit-equivalence vs dense** (`real_frac`, top-k agreement) as the tolerance check.
- Perf: decode + prefill tps vs dense, sparsity-ratio sweep, at 32k/256k (3090) then
  512k/1M (bs2).

## Knobs (default OFF, byte-identical)

- `--sparse-attn {off,on}` (or auto by ctx threshold)
- `--sparse-attn-block-size` (B_SEL, default 64)
- `--sparse-attn-topk` (K_SEL blocks) and `--sparse-attn-recent` (always-keep window)
- env mirrors for dev (e.g. `OPENCOTI_SPARSE_ATTN_*`).

## Staging

- **S1** — per-block min/max side-cache + write-path plumbing (host). Off ⇒ byte-identical.
- **S2** — Quest selector (query·bounds → top-K + recent/sink mask) + KV_max/KV_min carry.
- **S3** — FA-VEC kernel block-skip in the tile loop.
- **S4** — gate: RULER-niah @256k + logit-equiv vs dense + decode/prefill tps (3090).
- **S5** — compose with sparse-V #546 and DCA bands; re-gate.
- **S6** — capture patch (vendor-backup + snapshot-diff, byte-identical re-apply) +
  this doc's measured table + UPSTREAM_SYNC markers + STATE_SUMMARY + .wolf.

## S2/S3 engineering design (mapped + decided 2026-06-25)

Integration reuses the proven `KV_max`/`KV_min` shape (pool-alloc + pre-pass fill +
kernel param), but with one decisive difference: the per-step fill **writes** kmin/kmax
and the selector **reads** them *in the same graph*, so ordering is enforced with real
ggml **src edges** (two new graph ops) — **not** the DCA thread-local side-channel
(`opencoti_fattn_dca_pos_q`, `ggml-cuda/fattn.cu:27-29`), which is safe only for
analytical inputs with no shared-buffer write-then-read.

**Three pieces:**

1. **Fill op** `sparse_attn_fill` (new GGML custom op; pattern = the `ggml_streaming_*`
   / `ggml_turbo_wht` ops #345/#391).
   - Inserted in `build_attn` right after each `cpy_k` (`llama-graph.cpp:2771`; iSWA
     `:2985`, `:3067`), gated on `cparams.sparse_attn_enabled`.
   - srcs: `k_cur` (new tokens' K, **pre-quant f16**) + `k_idxs` (scatter positions).
     dst-update: `kmin_l`/`kmax_l` cache tensors (`build_forward_expand`, like `cpy_k`;
     the write into the persistent cache tensor + the selector's read edge needs the
     `ggml_view_1d` dep-tie idiom — bug-225 precedent — so the scheduler sees fill→select).
   - CUDA kernel: per (KV-head, channel), reduce min/max over each affected block's
     positions; **min/max-merge with the existing** kmin/kmax (blocks fill incrementally
     across decode). Affected blocks = `[n_past/B, (n_past+n_new)/B]`.
   - Off ⇒ op not added ⇒ byte-identical.

2. **Selector op** `sparse_attn_select` (new GGML custom op; pre-pass shape mirrors
   `flash_attn_dca_to_KV_max`, `fattn-common.cuh:891`).
   - Inserted after `get_k` (`:2813`) before the FA op.
   - srcs: `Q` (rotated FA query) + `kmin_l` + `kmax_l` (the read edge orders it after the
     fill). dst: `block_sel` bitmask (`n_blocks` bits per `(seq, KV-head)`, I32-packed).
   - CUDA kernel: per block b, UB = `Σ_d max(q_d·kmin_b[d], q_d·kmax_b[d])`; **GQA** → max
     UB over the query group per KV-head (one block set per KV-head, NSA co-load); top-`K_SEL`
     by UB (smem partial-sort / threshold) **OR'd with** the recent-window + sink blocks;
     write bits.

3. **FA-VEC block-skip** (S3).
   - `block_sel` rides as an **extra src on the FA op** — set `result->src[5] = block_sel`
     after `ggml_flash_attn_ext` (mirror of `sinks`=src[4]); the scheduler builds the
     selector→FA edge from the src slot regardless of the core op. `launch_fattn` passes
     `block_sel->data` as a new kernel param `const int * __restrict__ block_sel` alongside
     `KV_max`/`KV_min` (`fattn-vec.cuh:83-84`, `fattn-common.cuh:27-28`).
   - Tile loop (`fattn-vec.cuh:363`, immediately after the existing `k_VKQ_min` skip):
     `if (block_sel && !(block_sel[base + ((k_VKQ_0/B)>>5)] & (1u<<((k_VKQ_0/B)&31)))) continue;`.
     B (= `B_SEL`) aligned to the FA tile stride (`nthreads`) and the quant blck (32) ⇒
     `B_SEL` a multiple of both for clean skipping.
   - Threaded through every FA-VEC instance signature (exactly as `KV_max` was). The MMA
     prefill path is a **separate later stage** (S1–S4 target FA-VEC decode).

**Gate ladder (NEVER greedy needle — bug-263/270/354):**
- **Plumbing anchor:** `K_SEL = n_blocks` (select-all) ⇒ sparse must equal dense ⇒
  logit-equiv `real_frac=0`. Proves the carry + skip are exact before any real sparsity.
- Then reduce `K_SEL`: RULER-niah@256k (needle's block must survive) + logit-equiv
  tolerance + decode/prefill tps vs dense, sparsity-ratio sweep (3090 @ 32k/256k → bs2 1M).

**Build reality:** fill+selector+skip ⇒ CUDA DSO rebuild (~25 min/iter, restamp BOTH DSO
paths). Min 2 builds (B1: fill — verify kmin/kmax vs a CPU min/max reference via a debug
dump; B2: selector+skip — select-all anchor then sparsity sweep). Plan 2–4 iterations.

## Open questions (resolve during impl)

- Block size vs the FA-VEC 128-thread tile stride and the quant block size (q5_0/q4_0
  blck=32) — align B_SEL to a multiple of both for clean skipping.
- GQA: bounds are per-KV-head; the selector reduces the GQA query group to a per-KV-head
  representative (max over the group's per-block upper bounds) so all grouped Q-heads
  share one block set (NSA-style co-load) — avoids per-Q-head mask divergence.
- Interaction with the DCA fused MMA path (prefill) — S1–S4 target FA-VEC decode; the
  MMA/prefill block-selector (TheTom's 2.9×@128K prefill analogue) is a later stage.
- Where to compute the selector: a small dedicated CUDA op vs folding into the FA launch.

## Comparison vs TheTom (S4 — user-required, "test the same as theTom to compare")

Reproduce TheTom's exact block-selector benchmark so our numbers sit directly beside
theirs. Their config (from `turboquant_plus/docs/papers/block-selector-sparse-attention.md`,
fetched 2026-06-25):

- **Model:** `Qwen2.5-14B-Instruct-1M-4bit` — **we have this exact model on bs2** (the #550
  DCA vehicle; native-1M, no-YaRN — see `project_qwen25_1m_native_no_yarn.md`). Same model ⇒
  direct apples-to-apples.
- **Grid:** batch `B ∈ {1,2,4,8}` × ctx `{16K … 131072}` (their 19-cell grid). B>1 uses our
  PolyKV `--parallel`. Metric = **dense-vs-sparse decode + prefill tps ratio**.
- **Block size:** 64 — matches our locked `B_SEL=64`.
- **Their headline (M5 Max / MLX):** decode **1.19× (B4/16K) → 1.73× (B8/28K) → 1.48× (B4/48K)**;
  prefill **2.49× (64K) → 2.90× (128K)**. These are the numbers to match/beat.
- **Their quality validation is INCOMPLETE** — *no* clean end-to-end retrieval benchmark;
  only cosine-vs-dense (0.999 prefill / 0.99995 long-ctx adaptive-topK). **We add a real
  RULER-niah@{256K,1M} retrieval gate they lack** — a strict improvement on their eval.

**Why ours should win the decode column:** TheTom's selector is a **JL-projection
float-additive mask** (`contentDim=32`, `-∞` for non-selected) — *compute-only masking, no
bandwidth save* — because MLX/Metal couldn't do a real K-skip (cerebrum 2026-06-21). Ours is
**Quest min/max block-skip on CUDA FA-VEC** that `continue`s past unselected blocks BEFORE
the K-read + QK + V — a real bandwidth save — so our decode ratio should *exceed* their
1.19–1.73× at the same sparsity, not just match it. Prefill (MMA path) is a later stage
(their 2.49–2.90× is the prefill-selector analogue, S5+).

> **CORRECTION (2026-06-27), see §S4 results:** the "should *exceed* their 1.19–1.73×"
> claim was over-stated *for the batched column*. TheTom's wins are at **batch 4–8**;
> ours are at **context length** — different mechanisms (his fixed mask-cost is
> amortized across streams; our real K-skip is a *fixed per-stream* cost that batching
> can't amortize). So at his batch points we sit near break-even, not above. Where the
> prediction holds is the axis he does *not* measure — single-stream long-context — and
> the 3090 ladder (0.908→0.992, climbing) is the first hard evidence for it.

**S4 deliverable:** a side-by-side table — same model, same B×ctx grid, dense-vs-sparse
decode/prefill ratio (ours, on the 3090 ≤128K then bs2 to 1M) next to TheTom's MLX column,
PLUS our RULER-niah column. Run on bs2 (the 14B-1M lives there) to mirror their long-ctx grid.

## S4 results — 3090 single-stream ctx-ladder + batch grid (2026-06-27)

The TheTom comparison ran in two parts on the 3090 (bs2/14B-1M busy; the same-model
1M grid is still pending). Model: **Qwen3-4B-Q4_K_M**, f16 KV, 25% block coverage,
block size 64, Quest selector (`--sparse-attn on`). Metric = dense-vs-sparse **decode**
throughput ratio. Decode tps read from the server log's authoritative
`eval time … / 128 tokens … tokens per second` line — *not* the HTTP JSON, which is
unreliable under the bug-743 parse-race (see Gotchas).

### Single-stream ctx-ladder (B1) — the decisive axis

| ctx (B1) | dense tps | sparse tps | ratio | gap to dense |
|---------:|----------:|-----------:|------:|-------------:|
| 16384 | 74.50 | 67.65 | 0.908 | −9.2% |
| 24576 | 61.54 | 59.37 | 0.965 | −3.5% |
| 32768 | 53.41 | 52.06 | 0.975 | −2.5% |
| 40000 | 47.46 | 47.06 | **0.992** | −0.8% |

Monotonic climb; the gap halves roughly every ~8–16K. At Qwen3-4B's native 40960
ceiling we are within measurement noise of dense, and the trend extrapolates to
crossover (>1.0) just beyond — i.e. the 256k/1M serving regime. This is the expected
signature of a *real bandwidth-saving K-skip*: saved K-reads grow with cache size
while the selector's fill + QK-estimate + `block_sel`-gather cost is ~fixed, so the
ratio rises with context. (Script: `.opencoti/sparse-ctx-ladder.sh`, gitignored.)

### Batch grid (B1/B2/B4) — confirms batch is NOT our lever

| ctx/seq | B | dense_agg | sparse_agg | ratio | note |
|--------:|--:|----------:|-----------:|------:|------|
| 16384 | 1 | 74.5 | 67.3 | 0.903 | clean |
| 16384 | 2 | 62.6 | 12.0 | 0.192 | bug-743 parse-race (one req died, n<B) |
| 16384 | 4 | 42.4 | 11.3 | 0.267 | bug-743 parse-race (n=3) |
| 24576 | 1 | 60.9 | 58.1 | 0.954 | clean |
| 24576 | 2 | 44.9 | 43.2 | 0.962 | clean — sparse ~3% slower on **both** slots |
| 24576 | 4 | 27.7 | 27.7 | 1.000 | clean — uniform ~3% penalty washed out by scheduler asymmetry |

Per-slot eval times on the clean 24K cells show sparse ~3% slower per stream (e.g.
B2 dense 39.6/5.3 tps vs sparse 38.0/5.2). The 24K "1.000" is *not* a batched win — it
is the uniform per-stream penalty hidden by one-fast-slot/three-starved scheduler
asymmetry. So batching gives our selector **no** multiplier. (Script:
`.opencoti/sparse-batch-grid.sh`.)

### Verdict — we win a different game than TheTom

| | TheTom (turboquant_plus) | ours (#551 Quest) |
|---|---|---|
| Selector | JL-projection float-additive mask (compute-only, no bandwidth save) | Quest min/max block-skip on CUDA FA-VEC (real K/V-read skip) |
| Winning axis | **batch** (1.19× B4/16K → 1.73× B8/28K → 1.48× B4/48K) | **context** (0.908@16K → 0.992@40K, B1) |
| B1 behaviour | structurally ≤1.0 (mask adds work, saves nothing at B1) | climbs to ~1.0 and past it with ctx |
| Quality eval | cosine-vs-dense only (0.999) | RULER-niah retrieval gate (a real gate he lacks) |
| Hardware/model | M5 Max / MLX / 14B-1M-4bit | 3090 / CUDA / Qwen3-4B f16 |

His mask saves no bandwidth, so his win **needs** batch to amortize a fixed selector
cost across streams; remove the batch and his advantage disappears. Ours saves real
bandwidth per stream, so it scales with **context** and batching can't amortize a
per-stream cost. Consequences:

- **At B1 (single-stream serving) we should beat him** — we climb to ~1.0 with ctx via
  real K-skip; his compute-only mask is structurally ≤1.0 at B1 (he doesn't report B1).
- **At B8/28K (his sweet spot) he wins on his hardware and we get no batch multiplier** —
  we sit near break-even there.
- **On validation we are strictly ahead** — RULER-niah vs cosine-0.999.

A true number-against-his-column comparison is still **not** apples-to-apples (different
HW/model/quant). The only direct side-by-side is the still-pending S4 deliverable: the
same 14B-1M-4bit, the same B×ctx grid, on bs2 to 1M, **plus** our RULER-niah column.

### Gotchas surfaced

- **bug-743 (parse-race):** under load the server can feed a request's *generated*
  filler continuation back into the JSON parser → `500 "Failed to parse input at pos 0"`,
  which `curl` receives as the response (0 tps). It fires intermittently even at
  `--parallel 1`. **Always read decode tps from the server log, not the HTTP JSON.** A
  reliability bug to fix before batched sparse can be trusted in production.

## DeepSeek Sparse Attention (DSA) — investigated, NOT portable (2026-06-27)

Investigated per user request (Reddit thread on a llama.cpp DSA effort). DSA
(arXiv:2512.02556) is **not transferable** to our Qwen/Gemma targets:

- It needs a **trained "lightning indexer" head** (FP8, ReLU-gated, KL-distilled from
  dense attention) plus ~**946B tokens** of base-weight sparsity adaptation, and is
  **MLA/MQA-coupled** — our models are GQA with no indexer weights.
- The llama.cpp implementation (fairydreaming/sszymczyk, fork branch `deepseek-dsa`,
  merged upstream as **PR #23346**, 2026-05-29) is **correctness-only** — the author's
  own note says it "doesn't improve long-context performance yet" (oversized compute
  buffers).
- The one transferable idea — a small trained indexer head — is a **training project**,
  not a port. Out of scope for the vendored-fork decode lever.

## All-KV-types composition (bug-747, 2026-06-27)

Sparse attention originally composed **only with f16 KV**: the B1 fill op
(`GGML_OP_SPARSE_ATTN_FILL`) read the *quantized cache* K view, and its CUDA
`supports_op` + launcher required `src[1]` to be `GGML_TYPE_F16`. With any quantized KV
cache (`q8_0`/`q4_0`/`q6_0`/turbo*/tcq*) the cache K is non-f16 → `supports_op=0` on
CUDA0 → the scheduler could not place the op on the buffer holding the pre-allocated
`kbounds` side-cache → **abort at graph-reserve** (`ggml-backend.cpp:919`,
`cannot run the operation`). (Sibling of bug-737, which was the I64 `k_idxs` variant of
the same abort.)

**Fix — k_cur incremental-merge fill.** The fill now reads **`k_cur`** (the *new* token K
projection, always f16/f32 *before* cache quantization) instead of the quantized cache
view. The kernel is templated on K type (`half`/`float` via `sparse_ld<>`), derives
`n_new = k_cur->ne[2]` and the column stride `nb1 = k_cur->nb[2]`, maps cache positions
via `p0 = k_idxs[0]`, and for each block either **MERGEs** (min/max) into the prior
`kbounds` (the straddling last block at decode / unaligned prefill chunk) or
**OVERWRITEs** (a fully-new block). Because min/max is exact, this reproduces the old
dense recompute **byte-identically on f16**, and because the cache is quantized *downstream*
(in `cpy_k`), the Quest bound is the exact pre-quant envelope — so sparse now composes
with **every** cache KV type for free. `supports_op` + launcher relaxed to `f16||f32`;
the graph passes `k_fill = ggml_is_contiguous(k_cur) ? k_cur : ggml_cont(k_cur)`.

### Gate (Qwen2.5-14B-Instruct-1M-Q6_K, RTX 3090, RULER niah_single_1, 12 samples)

| ctx | KV | mode | coverage | niah | notes |
|----:|----|------|---------:|-----:|-------|
| 24k | f16 | dense  | 100% | 100.0 | baseline |
| 24k | f16 | sparse |  25% | 91.67 | lossy regime (below f16 lossless threshold) |
| 24k | f16 | sparse |  65% | **100.0** | f16 lossless reference |
| 24k | q8_0 | dense | 100% | 100.0 | q8 alone is clean |
| 24k | q8_0 | sparse | 25% | 75.0  | lossy regime |
| 24k | q8_0 | sparse | 65% | **100.0** | **composition lossless** |
| 24k | q8_0 | sparse | 80% | 100.0 | composition lossless |
| 48k | q8_0 | dense | 100% | 100.0 | |
| 48k | q8_0 | sparse | 25% | 83.33 | lossy regime (improves vs 24k: more absolute blocks) |

**Composition is lossless.** q8_0⊕sparse holds niah=100 at the same coverage (≥65%) where
f16⊕sparse is lossless. The 25%-coverage gap (q8 75 vs f16 91.67) is *not* a composition
defect — the block *selection* is identical (both fill `kbounds` from the same f16 `k_cur`),
so the gap is q8 quantization noise on the attended needle block plus 12-sample variance in
the sub-threshold lossy regime. At shippable coverage the penalty vanishes.

### The decode win lives on quantized KV

Steady-state decode (max over niah gen samples), niah=100 configs:

| KV | dense | sparse@65% | ratio |
|----|------:|-----------:|------:|
| f16 | 27.4 tps | 20.2 tps | 0.74× (loss) |
| q8_0 | 15.2 tps | ~17.4 tps | ~1.15× (win) — cross-gate hint |

**Same-boot A/B confirmation (q8_0, Qwen2.5-14B-1M-Q6_K, 24k, `sparse-q8-win-confirm.sh`).**
To remove cross-run variance the dense baseline and every sparse coverage were run
back-to-back in one harness boot. The win is real and grows monotonically as coverage drops,
while niah stays 100 all the way down to 50% coverage:

| coverage | niah | steady tps | vs dense |
|---------:|-----:|-----------:|---------:|
| 100% (dense) | 100 | 13.71 | 1.00× |
| 80% | 100 | 15.15 | 1.10× |
| 65% | 100 | 17.63 | 1.29× |
| 50% | 100 | 18.33 | **1.34×** |

Best lossless operating point in this sweep: **50% coverage → 1.34× decode at full niah.**
(25% coverage is lossy — q8 75 — so the lossless band bottoms out between 25% and 50%.)
The same-boot dense baseline (13.71) ran ~10% below the cross-gate extraction (15.2),
which is exactly the run-to-run variance the same-boot A/B exists to eliminate — the
sparse/dense *ratio* is the trustworthy number, not the absolute tps.

On **f16**, sparse@lossless coverage is *slower* than dense: a 14B model at 24k is not
KV-bandwidth-bound enough for a 35%-block skip to overcome the per-stream selector cost.
On **q8_0** — the actual long-context serving config — sparse is a **lossless decode win**,
because the dense path must **dequantize every block** (`q8→f16` lift) before FA, and the
block-skip skips that dequant work too. That extra saved compute is what tips q8
positive where f16 stays negative. So bug-747 didn't just make q8⊕sparse *work* — quantized
KV is *where the win is*, even on a model that loses on f16.

### Stacking with sparse-V (#546) — orthogonal on speed, tau must be retuned per config (2026-06-27)

sparse-attn (#551, block-skip) and sparse-V (#546, attention-gated V-skip, `TURBO_SPARSE_V_TAU`)
both live in `fattn-vec.cuh`: the block-selector chooses which blocks to visit; the V-skip drops
negligible-weight positions *inside* visited blocks. Same-boot 4-cell A/B, q8_0 KV,
Qwen2.5-14B-1M-Q6_K, 24k, block coverage 50%, sparse-V tau 0.1 (`sparse-stack-ab.sh`):

| cell | niah | tps | vs dense |
|------|-----:|----:|---------:|
| dense | 100 | 17.91 | 1.00× |
| block-skip @50% | **100** | 19.87 | 1.11× (lossless) |
| sparse-V τ=0.1 | **0** | 20.11 | 1.12× (lossy) |
| both | **0** | 23.49 | **1.31×** |

Two findings: (1) the levers **compound on throughput** — `both` (1.31×) is faster than either alone,
confirming they cut on orthogonal axes (which blocks vs which positions-within-block). (2) sparse-V's
**τ=0.1 does not transfer**: it was tuned lossless on turbo/turbo_tcq KV on 27B and Gemma-A4B (#546,
niah 100), but on q8_0 KV / Qwen2.5-14B-1M it destroys the needle (niah 0) — τ=0.1 is far too aggressive
for this model's q8 attention distribution. The lossless compound win is recoverable via a per-config
τ-sweep (0.1 → 0.01 → 0.001) to find where V-skip rejoins niah=100; block-skip alone remains the
proven lossless lever here (1.11× @ 50%).

## Sparse-V auto-policy (#565, patch 0088) — distinct from the block-selector above

NB: this is the **sparse-V** decode lever (#546, quantized-V attention-gated skip in the FA-VEC
kernel), NOT the Quest/vslash block-selector that is the subject of the rest of this doc. They are
independent. Sparse-V is shipped + validated; this section documents only its **auto-policy**.

The win is **arch-dependent**: lossless on interleaved-SWA (Gemma-4 — small concentrated global
V-cache), no-win on full-attention (Qwen — diffuse weights near the mean). Rather than make every
standalone `llamafile --server` user discover `TURBO_SPARSE_V_TAU`, the **binary self-configures**:

- **Where:** `src/llama-context.cpp`, `llama_context` ctor (`// opencoti-hook: sparse-v auto-policy (#565)`).
- **Rule:** if `getenv("TURBO_SPARSE_V_TAU")==NULL` AND `hparams.swa_type != LLAMA_SWA_TYPE_NONE`
  (iSWA) AND `params.type_v` is quantized (not f16/bf16/f32) → `setenv("TURBO_SPARSE_V_TAU","0.05",0)`.
- **Why host-side env (not a DSO setter):** ggml-cuda is a runtime-loaded DSO; the kernel already
  reads the threshold from `TURBO_SPARSE_V_TAU`. Pre-populating that env is host-only (no DSO ABI
  change, DSO byte-identical), and is the only channel that also reaches a standalone CLI run.
- **Override:** `overwrite=0` → an explicit `TURBO_SPARSE_V_TAU` (or `_EPS`/`_SINK`/`_RECENT`) wins.
- **Safety:** full-attn models leave the env unset → kernel τ=0 → dense, byte-identical. f16/bf16 V is
  inert via the kernel's own `type_V != F16/BF16` guard even if the env were set.
- **Visibility:** announced once at `LLAMA_LOG_WARN` (llamafile suppresses the model-load INFO block;
  a silent INFO-only auto-change is invisible — bug-2090).
- **Sampling:** non-gating. Sparse-V is a KV-bandwidth decode lever; correctness is proven at greedy
  (temp 0, the strictest case), so the policy is sampling-agnostic.

Verified (3090): decision-table 4/4 + A/B @ 24k lossless (niah 100==100) and non-slower; the
measurable decode win lands at 64k+ (production 50-sample: A4B/31B niah 100, +2.7/3.8% @ 64k →
+1.3/3.1% @ 256k). See docs/evaluations/context.md (2026-06-29).

### Adaptive τ (the `eps` mode) — built and characterized, NOT the shipped default (#565 verdict)

#565's literal goal was a *runtime weight-distribution-driven* threshold (not a fixed peak floor). It
**exists in the binary** as the `eps` mode: `fattn-vec.cuh:515/566` computes the per-query threshold
`thr = eps · d_inv_nproc · KQ_sum[jj]` — recomputed from the running softmax denominator so it adapts
to each token's actual attention mass (skip iff the dropped *normalized mass* < `eps`). Selectable via
`TURBO_SPARSE_V_EPS`; `eps=0` ⇒ falls back to the static `d_sparse_v_tau`.

**It is NOT what the auto-policy regulates** — the policy above only sets the *static* `TURBO_SPARSE_V_TAU`;
it never sets `TURBO_SPARSE_V_EPS`. That is deliberate, not a gap: the adaptive `eps` mode was measured
**uniformly lossless but ~zero speedup** (Qwen-14B-1M full-attn: `eps` 0.005–0.10 all niah=100, tps flat
7.8–7.9 vs dense 7.83). The reason is structural — a mass-relative threshold can only harvest the
*sub-mean* tail, and on diffuse full-attention weights cluster near the mean, so almost nothing is
skippable regardless of `eps`; on iSWA the static τ=0.05 already captures the concentrated-global-cache
win more aggressively. So adaptive τ ships as a **safe opt-in knob** (never slower/lossy, `eps=0`
byte-identical) while the static τ via auto-policy is the production win. **Verdict:** the question "does a
runtime-adaptive threshold beat static τ on the architectures we have?" is answered — *no*. Reopen only for
a fundamentally different full-attention adaptive scheme (which is the content-selection problem #551, itself
a closed dead-end). #565 closed as **built + characterized**, 2026-06-30.