docs/features/sparse_attn.md

Sparse Attention — Quest-style block-selector for long-context decode (#551)

Status: BUILT + CHARACTERIZED on the 3090 (2026-06-27); 1M validation on bs2 pending. The real 1M decode lever — companion to the DCA + quant-KV prefill work (#444/#445/#554). Default-OFF, byte-identical when off. Develops on the 3090 (A4B / Qwen3-4B / 27B-Q4 @ ≤256k); 1M validation on bs2.

Headline result (§S4 results): the sparse/dense decode ratio is context-driven, not batch-driven. Single-stream (B1), Qwen3-4B, f16 KV, 25% block coverage on the 3090, the ratio climbs monotonically and the gap to dense closes 9.2% → 0.8% from 16K to the model's 40K ceiling (0.908 → 0.992). Crossover (>1.0, a real decode win) extrapolates just past 40K — the 256k/1M regime on bs2. Batch is not our lever (unlike TheTom — see §S4 results for why).

Why this, and why now

The 1M serving path has two halves. Prefill is handled by DCA's analytical band + quantized KV (q5_0/q4_0 — f16 KV at 1M is 206 GB, impossible), and #554 removed the whole-cache-lift prefill cliff. Decode is still O(n_kv) per token: at 1M every generated token attends to ~1M KV entries, memory-bound and slow. The only thing that breaks that is sparse attention — attend to a small, query-chosen subset of KV blocks instead of all of them.

Method: Quest-style min/max block bounds (decided 2026-06-25)

Reference scouted: TheTom/turboquant_plus docs/papers/block-selector-sparse-attention.md. Decisive finding for us: that work is Apple Silicon / MLX, where TheTom found custom sparse kernels with real K-skipping LOSE (14× at B=8) to the tuned sdpa_vector_2pass, so he uses a float mask that still streams the full K/V (saves compute, not bandwidth). He explicitly notes the literature wins (Quest, NSA, SeerAttention, DuoAttention) "all target CUDA, where FlashAttention's sparse kernel is mature." We are CUDA. Our FA-VEC already has the KV_max/KV_min tile-trim (fattn-vec.cuh:363) that lets us skip whole KV blocks before the QK dot — real K-read + QK + V savings — which TheTom's Metal stack could not. So we build the thing he couldn't, using the cleaner CUDA ancestor:

Quest (Tang 2024). Parameter-free, exact upper-bound:

• Per-(KV-head, block) min/max key bounds. Block size B_SEL (start 64; tune). Store kmin[head_dim], kmax[head_dim] per block — a tiny side-cache (n_blocks × head_dim × 2 × f16). For quant-KV, compute min/max from the f16 values before quantizing (composes with q5_0/q4_0).
• Per-query upper bound. For query q, the max possible q·k over a block is, channel-wise, Σ_d max(q_d·kmin_d, q_d·kmax_d) (use kmax where q_d>0, kmin where q_d<0). O(n_blocks·head_dim) — negligible vs O(n_kv·head_dim) dense attention.
• Top-K block selection. Keep the top-K_SEL blocks by upper bound PLUS the recent window + attention-sink block (streaming-LLM safety so recency/retrieval is never silently dropped). K_SEL and window are knobs; tune on RULER-niah.
• Kernel skip. Extend the FA-VEC tile loop to continue past unselected blocks — the real K+QK+V skip.

Budget policy: fixed top-K + recent/sink (decided). TheTom found adaptive top-K "never beat union top-K" and converged loosely at long ctx; fixed-K is predictable in VRAM/compute and easy to gate.

Compose (three sparsity levers, three pipeline stages)

1. Block-selector (this work) — skip K blocks before QK (saves K-read + QK + V).
2. DCA analytical band — the selector narrows within each INTRA/SUCC/INTER band.
3. sparse-V (#546) — within selected blocks, skip V dequant for negligible-weight positions (the residual, post-softmax). Plus quant-KV (min/max from dequant K at write). The three are independent gates at the block / band / position granularities.

Where it lives (insertion points, from the Explore map)

• Write path (S1): per-block min/max side-cache, filled as K is written (KV-cache write in llama-graph.cpp / the cpy path). Additive tensor; off ⇒ not allocated.
• Selector (S2): host/graph pre-pass computing the per-query block mask from the bounds; carried to the kernel by extending the existing KV_max/KV_min mechanism (fattn-common.cuh:27-28, fattn-vec.cuh:352-353) with a block-select bitmask/priority.
• Kernel (S3): block-skip continue in the FA-VEC tile loop at fattn-vec.cuh:363 (the long-ctx decode hot path is FA-VEC, not MMA).

S1 integration anchors (mapped 2026-06-25, read-only)

All file:line in vendors/sources/llamafile/llama.cpp/:

• K write (scatter): src/llama-graph.cpp:2771 — mctx_cur->cpy_k(ctx0, k_cur, k_idxs, il) (impl in llama-kv-cache.cpp). The per-block min/max must be derived from k_cur here (pre-quant) or right after, per write.
• KV tensor alloc (add the side-tensor here): src/llama-kv-cache.cpp:637 (GPU ggml_new_tensor_3d(ctx_s, type_k, n_embd_k_gpu, wc, 1)) + :686 (host); stored in the layers vector (llama_kv_cache_layer, pushed :738). Add a parallel per-(KV-head, block) kmin/kmax tensor alongside k_l.
• Host-fill model: set_input_dca src/llama-kv-cache.cpp:3899-4040 (dispatch :5250) — the idiom for filling a per-forward input tensor (direct tensor->data writes / ggml_backend_tensor_set); mirror for the selector's per-block mask in S2.
• Flag plumbing (mirror --dca for --sparse-attn*): common/arg.cpp:1516,1528 (add_opt) · src/llama-cparams.h:56-57 (cparams fields) · common/common.cpp:1649-1650 (params→cparams) · tools/server/server-context.cpp:1102 (server gate).
• Sizing accessors: llama-hparams.h:268 n_head_kv(il) · :283 n_embd_head_k(il) · cparams.n_ctx (graph ctor llama-graph.cpp:1009).
• S1 sequence: vendor-backup WHOLE tree FIRST → add --sparse-attn{,-topk,-block-size} flags (default off) → alloc the kmin/kmax side-tensor (only when enabled) → fill it at/after the cpy_k write → build host + verify default-off path is byte-identical (no new tensor allocated, no graph change). Selector + kernel skip are S2/S3.

Correctness gate (NEVER greedy needle — bug-263/270/354)

• RULER-niah @256k on the 3090 (A4B + Qwen3-4B): the needle's block MUST survive selection — niah recall is the primary signal that block-selection didn't drop the answer.
• Logit-equivalence vs dense (real_frac, top-k agreement) as the tolerance check.
• Perf: decode + prefill tps vs dense, sparsity-ratio sweep, at 32k/256k (3090) then 512k/1M (bs2).

Knobs (default OFF, byte-identical)

• --sparse-attn {off,on} (or auto by ctx threshold)
• --sparse-attn-block-size (B_SEL, default 64)
• --sparse-attn-topk (K_SEL blocks) and --sparse-attn-recent (always-keep window)
• env mirrors for dev (e.g. OPENCOTI_SPARSE_ATTN_*).

Staging

• S1 — per-block min/max side-cache + write-path plumbing (host). Off ⇒ byte-identical.
• S2 — Quest selector (query·bounds → top-K + recent/sink mask) + KV_max/KV_min carry.
• S3 — FA-VEC kernel block-skip in the tile loop.
• S4 — gate: RULER-niah @256k + logit-equiv vs dense + decode/prefill tps (3090).
• S5 — compose with sparse-V #546 and DCA bands; re-gate.
• S6 — capture patch (vendor-backup + snapshot-diff, byte-identical re-apply) + this doc's measured table + UPSTREAM_SYNC markers + STATE_SUMMARY + .wolf.

S2/S3 engineering design (mapped + decided 2026-06-25)

Integration reuses the proven KV_max/KV_min shape (pool-alloc + pre-pass fill + kernel param), but with one decisive difference: the per-step fill writes kmin/kmax and the selector reads them in the same graph, so ordering is enforced with real ggml src edges (two new graph ops) — not the DCA thread-local side-channel (opencoti_fattn_dca_pos_q, ggml-cuda/fattn.cu:27-29), which is safe only for analytical inputs with no shared-buffer write-then-read.

Three pieces:

1. Fill op sparse_attn_fill (new GGML custom op; pattern = the ggml_streaming_* / ggml_turbo_wht ops #345/#391).

   • Inserted in build_attn right after each cpy_k (llama-graph.cpp:2771; iSWA :2985, :3067), gated on cparams.sparse_attn_enabled.
   • srcs: k_cur (new tokens' K, pre-quant f16) + k_idxs (scatter positions). dst-update: kmin_l/kmax_l cache tensors (build_forward_expand, like cpy_k; the write into the persistent cache tensor + the selector's read edge needs the ggml_view_1d dep-tie idiom — bug-225 precedent — so the scheduler sees fill→select).
   • CUDA kernel: per (KV-head, channel), reduce min/max over each affected block's positions; min/max-merge with the existing kmin/kmax (blocks fill incrementally across decode). Affected blocks = [n_past/B, (n_past+n_new)/B].
   • Off ⇒ op not added ⇒ byte-identical.

2. Selector op sparse_attn_select (new GGML custom op; pre-pass shape mirrors flash_attn_dca_to_KV_max, fattn-common.cuh:891).

   • Inserted after get_k (:2813) before the FA op.
   • srcs: Q (rotated FA query) + kmin_l + kmax_l (the read edge orders it after the fill). dst: block_sel bitmask (n_blocks bits per (seq, KV-head), I32-packed).
   • CUDA kernel: per block b, UB = Σ_d max(q_d·kmin_b[d], q_d·kmax_b[d]); GQA → max UB over the query group per KV-head (one block set per KV-head, NSA co-load); top-K_SEL by UB (smem partial-sort / threshold) OR'd with the recent-window + sink blocks; write bits.

3. FA-VEC block-skip (S3).

   • block_sel rides as an extra src on the FA op — set result->src[5] = block_sel after ggml_flash_attn_ext (mirror of sinks=src[4]); the scheduler builds the selector→FA edge from the src slot regardless of the core op. launch_fattn passes block_sel->data as a new kernel param const int * __restrict__ block_sel alongside KV_max/KV_min (fattn-vec.cuh:83-84, fattn-common.cuh:27-28).
   • Tile loop (fattn-vec.cuh:363, immediately after the existing k_VKQ_min skip): if (block_sel && !(block_sel[base + ((k_VKQ_0/B)>>5)] & (1u<<((k_VKQ_0/B)&31)))) continue;. B (= B_SEL) aligned to the FA tile stride (nthreads) and the quant blck (32) ⇒ B_SEL a multiple of both for clean skipping.
   • Threaded through every FA-VEC instance signature (exactly as KV_max was). The MMA prefill path is a separate later stage (S1–S4 target FA-VEC decode).

Gate ladder (NEVER greedy needle — bug-263/270/354):

• Plumbing anchor: K_SEL = n_blocks (select-all) ⇒ sparse must equal dense ⇒ logit-equiv real_frac=0. Proves the carry + skip are exact before any real sparsity.
• Then reduce K_SEL: RULER-niah@256k (needle's block must survive) + logit-equiv tolerance + decode/prefill tps vs dense, sparsity-ratio sweep (3090 @ 32k/256k → bs2 1M).

Build reality: fill+selector+skip ⇒ CUDA DSO rebuild (~25 min/iter, restamp BOTH DSO paths). Min 2 builds (B1: fill — verify kmin/kmax vs a CPU min/max reference via a debug dump; B2: selector+skip — select-all anchor then sparsity sweep). Plan 2–4 iterations.

Open questions (resolve during impl)

• Block size vs the FA-VEC 128-thread tile stride and the quant block size (q5_0/q4_0 blck=32) — align B_SEL to a multiple of both for clean skipping.
• GQA: bounds are per-KV-head; the selector reduces the GQA query group to a per-KV-head representative (max over the group's per-block upper bounds) so all grouped Q-heads share one block set (NSA-style co-load) — avoids per-Q-head mask divergence.
• Interaction with the DCA fused MMA path (prefill) — S1–S4 target FA-VEC decode; the MMA/prefill block-selector (TheTom's 2.9×@128K prefill analogue) is a later stage.
• Where to compute the selector: a small dedicated CUDA op vs folding into the FA launch.

Comparison vs TheTom (S4 — user-required, "test the same as theTom to compare")

Reproduce TheTom's exact block-selector benchmark so our numbers sit directly beside theirs. Their config (from turboquant_plus/docs/papers/block-selector-sparse-attention.md, fetched 2026-06-25):

• Model: Qwen2.5-14B-Instruct-1M-4bit — we have this exact model on bs2 (the #550 DCA vehicle; native-1M, no-YaRN — see project_qwen25_1m_native_no_yarn.md). Same model ⇒ direct apples-to-apples.
• Grid: batch B ∈ {1,2,4,8} × ctx {16K … 131072} (their 19-cell grid). B>1 uses our PolyKV --parallel. Metric = dense-vs-sparse decode + prefill tps ratio.
• Block size: 64 — matches our locked B_SEL=64.
• Their headline (M5 Max / MLX): decode 1.19× (B4/16K) → 1.73× (B8/28K) → 1.48× (B4/48K); prefill 2.49× (64K) → 2.90× (128K). These are the numbers to match/beat.
• Their quality validation is INCOMPLETE — no clean end-to-end retrieval benchmark; only cosine-vs-dense (0.999 prefill / 0.99995 long-ctx adaptive-topK). We add a real RULER-niah@{256K,1M} retrieval gate they lack — a strict improvement on their eval.

Why ours should win the decode column: TheTom's selector is a JL-projection float-additive mask (contentDim=32, -∞ for non-selected) — compute-only masking, no bandwidth save — because MLX/Metal couldn't do a real K-skip (cerebrum 2026-06-21). Ours is Quest min/max block-skip on CUDA FA-VEC that continues past unselected blocks BEFORE the K-read + QK + V — a real bandwidth save — so our decode ratio should exceed their 1.19–1.73× at the same sparsity, not just match it. Prefill (MMA path) is a later stage (their 2.49–2.90× is the prefill-selector analogue, S5+).

CORRECTION (2026-06-27), see §S4 results: the "should exceed their 1.19–1.73×" claim was over-stated for the batched column. TheTom's wins are at batch 4–8; ours are at context length — different mechanisms (his fixed mask-cost is amortized across streams; our real K-skip is a fixed per-stream cost that batching can't amortize). So at his batch points we sit near break-even, not above. Where the prediction holds is the axis he does not measure — single-stream long-context — and the 3090 ladder (0.908→0.992, climbing) is the first hard evidence for it.

S4 deliverable: a side-by-side table — same model, same B×ctx grid, dense-vs-sparse decode/prefill ratio (ours, on the 3090 ≤128K then bs2 to 1M) next to TheTom's MLX column, PLUS our RULER-niah column. Run on bs2 (the 14B-1M lives there) to mirror their long-ctx grid.

S4 results — 3090 single-stream ctx-ladder + batch grid (2026-06-27)

The TheTom comparison ran in two parts on the 3090 (bs2/14B-1M busy; the same-model 1M grid is still pending). Model: Qwen3-4B-Q4_K_M, f16 KV, 25% block coverage, block size 64, Quest selector (--sparse-attn on). Metric = dense-vs-sparse decode throughput ratio. Decode tps read from the server log's authoritative eval time … / 128 tokens … tokens per second line — not the HTTP JSON, which is unreliable under the bug-743 parse-race (see Gotchas).

Single-stream ctx-ladder (B1) — the decisive axis

ctx (B1)	dense tps	sparse tps	ratio	gap to dense
16384	74.50	67.65	0.908	−9.2%
24576	61.54	59.37	0.965	−3.5%
32768	53.41	52.06	0.975	−2.5%
40000	47.46	47.06	0.992	−0.8%

Monotonic climb; the gap halves roughly every ~8–16K. At Qwen3-4B's native 40960 ceiling we are within measurement noise of dense, and the trend extrapolates to crossover (>1.0) just beyond — i.e. the 256k/1M serving regime. This is the expected signature of a real bandwidth-saving K-skip: saved K-reads grow with cache size while the selector's fill + QK-estimate + block_sel-gather cost is ~fixed, so the ratio rises with context. (Script: .opencoti/sparse-ctx-ladder.sh, gitignored.)

Batch grid (B1/B2/B4) — confirms batch is NOT our lever

ctx/seq	B	dense_agg	sparse_agg	ratio	note
16384	1	74.5	67.3	0.903	clean
16384	2	62.6	12.0	0.192	bug-743 parse-race (one req died, n<B)
16384	4	42.4	11.3	0.267	bug-743 parse-race (n=3)
24576	1	60.9	58.1	0.954	clean
24576	2	44.9	43.2	0.962	clean — sparse ~3% slower on both slots
24576	4	27.7	27.7	1.000	clean — uniform ~3% penalty washed out by scheduler asymmetry

Per-slot eval times on the clean 24K cells show sparse ~3% slower per stream (e.g. B2 dense 39.6/5.3 tps vs sparse 38.0/5.2). The 24K "1.000" is not a batched win — it is the uniform per-stream penalty hidden by one-fast-slot/three-starved scheduler asymmetry. So batching gives our selector no multiplier. (Script: .opencoti/sparse-batch-grid.sh.)

Verdict — we win a different game than TheTom

	TheTom (turboquant_plus)	ours (#551 Quest)
Selector	JL-projection float-additive mask (compute-only, no bandwidth save)	Quest min/max block-skip on CUDA FA-VEC (real K/V-read skip)
Winning axis	batch (1.19× B4/16K → 1.73× B8/28K → 1.48× B4/48K)	context (0.908@16K → 0.992@40K, B1)
B1 behaviour	structurally ≤1.0 (mask adds work, saves nothing at B1)	climbs to ~1.0 and past it with ctx
Quality eval	cosine-vs-dense only (0.999)	RULER-niah retrieval gate (a real gate he lacks)
Hardware/model	M5 Max / MLX / 14B-1M-4bit	3090 / CUDA / Qwen3-4B f16

His mask saves no bandwidth, so his win needs batch to amortize a fixed selector cost across streams; remove the batch and his advantage disappears. Ours saves real bandwidth per stream, so it scales with context and batching can't amortize a per-stream cost. Consequences:

• At B1 (single-stream serving) we should beat him — we climb to ~1.0 with ctx via real K-skip; his compute-only mask is structurally ≤1.0 at B1 (he doesn't report B1).
• At B8/28K (his sweet spot) he wins on his hardware and we get no batch multiplier — we sit near break-even there.
• On validation we are strictly ahead — RULER-niah vs cosine-0.999.

A true number-against-his-column comparison is still not apples-to-apples (different HW/model/quant). The only direct side-by-side is the still-pending S4 deliverable: the same 14B-1M-4bit, the same B×ctx grid, on bs2 to 1M, plus our RULER-niah column.

Gotchas surfaced

• bug-743 (parse-race): under load the server can feed a request's generated filler continuation back into the JSON parser → 500 "Failed to parse input at pos 0", which curl receives as the response (0 tps). It fires intermittently even at --parallel 1. Always read decode tps from the server log, not the HTTP JSON. A reliability bug to fix before batched sparse can be trusted in production.

DeepSeek Sparse Attention (DSA) — investigated, NOT portable (2026-06-27)

Investigated per user request (Reddit thread on a llama.cpp DSA effort). DSA (arXiv:2512.02556) is not transferable to our Qwen/Gemma targets:

• It needs a trained "lightning indexer" head (FP8, ReLU-gated, KL-distilled from dense attention) plus ~946B tokens of base-weight sparsity adaptation, and is MLA/MQA-coupled — our models are GQA with no indexer weights.
• The llama.cpp implementation (fairydreaming/sszymczyk, fork branch deepseek-dsa, merged upstream as PR #23346, 2026-05-29) is correctness-only — the author's own note says it "doesn't improve long-context performance yet" (oversized compute buffers).
• The one transferable idea — a small trained indexer head — is a training project, not a port. Out of scope for the vendored-fork decode lever.

All-KV-types composition (bug-747, 2026-06-27)

Sparse attention originally composed only with f16 KV: the B1 fill op (GGML_OP_SPARSE_ATTN_FILL) read the quantized cache K view, and its CUDA supports_op + launcher required src[1] to be GGML_TYPE_F16. With any quantized KV cache (q8_0/q4_0/q6_0/turbo*/tcq*) the cache K is non-f16 → supports_op=0 on CUDA0 → the scheduler could not place the op on the buffer holding the pre-allocated kbounds side-cache → abort at graph-reserve (ggml-backend.cpp:919, cannot run the operation). (Sibling of bug-737, which was the I64 k_idxs variant of the same abort.)

Fix — k_cur incremental-merge fill. The fill now reads k_cur (the new token K projection, always f16/f32 before cache quantization) instead of the quantized cache view. The kernel is templated on K type (half/float via sparse_ld<>), derives n_new = k_cur->ne[2] and the column stride nb1 = k_cur->nb[2], maps cache positions via p0 = k_idxs[0], and for each block either MERGEs (min/max) into the prior kbounds (the straddling last block at decode / unaligned prefill chunk) or OVERWRITEs (a fully-new block). Because min/max is exact, this reproduces the old dense recompute byte-identically on f16, and because the cache is quantized downstream (in cpy_k), the Quest bound is the exact pre-quant envelope — so sparse now composes with every cache KV type for free. supports_op + launcher relaxed to f16||f32; the graph passes k_fill = ggml_is_contiguous(k_cur) ? k_cur : ggml_cont(k_cur).

Gate (Qwen2.5-14B-Instruct-1M-Q6_K, RTX 3090, RULER niah_single_1, 12 samples)

ctx	KV	mode	coverage	niah	notes
24k	f16	dense	100%	100.0	baseline
24k	f16	sparse	25%	91.67	lossy regime (below f16 lossless threshold)
24k	f16	sparse	65%	100.0	f16 lossless reference
24k	q8_0	dense	100%	100.0	q8 alone is clean
24k	q8_0	sparse	25%	75.0	lossy regime
24k	q8_0	sparse	65%	100.0	composition lossless
24k	q8_0	sparse	80%	100.0	composition lossless
48k	q8_0	dense	100%	100.0
48k	q8_0	sparse	25%	83.33	lossy regime (improves vs 24k: more absolute blocks)

Composition is lossless. q8_0⊕sparse holds niah=100 at the same coverage (≥65%) where f16⊕sparse is lossless. The 25%-coverage gap (q8 75 vs f16 91.67) is not a composition defect — the block selection is identical (both fill kbounds from the same f16 k_cur), so the gap is q8 quantization noise on the attended needle block plus 12-sample variance in the sub-threshold lossy regime. At shippable coverage the penalty vanishes.

The decode win lives on quantized KV

Steady-state decode (max over niah gen samples), niah=100 configs:

KV	dense	sparse@65%	ratio
f16	27.4 tps	20.2 tps	0.74× (loss)
q8_0	15.2 tps	~17.4 tps	~1.15× (win) — cross-gate hint

Same-boot A/B confirmation (q8_0, Qwen2.5-14B-1M-Q6_K, 24k, sparse-q8-win-confirm.sh). To remove cross-run variance the dense baseline and every sparse coverage were run back-to-back in one harness boot. The win is real and grows monotonically as coverage drops, while niah stays 100 all the way down to 50% coverage:

coverage	niah	steady tps	vs dense
100% (dense)	100	13.71	1.00×
80%	100	15.15	1.10×
65%	100	17.63	1.29×
50%	100	18.33	1.34×

Best lossless operating point in this sweep: 50% coverage → 1.34× decode at full niah. (25% coverage is lossy — q8 75 — so the lossless band bottoms out between 25% and 50%.) The same-boot dense baseline (13.71) ran ~10% below the cross-gate extraction (15.2), which is exactly the run-to-run variance the same-boot A/B exists to eliminate — the sparse/dense ratio is the trustworthy number, not the absolute tps.

On f16, sparse@lossless coverage is slower than dense: a 14B model at 24k is not KV-bandwidth-bound enough for a 35%-block skip to overcome the per-stream selector cost. On q8_0 — the actual long-context serving config — sparse is a lossless decode win, because the dense path must dequantize every block (q8→f16 lift) before FA, and the block-skip skips that dequant work too. That extra saved compute is what tips q8 positive where f16 stays negative. So bug-747 didn't just make q8⊕sparse work — quantized KV is where the win is, even on a model that loses on f16.

Stacking with sparse-V (#546) — orthogonal on speed, tau must be retuned per config (2026-06-27)

sparse-attn (#551, block-skip) and sparse-V (#546, attention-gated V-skip, TURBO_SPARSE_V_TAU) both live in fattn-vec.cuh: the block-selector chooses which blocks to visit; the V-skip drops negligible-weight positions inside visited blocks. Same-boot 4-cell A/B, q8_0 KV, Qwen2.5-14B-1M-Q6_K, 24k, block coverage 50%, sparse-V tau 0.1 (sparse-stack-ab.sh):

cell	niah	tps	vs dense
dense	100	17.91	1.00×
block-skip @50%	100	19.87	1.11× (lossless)
sparse-V τ=0.1	0	20.11	1.12× (lossy)
both	0	23.49	1.31×

Two findings: (1) the levers compound on throughput — both (1.31×) is faster than either alone, confirming they cut on orthogonal axes (which blocks vs which positions-within-block). (2) sparse-V's τ=0.1 does not transfer: it was tuned lossless on turbo/turbo_tcq KV on 27B and Gemma-A4B (#546, niah 100), but on q8_0 KV / Qwen2.5-14B-1M it destroys the needle (niah 0) — τ=0.1 is far too aggressive for this model's q8 attention distribution. The lossless compound win is recoverable via a per-config τ-sweep (0.1 → 0.01 → 0.001) to find where V-skip rejoins niah=100; block-skip alone remains the proven lossless lever here (1.11× @ 50%).

Sparse-V auto-policy (#565, patch 0088) — distinct from the block-selector above

NB: this is the sparse-V decode lever (#546, quantized-V attention-gated skip in the FA-VEC kernel), NOT the Quest/vslash block-selector that is the subject of the rest of this doc. They are independent. Sparse-V is shipped + validated; this section documents only its auto-policy.

The win is arch-dependent: lossless on interleaved-SWA (Gemma-4 — small concentrated global V-cache), no-win on full-attention (Qwen — diffuse weights near the mean). Rather than make every standalone llamafile --server user discover TURBO_SPARSE_V_TAU, the binary self-configures:

• Where: src/llama-context.cpp, llama_context ctor (// opencoti-hook: sparse-v auto-policy (#565)).
• Rule: if getenv("TURBO_SPARSE_V_TAU")==NULL AND hparams.swa_type != LLAMA_SWA_TYPE_NONE (iSWA) AND params.type_v is quantized (not f16/bf16/f32) → setenv("TURBO_SPARSE_V_TAU","0.05",0).
• Why host-side env (not a DSO setter): ggml-cuda is a runtime-loaded DSO; the kernel already reads the threshold from TURBO_SPARSE_V_TAU. Pre-populating that env is host-only (no DSO ABI change, DSO byte-identical), and is the only channel that also reaches a standalone CLI run.
• Override: overwrite=0 → an explicit TURBO_SPARSE_V_TAU (or _EPS/_SINK/_RECENT) wins.
• Safety: full-attn models leave the env unset → kernel τ=0 → dense, byte-identical. f16/bf16 V is inert via the kernel's own type_V != F16/BF16 guard even if the env were set.
• Visibility: announced once at LLAMA_LOG_WARN (llamafile suppresses the model-load INFO block; a silent INFO-only auto-change is invisible — bug-2090).
• Sampling: non-gating. Sparse-V is a KV-bandwidth decode lever; correctness is proven at greedy (temp 0, the strictest case), so the policy is sampling-agnostic.

Verified (3090): decision-table 4/4 + A/B @ 24k lossless (niah 100==100) and non-slower; the measurable decode win lands at 64k+ (production 50-sample: A4B/31B niah 100, +2.7/3.8% @ 64k → +1.3/3.1% @ 256k). See docs/evaluations/context.md (2026-06-29).

Adaptive τ (the eps mode) — built and characterized, NOT the shipped default (#565 verdict)

#565's literal goal was a runtime weight-distribution-driven threshold (not a fixed peak floor). It exists in the binary as the eps mode: fattn-vec.cuh:515/566 computes the per-query threshold thr = eps · d_inv_nproc · KQ_sum[jj] — recomputed from the running softmax denominator so it adapts to each token's actual attention mass (skip iff the dropped normalized mass < eps). Selectable via TURBO_SPARSE_V_EPS; eps=0 ⇒ falls back to the static d_sparse_v_tau.

It is NOT what the auto-policy regulates — the policy above only sets the static TURBO_SPARSE_V_TAU; it never sets TURBO_SPARSE_V_EPS. That is deliberate, not a gap: the adaptive eps mode was measured uniformly lossless but ~zero speedup (Qwen-14B-1M full-attn: eps 0.005–0.10 all niah=100, tps flat 7.8–7.9 vs dense 7.83). The reason is structural — a mass-relative threshold can only harvest the sub-mean tail, and on diffuse full-attention weights cluster near the mean, so almost nothing is skippable regardless of eps; on iSWA the static τ=0.05 already captures the concentrated-global-cache win more aggressively. So adaptive τ ships as a safe opt-in knob (never slower/lossy, eps=0 byte-identical) while the static τ via auto-policy is the production win. Verdict: the question "does a runtime-adaptive threshold beat static τ on the architectures we have?" is answered — no. Reopen only for a fundamentally different full-attention adaptive scheme (which is the content-selection problem #551, itself a closed dead-end). #565 closed as built + characterized, 2026-06-30.