opencoti patch 0098 — rolling-KV: dst_lse emission for head_dim<=256 decode (bug-1843, task #623) Extends the Stage-3a LSE channel (opencoti_fattn_dst_lse, drained by launch_fattn's flash_attn_combine_results / stream-k fixup) to head_dim<=256 n_q==1 decode at all three streaming-FA call sites (S3b overlap, S3a single-stream, S2 resident-view). Previously only head_dim>256 armed the channel; Qwen-class D128 decode fell through to the per-tile streaming_lse_kernel recompute — a <<>> full key-rescan per tile that was MEASURED as ~80% of spill-decode wall (bug-1843 A/B on the 3090: 1.14 tps base vs 6.05 tps OPENCOTI_LSE_NOOP, Qwen3-8B-Q8_0 ctx40960 vt14000). Mechanism: after the per-tile FA call, `lse_emitted = decode_lse && opencoti_fattn_dst_lse_written` — the written flag reports whether a finalize (stream-k fixup or parallel-blocks combine) actually ran; the recompute now fires only when the dispatch resolved to a no-finalize single-block path. The combine emits the identical per-row value (kqmax + logf(denominator), row order h + n_head*(qi + n_q*b)) the recompute produced. The `this_kv > 2*FATTN_KQ_STRIDE` guard is retained everywhere (bug-263 conservatism). Prefill and D>256 decode are behavior-identical. Gates (2026-07-05, 3090, DSO df46dc36): - perf: spill_base 1.14 -> 6.04 tps (5.3x, at the NOOP ceiling), needle PRESENT; resident 42.5 tps unchanged; OPENCOTI_LSE_NOOP now speed-inert (6.02) proving the recompute is structurally off the path. - correctness: teacher-forced logit-equiv (Qwen3-8B D128, ours-resident REF vs POSITION_WINDOW 256/16384 cells resident): real_frac=0.0, divergences=0, frac_full_agree=1.0, mean_tv 0.0068 -> PASS. DSO-only (fattn.cu; no header change, no host rebuild needed). --- a/llama.cpp/ggml/src/ggml-cuda/fattn.cu +++ b/llama.cpp/ggml/src/ggml-cuda/fattn.cu @@ -1596,11 +1596,20 @@ // ntiles_KV=16 at 512, not 2). `>` is kept only because it is no worse // than `>=`; the real mechanism is unisolated. Real Stage-3c tail tiles // can be ≤512 at decode → MUST resolve before relying on this path. - const bool decode_lse = (n_q == 1 && head_dim > 256 && this_kv > 2*FATTN_KQ_STRIDE); + // opencoti-hook: f5-rolling-kv bug-1843 — head_dim<=256 (Qwen-class VEC + // decode) now ALSO arms the channel: the parallel-blocks combine emits + // the per-row lse (Stage-3a formula in flash_attn_combine_results), and + // opencoti_fattn_dst_lse_written reports whether a finalize actually ran + // (T87.pD-opt O2 pattern) so the streaming_lse recompute below fires only + // when the dispatch resolved to a no-finalize single-block path. The + // recompute was measured at ~80% of spill-decode cost (1.14 vs 6.05 tps + // A/B via OPENCOTI_LSE_NOOP). + const bool decode_lse = (n_q == 1 && this_kv > 2*FATTN_KQ_STRIDE); if (decode_lse) opencoti_fattn_dst_lse = lse_t; ggml_cuda_flash_attn_ext(ctx, &dst_t); + const bool lse_emitted = decode_lse && opencoti_fattn_dst_lse_written; - if (!decode_lse) { + if (!lse_emitted) { // opencoti F5 bug-1840 fix (b) DIAGNOSTIC — OPENCOTI_LSE_NOOP skips // the per-tile streaming_lse recompute (a low-parallelism // <<>> rescan run once per tile because Qwen/head_dim≤256 @@ -1727,11 +1736,14 @@ // (softcap-aware). The `> 2*FATTN_KQ_STRIDE` boundary is NOT a fix for // bug-263 (UNRESOLVED: decode tile_kv_full==512 corrupts on either lse // path) — kept only as no-worse-than-`>=`. See .wolf/buglog.json. - const bool decode_lse = (n_q == 1 && head_dim > 256 && this_kv > 2*FATTN_KQ_STRIDE); + // bug-1843: head_dim<=256 also arms the channel; recompute only when no + // finalize ran (opencoti_fattn_dst_lse_written) — see the S3b note above. + const bool decode_lse = (n_q == 1 && this_kv > 2*FATTN_KQ_STRIDE); if (decode_lse) opencoti_fattn_dst_lse = lse_t; ggml_cuda_flash_attn_ext(ctx, &dst_t); + const bool lse_emitted = decode_lse && opencoti_fattn_dst_lse_written; - if (!decode_lse) { + if (!lse_emitted) { streaming_lse_kernel<<>>( (const char *)Q->data, sK, mask ? (const char *)mask->data + (size_t)kv0*mask->nb[0] : nullptr, @@ -1764,7 +1776,10 @@ // boundary is NOT a fix for bug-263 (UNRESOLVED: decode tile_kv_full==512 // corrupts on either lse path) — kept only as no-worse-than-`>=`. See the // S3b overlap path's full note + .wolf/buglog.json. - const bool decode_lse = (n_q == 1 && head_dim > 256 && this_kv > 2*FATTN_KQ_STRIDE); + // bug-1843: head_dim<=256 also arms the channel; recompute only when no + // finalize ran (opencoti_fattn_dst_lse_written) — see the S3b note above. + const bool decode_lse = (n_q == 1 && this_kv > 2*FATTN_KQ_STRIDE); + bool lse_emitted = false; if (this_kv > 0) { // FA over the tile's key-slice → O_t (stock kernel, untouched). ggml_tensor Kt = *K; Kt.ne[1] = this_kv; Kt.data = (char *)K->data + (size_t)kv0*K->nb[1]; @@ -1784,10 +1799,11 @@ dst_t.data = O_tiles.ptr + (size_t)t * nel; if (decode_lse) opencoti_fattn_dst_lse = lse_t; ggml_cuda_flash_attn_ext(ctx, &dst_t); + lse_emitted = decode_lse && opencoti_fattn_dst_lse_written; } // lse_t (this_kv==0 → empty tile: kernel writes -inf, combine ignores it). - if (!decode_lse) { + if (!lse_emitted) { streaming_lse_kernel<<>>( (const char *)Q->data, (const char *)K->data + (size_t)kv0*K->nb[1],