docs/evaluations/decode-levers.md

# Decode-throughput lever map — turbo / turbo_tcq / scalar (A4B, RTX 3090)

**Source:** nsys per-kernel decode profiles (`.opencoti/nsys/lever_{scalar,turbo3,tcq}.nsys-rep`),
A4B-98e-v6-coder Q4_K_M, ctx 4096, n_predict 96, DSO `60b17cb1` (post-#535 vectorized K-read).
Cross-config absolute totals are NOT comparable (different #decode-steps captured); the comparable
views are **per-phase % of that config's own total** and **per-call avg µs**.

> **SCOPE (WS2 / #629).** Every "turbo/TCQ costs decode vs scalar" number in this document is
> **noDCA, short-ctx (ctx 4096), f16-lift** — the regime where the centroid-unpack / codebook lookup
> is pure overhead. **With DCA-on at long ctx the picture inverts:** the WS2 in-register turbo-V kernel
> (DEFAULT-ON) makes turbo/TCQ-V a decode **WIN of +68–95% @256k** over the f16-lift ref, because it
> kills the context-growing whole-cache `cpy_turboN_f16` materialize that dominates the long-ctx step.
> The gaps below are the short-ctx floor, not the serving verdict — see
> [context.md §5x](context.md).

## Measured baseline (decode tps, A4B)
scalar q8_0 **68.7** · turbo3 **58.2** · turbo3_tcq **55.6**. Gap turbo3→scalar = −15% — noDCA /
short-ctx only; DCA-on long-ctx inverts (§5x, WS2 in-register turbo-V is +68–95%).

## Per-call FA-VEC K-read (config-independent)
| D | scalar q8_0 | turbo3 | turbo3_tcq |
|---|---|---|---|
| 256 (local layers) | 21.1 µs | 63.0 µs (3.0×) | 49.1 µs (2.3×) |
| 512 (global layers) | — | 270.4 µs | 210.6 µs |

`mul_mat_vec_q` experts = 17.8 µs/call for ALL THREE (KV-type-independent — confirms FFN is shared).

## Phase fractions (% of each config's decode kernel time)
| Phase | scalar | turbo3 | tcq |
|---|---|---|---|
| FFN/MoE experts (mul_mat_vec_q) | 54.3 | 44.4 | 41.6 |
| NORM/softmax/act/elementwise | 15.9 | 13.0 | 12.2 |
| Turbo/TCQ K-read (FA-VEC) | (4.5 scalar) | 20.5 | 15.0 |
| TCQ KV-write (Viterbi enc/set_rows) | 2.5 | 2.3 | **13.6** |
| KVwrite cpy/quantize | 4.6 | 3.7 | 3.5 |
| WHT rotate (turbo) | 2.0 | 2.9 | 2.7 |
| embed/get_rows/argsort | 3.7 | 3.0 | 2.8 |
| ROPE | 1.1 | 0.9 | 0.9 |
| stream-k fixup/combine | 0.9 | 0.6 | 0.5 |
| FFN other matmul | 3.2 | 3.9 | 3.0 |
| OTHER (launch slack etc.) | 7.2 | 4.8 | 4.2 |

## LEVERS by phase — ranked by (impact × tractability)

### L1 — SPECULATIVE DECODE (cross-cutting MULTIPLIER) ⭐ biggest
MTP draft head already ships 1.46× on f16. If turbo/tcq COMPOSE with MTP, that ~1.4× multiplies the
WHOLE decode — bigger than any single-kernel tweak. STATUS: unverified for turbo/tcq KV. Effort: gate-only
(no kernel) if it already composes; medium if KV-write-on-accept needs work. **Verify first.**

### L2 — FFN/MoE experts (mul_mat_vec_q) — 42–54%, SHARED, biggest absolute
17.8 µs/call, dominates ALL configs. Improving it raises turbo AND scalar (doesn't close the gap, but the
user's goal is "improve turbo" in absolute terms). Levers: (a) fused up+gate op (GGML_OP_MOE_FUSED_UP_GATE
exists — verify it's ON the A4B decode path, not just prefill); (b) mmvq nrows batching across the 2 active
experts; (c) ik_llama fused-MoE (#292 OOAE) decode path. Effort: medium-high. Risk: numerics (cosine gate).

**(a) MEASURED 2026-06-23 — INAPPLICABLE TO A4B (op never engages); do NOT default-flip.** `--fused-moe-up-gate`
is a structural **no-op on Gemma A4B**: the boot `-v` log shows its experts load as a single combined
**`blk.N.ffn_gate_up_exps.weight`** tensor (up and gate projections already share one matmul). The graph hook
fires the fused op only when `fused_moe_up_gate && gate_exps && !gate_up_exps && n_tokens==1 && (GELU|SILU)`;
A4B *has* `gate_up_exps`, so `!gate_up_exps` is false → the op is never inserted, flipping the flag leaves the
graph identical (bit-identical output). **2×2 proof {graphs on/off}×{fmug on/off}, A4B Q4_K_M decode tok/s:**

| | fmug=OFF | fmug=ON | fused-gain |
|---|---|---|---|
| graphs ON | 92.70 | 92.45 | −0.3% |
| graphs OFF | 78.25 | 78.33 | +0.1% |

Fused-gain ≈ 0 in **both** rows ⇒ this is NOT subsumption-by-graphs (that would leave the graphs-OFF gain
visible); it's **the op never firing** because the experts are pre-fused. (Graphs reconfirmed orthogonally:
+18.5% at fmug-off.) So L2(a) is only meaningful for GGUFs with **separate** `ffn_gate_exps`+`ffn_up_exps`
experts — check the target model's tensor names first. For pre-fused-expert models like A4B, the up/gate fusion
is already baked into the weights. What remains in L2 is the *matmul itself* (b/c) — a genuine compute-bound
win, but medium-high effort with numeric risk; deferred unless the FFN matmul becomes the proven bottleneck
post-L1/L6. NOTE: on a model with separate experts AND graphs on, expect fused-MoE to be subsumed (graphs
already kill its launch saving) — the residual would only be the saved intermediate-buffer traffic.

**(a) CONTRAST — Qwen DOES have separate experts, op fires.** Qwen3.6-35B-A3B (qwen35moe, 256 experts) loads
**separate** `ffn_gate_exps`+`ffn_up_exps` (vs A4B's combined `ffn_gate_up_exps`), so the fused op engages.
2×2 {graphs}×{fmug}, f16 decode tok/s:

| | fmug=OFF | fmug=ON | fused-gain |
|---|---|---|---|
| graphs ON | 89.17 | 91.31 | **+2.4%** |
| graphs OFF | 76.97 | 76.26 | −0.9% (noise) |

So fused-MoE is **model-dependent**: inert on A4B (pre-fused weights), a small real win on Qwen-style separate-
expert MoEs — and, interestingly, the gain shows up *with graphs ON* (+2.4%), not off (the fewer graph nodes
help replay; off-graph it's neutral/noise). **CLOSED 2026-06-24 (greedy-identity gate):** on Qwen3.6-35B-A3B
(`qwen35moe`, 256 experts, separate `ffn_gate/up_exps`) fused-MoE off-vs-on is **greedy BIT-IDENTICAL** at
temp=0 for BOTH f16 and turbo3_tcq KV (764B==764B, 661B==661B) — numerically safe. But under the SHIPPING
config (graphs ON) the re-measured gain is only **+0.7–0.9%**, not +2.4%: the +2.4% was a graphs-OFF baseline
artifact (graphs already amortize the launch overhead fused-MoE targets, leaving a sub-1% residual once graphs
are on). **Verdict: safe but sub-1% under the shipping config — NOT worth a default-flip + the validated-model-
surface expansion; the flag stays opt-in (default OFF).** fused-MoE is a minor, situational lever, NOT a
headline like L6/L1.

**(b) "mmvq nrows-batch across active experts" — ALREADY IMPLEMENTED (nothing to build), 2026-06-23.**
`MMVQ_MAX_BATCH_SIZE = 8` (`mmvq.cuh:3`) and both targets use `n_expert_used = 8`, so `ne2=8 ≤ 8` takes the
batched MUL_MAT_ID path at `ggml-cuda.cu:2677-2681` → `ggml_cuda_mul_mat_vec_q(src0, src1, ids, dst)`: one
launch with all 8 active experts mapped onto the grid's `nchannels_dst` axis (`mmvq.cu:673`, grid
`(nblocks, nchannels_dst, ntokens)`). There is no per-expert-launch loop to collapse. (Only `>8`-expert-used
models fall to the slower MMQ path — not our targets.) Reduces no bytes-read; moot.

**(c) "ik_llama OOAE + fused-MoE" — fused-MoE shipped (0042), OOAE irrelevant to the matmul, 2026-06-23.**
The fused up+gate op is already the `0042` path (`ggml-cuda.cu:3030`). ik_llama's other piece is `topk-moe`
(fused softmax+top-k expert *selection*); our tree uses standard `argsort_top_k`. topk-moe speeds *selection*,
not the expert matmul, and reads the same expert weights — orthogonal to the 42–54% FFN cost. (#292 was logged
"OOAE + fused-MoE pair" but only the fused-MoE half landed; the topk-moe half is absent and would not move
decode tps.)

**ROOT REASON b AND c ARE DEAD ENDS (verified):** the expert matmul is a GEMV over quantized weights —
arithmetic intensity ≈3.5 FLOP/byte ≪ the RTX 3090 roofline ridge (~37) ⇒ hard **bandwidth-bound** on
*reading the expert weights*. Neither b nor c reduces those bytes. The only levers that attack a bandwidth-
bound weight read: **L1/MTP** (amortizes each weight read across K draft tokens per verify step — the proven
+55%) or **lower-bit experts** (quant choice). A faster GEMV *kernel* would help only if mmvq leaves bandwidth
on the table — a separate #397-class effort to be justified by an `ncu` DRAM-throughput measurement first, NOT
b/c. **Conclusion: L2(b)/(c) require no work; closed.**

**#397 GEMV-kernel verdict — MEASURED (ncu, 2026-06-23, A4B Q4_K_M decode):** the time-dominant expert
`mul_mat_vec_q<12,…>` launches run at **77–81 % of the 3090's peak DRAM bandwidth** (time-weighted 61.6 %
across all decode mmvq launches; SM ~41 %). The heavy kernels are already near the memory-bound practical
ceiling (~85–90 %), so a faster GEMV buys at most ~1.1–1.25× on the dominant slice and less overall.
**#397 (int8-dp4a turbo KQ / faster-GEMV) is NOT worth pursuing for decode tps** — the real levers remain
L1/MTP and lower-bit experts, which cut *bytes read*, not read *speed*. Method (reusable): the cosmopolitan
APE binary can't be `execve`'d by ncu/nsys directly ("Unknown executable format") — copy it and
`./copy --assimilate` to a native ELF (cosmo runtime intercepts the flag pre-main), then
`ncu --kernel-name regex:mul_mat_vec --launch-skip N --launch-count M --metrics
gpu__dram_throughput.avg.pct_of_peak_sustained_elapsed,…` (set `GGML_CUDA_DISABLE_GRAPHS=1` so each launch is
individually filterable; `mul_mat_vec_q` is decode-only — prefill uses the batched `mul_mat` path).

### L3 — TCQ decode-time Viterbi encode/set_rows — 13.6% of tcq, TCQ-SPECIFIC ⭐ — (a) DONE (#541); (b) REJECTED (#545); (c) UNATTEMPTED
The trellis encode runs on EVERY decode KV-write. 13.6% of tcq decode = the single biggest tcq-vs-turbo3
penalty. Levers: (a) **codebook-hoist in the ENCODE kernel — DONE (#541, DSO b4ccda34)**: hoisted
`d_turbo3_tcq_codebook[16*lane+i]` out of the 128-step Viterbi DP into a per-lane `cb[16]` register array
(turbo2: `cb[8]`). Byte-identical CSE (encode REG:60/STACK:0, occupancy 3/SM held). Result **turbo3_tcq decode
59.7 → 60.4 (+1.2%)**; scalar 87.6 / turbo3 76.2 unchanged. 27B KLD healthy on both edited kernels
(turbo3_tcq 0.034606 sametop 95.5%, turbo2_tcq 0.069940 PASS sametop 92.6%). VERDICT: the codebook reads were
NOT the encode bottleneck — the 128-step serial DP is — so the hoist only buys ~1%. The big tcq win is (b).
(b) cheaper 1-token decode-time encode (the Viterbi DP is NOT amortizable to O(8): the 128-step trellis spans
the 128 HEAD-DIM elements of the single new token's vector, all new each step; the candidate lever was a
greedy walk over the 128 element-steps, O(128·8) vs O(128·512)). **REJECTED 2026-06-24 (#545, P1 spike,
flag-gated greedy encode on DSO 9403e25f, then reverted to L3 byte-exact).** Same-DSO 27B fidelity-KLD:
**greedy-tcq 1.0575 (sametop 61.6%)** vs Viterbi-tcq 0.0346 (95.5%) vs plain turbo3 0.0470 (94.7%). Greedy
does not merely regress toward turbo3 — it **collapses ~30× past Viterbi and ~23× past plain turbo3**: the
512-entry codebook is jointly trained for the Viterbi-optimal paths, so a greedy walk lands on mismatched
centroids and each locally-best symbol paints the next step into a corner. **The trellis's MSE-optimal
noise-shaping is irreducible** — the ~8% decode-time encode cost is the *price* of tcq's quality edge, not a
removable inefficiency. Since plain turbo3 is both faster AND (at 0.047) far better than greedy-tcq (1.057),
the answer where greedy was the goal is simply "ship turbo3". (c) skip re-encode via an f16 ring + lazy
batched TCQ-encode — UNATTEMPTED; the only remaining idea, but it changes write semantics (a token's KV is
written f16 then re-encoded in a later batch), so it is a structural rework gated on the same KLD bar, not a
quick win. Effort: high. Risk: quality (KLD gate) + correctness (deferred-encode state IO).

**L3 round verdict (closes the decode-tps investigation):** the only banked win is L3(a) +1.2% on tcq. The
floor is real and now evidence-backed on both axes — turbo3 vs scalar q8_0 ≈ −10% (centroid unpack is a
Shift→Mask→LUT→Scale dependency chain vs `int8*scale`), turbo3_tcq vs turbo3 ≈ −8% to −13% (512-entry LUT +
9-bit unaligned extract). No free lever remains; further tcq speedup requires either format change (smaller
LUT → retrain, P2, quality-gated) or the (c) deferred-encode rework. **— noDCA / short-ctx only; with DCA-on
at long ctx the WS2 in-register turbo-V path is a decode WIN (+68–95%), see [context.md §5x](context.md).**

### L4 — NORM/softmax/elementwise — 12–16%, SHARED, many small kernels
rms_norm + soft_max + scale/add/residual = lots of tiny launch-bound kernels. Lever: fuse (norm+scale,
residual-add into the matmul epilogue, RoPE+norm). Also a CUDA-graph candidate. Effort: medium. Risk: low
(byte-identical-ish).

### L5 — Turbo K-read FA-VEC — 20.5% turbo3 / 15% tcq, TURBO-SPECIFIC — ✅ DONE (#535b, 2026-06-23)
Vectorized loads (#535) took D256 70→63 µs (+1.6%). Then **#535b smem-codebook port** (DSO 47b0bca5): the
plain turbo K-read/V-dequant indexed FATTN_TURBO{2,3,4}_CENTROIDS in `__constant__`, which serializes the
data-dependent per-element lookups up to 32-way/warp; the TCQ tiers already staged to per-block smem (#534).
Porting that staging to turbo2/3/4 is BYTE-IDENTICAL and gave:
- **D256 turbo3 K-read 63.0 → 48.4 µs (−23%)** — now *beats* turbo3_tcq's 49.1 (8-entry LUT < 512-entry codebook).
- **D512 turbo3 K-read 270.4 → 135.2 µs (−50%)**.
- **decode tps: turbo3 58.2 → 62.08 (+6.7%)**, turbo4 → 63.5, turbo2 → 66.5; scalar 68.96 unchanged.
- KLD turbo3‖f16 = 0.349646 == the #535 value (byte-identical); turbo4 0.109 sane.
- turbo2 quality stays bug-583 K-side degenerate (in-register 2-bit; pre-existing, reproduced byte-identically:
  turbo2K/f16V=7.39, f16K/turbo2V=0.043) — use turbo2 via DCA dequant-on-lift, never this in-register reader.
LESSON → cerebrum 2026-06-23: small data-dependent LUTs in hot decode loops belong in smem, not __constant__.

### L6 — CUDA graphs — ✅ DONE (#533/#539, 2026-06-23, ships as 0084-cuda-graphs)
Decode launches ~hundreds of tiny kernels/token; at 58 tps each token is ~17 ms but GPU SM throughput is
~4.5% (latency/launch-bound). **Root cause found: CUDA graph capture/replay was COMPILED OUT of the
llamafile build** — `USE_CUDA_GRAPH` is gated `#if defined(GGML_CUDA_USE_GRAPHS)` (common.cuh:1218) and the
llamafile `cuda.sh` never defined it (upstream CMake does, by default). So `#ifdef USE_CUDA_GRAPH`
(ggml-cuda.cu:3318/4478/4515/4593/4698) was absent and every token paid full per-launch CPU overhead.
**Fix = one build flag** `-DGGML_CUDA_USE_GRAPHS` (0084; `OPENCOTI_NO_CUDA_GRAPHS=1` to omit). MoE does NOT
disable it (the MUL_MAT_ID quantized-expert path stays graph-compatible; NEO head-split force-disables to
eager). **Measured (A4B Q4_K_M, airtight same-DSO `GGML_CUDA_DISABLE_GRAPHS` toggle):**
- **f16 decode 77.4 → 92.4 (+19.4%)**, **turbo3_tcq 53.6 → 60.6 (+13.1%)** — shared, lifts every KV tier.
- **Stacks with MTP** (L1): turbo3_tcq graphs+MTP ≈ **89 tok/s** (+69% over the 52.9 base; exceeds plain f16
  77.7, ~75% of f16-with-same-levers 119.7).
- **Numerically inert** — greedy bit-identical graphs-on vs -off across f16/turbo3/turbo3_tcq/q8_0/q4_0 + 8K
  ctx (graphs replay the identical kernel sequence; not a compute change).
- **GENERALIZES to Qwen (model-agnostic).** Qwen3.6-35B-A3B (qwen35moe, 256 experts) decode graphs on vs off:
  **f16 76.97→89.17 (+15.9%)**, **turbo3_tcq 68.52→78.49 (+14.6%)** — same magnitude as Gemma. The lever is
  not arch-specific; it lifts every model's latency-bound decode. (turbo3_tcq also confirmed working on Qwen
  head_dim 128.)
LESSON → cerebrum 2026-06-23: opencoti inherited llamafile's `cuda.sh`, which omits upstream-default build
defines — audit `cuda.sh` vs upstream CMake `GGML_CUDA_*` options for other free wins (graphs were one).

### L7 — WHT rotate (turbo) — 2–3%, turbo-specific, small
k_turbo_wht runs per KV-write. Minor. Fuse into set_rows. Low priority.

## Recommended sequence (phase-by-phase)
1. **L1 verify** turbo/tcq × MTP composition (cheap, possibly huge). 
2. **L3** TCQ decode-encode smem/cheap-encode (closes tcq's 13.6% — the clearest tcq win).
3. **L5b** V-read vectorization (cheap byte-identical, finishes the turbo K/V read).
4. **L2** FFN/MoE fused-up-gate verify + decode-path (biggest absolute, shared).
5. **L4 / L6** norm fusion + CUDA graphs (shared launch-bound reduction).