opencoti patch 0100 — DCA: defer quant lift in the single-chunk regime (bug-2119) dca.cpp gated defer_quant_lift on !single_chunk, so the single-chunk (INTRA) regime whole-cache dca_lift_to_f16'd the quantized K/V EVERY forward — a per-decode-step full-cache dequant that collapsed A4B 32k q8K/q4V decode to 4.69 tps. The comment assumed n_kv<=chunk meant "tiny cache"; at chunk 32768 it is the entire context. Stock ggml_flash_attn_ext converts scalar-quant natively, so the single-chunk branch defers exactly like analytical-bands. Gate (bs2, A4B q8K/q4V 32k chunk-32768): decode 4.69 -> 73.74 tps (DCA-off 77.13); INTRA text remains byte-identical to DCA-off. --- a/llama.cpp/src/dca.cpp +++ b/llama.cpp/src/dca.cpp @@ -255,16 +255,16 @@ // kernel dequants scalar-quant K/V to f16 IN ITS LAUNCH (single-pass to_fp16_nc, see // ggml_cuda_flash_attn_ext_mma_f16_dca_fused_case), so pass the quantized cache THROUGH instead of // whole-cache materializing it to f16 via dca_lift_to_f16 every forward (the prefill cliff). Gate: - // - !single_chunk: only the fused op carries the in-launch convert; single_chunk uses stock - // ggml_flash_attn_ext (launch_fattn already converts scalar-quant natively) and runs only while - // n_kv <= chunk (a tiny cache) so its lift cost is negligible — keep it on the safe lift path. - // - dca_is_inkernel_liftable: scalar quants the fused launch's to_fp16_nc can read. + // - dca_is_inkernel_liftable: scalar quants the fused launch's to_fp16_nc can read. bug-2119: the + // single_chunk branch defers too — it uses stock ggml_flash_attn_ext, whose launch_fattn + // converts scalar-quant natively (the ordinary DCA-off decode path). The old !single_chunk gate + // assumed n_kv <= chunk means "tiny cache, lift is negligible", but chunk can be 32k+: the + // whole-cache two-leg graph lift ran EVERY decode step (A4B 32k/chunk-32768 q8q4: 76 -> 4.7 tps). // - !v_trans: a transposed quant V view breaks the dim-0 (block) contiguity to_fp16_nc requires; the // permute(0,2,1,3) below keeps dim-0 intact, but a transpose would not — fall back to the lift. // The in-op convert is byte-identical to dca_lift_to_f16, so deferred output == lifted output. const bool v_trans_raw = v_raw->nb[1] > v_raw->nb[2]; const bool defer_quant_lift = - !inp_dca->single_chunk && dca_is_inkernel_liftable(k_raw->type) && dca_is_inkernel_liftable(v_raw->type) && !v_trans_raw;