opencoti patch 0104 — q6_0-K / q4_0-V FA-VEC instance (bug-2125 / #627) q6_0-K + q4_0-V (strong-K, cheap-V — the design invariant K precision >= V) is a first-class asymmetric KV combo. Under GGML_CUDA_FA_ALL_QUANTS the kernel selector admits K!=V and picks the VEC path, but no (q6_0, q4_0) FA-VEC instance existed, so the D<=256 dispatch (Gemma-4 local/SWA layers — the global layers use the DCA fused kernel's in-launch to_fp16_nc) fell through to GGML_ABORT("fatal error") at ggml_cuda_flash_attn_ext_vec, crashing at boot. Fix (mirrors the q6_0-q5_0 pattern, #512/#81): add the instance TU fattn-vec-instance-q6_0-q4_0.cu (head_dim 64/128/256), the FATTN_VEC_CASES_ALL_D(Q6_0, Q4_0) dispatch case, and the build-functions.sh 3c default-build inclusion (so the symbol is present without FA_ALL_QUANTS too). Verified bs2 256k DCA-on: boots HEALTHY (was abort) + niah_single_1 = 100.0. CUDA DSO change (needs a cuda rebuild). Only the q6_0/q4_0 direction is added — reverse q4_0/q6_0 is intentionally NOT added (K must stay the higher-bit type). --- a/llama.cpp/ggml/src/ggml-cuda/fattn.cu +++ b/llama.cpp/ggml/src/ggml-cuda/fattn.cu @@ -353,13 +353,17 @@ // ik_llama.cpp ships for q6_0 KV), registered OUTSIDE the FA_ALL_QUANTS block so they compile in // the DEFAULT build — the matching instance .cu files are added unconditionally in // collect_gpu_sources (build-functions.sh section 3c), like the turbo K==V instances below. - // q6_0-q6_0 is the symmetric (default-reachable) cell; the 4 mixed cells are reachable only under + // q6_0-q6_0 is the symmetric (default-reachable) cell; the 5 mixed cells are reachable only under // FA_ALL_QUANTS (the K!=V early-out in ggml_cuda_get_best_fattn_kernel), same as q5_0's mixed pairs. FATTN_VEC_CASES_ALL_D(GGML_TYPE_Q6_0, GGML_TYPE_Q6_0) FATTN_VEC_CASES_ALL_D(GGML_TYPE_Q8_0, GGML_TYPE_Q6_0) FATTN_VEC_CASES_ALL_D(GGML_TYPE_Q6_0, GGML_TYPE_Q5_0) FATTN_VEC_CASES_ALL_D(GGML_TYPE_Q6_0, GGML_TYPE_F16) FATTN_VEC_CASES_ALL_D(GGML_TYPE_F16, GGML_TYPE_Q6_0) + // opencoti-hook: q6_0 KV (#627/bug-2125) — strong-K q6_0 + cheap-V q4_0 asymmetric cell. Under + // FA_ALL_QUANTS the selector admits q6_0-K/q4_0-V, so the stock FA-VEC dispatch (local SWA layers) + // needs this instance; without it the D<=256 dispatch fell through to the GGML_ABORT below. + FATTN_VEC_CASES_ALL_D(GGML_TYPE_Q6_0, GGML_TYPE_Q4_0) // opencoti-hook: turboquant perf-lift (Level B) — F5 M6-S2 #392. Fused turbo K/V vec path // (always registered, independent of GGML_CUDA_FA_ALL_QUANTS). turbo2/3/4 at head_dim ∈ --- /dev/null +++ b/llama.cpp/ggml/src/ggml-cuda/template-instances/fattn-vec-instance-q6_0-q4_0.cu @@ -0,0 +1,11 @@ +// opencoti-hook: q6_0 KV (Anbeeld/beellama.cpp port) — strong-K + cheap-V asymmetric cell (#627/bug-2125). +// q6_0-K / q4_0-V is a first-class KV combo (6-bit K precision, 4-bit V footprint); the local SWA layers +// read it through the stock FA-VEC path (the global layers use the DCA fused kernel's to_fp16_nc dequant), +// so this instance must exist or ggml_cuda_flash_attn_ext_vec_case is missing and the D<=256 +// dispatch aborts at fattn.cu (GGML_ABORT "fatal error"). Mirrors fattn-vec-instance-q6_0-q5_0.cu. + +#include "../fattn-vec.cuh" + +DECL_FATTN_VEC_CASE( 64, GGML_TYPE_Q6_0, GGML_TYPE_Q4_0); +DECL_FATTN_VEC_CASE(128, GGML_TYPE_Q6_0, GGML_TYPE_Q4_0); +DECL_FATTN_VEC_CASE(256, GGML_TYPE_Q6_0, GGML_TYPE_Q4_0); --- a/llamafile/build-functions.sh +++ b/llamafile/build-functions.sh @@ -162,7 +162,8 @@ fi # 3c. opencoti-hook: q6_0 KV (Anbeeld/beellama.cpp port). q6_0 is a scalar KV type read by the - # FA-VEC path; its 5 Anbeeld instances must compile in the DEFAULT build (same #468/bug-339 + # FA-VEC path; its 6 instances (5 Anbeeld + q6_0/q4_0 strong-K/cheap-V, #627) must compile in + # the DEFAULT build (same #468/bug-339 # reasoning as the turbo block above — the default build does NOT glob fattn-vec, so without # this the DSO lacks ggml_cuda_flash_attn_ext_vec_case and dlopen fails the instant # q6_0 KV reaches flash-attention). No-op under FA_ALL_QUANTS=1 (already globbed) and on a @@ -171,6 +172,7 @@ for f in "$ti_dir"/fattn-vec-instance-q6_0-q6_0.cu \ "$ti_dir"/fattn-vec-instance-q8_0-q6_0.cu \ "$ti_dir"/fattn-vec-instance-q6_0-q5_0.cu \ + "$ti_dir"/fattn-vec-instance-q6_0-q4_0.cu \ "$ti_dir"/fattn-vec-instance-q6_0-f16.cu \ "$ti_dir"/fattn-vec-instance-f16-q6_0.cu; do [ -f "$f" ] && CUDA_SOURCES="$CUDA_SOURCES $f"