From: opencoti
Subject: [PATCH 0088] sparse-v auto-policy: self-configure quantized-V skip from model arch (#565)

The FA-VEC quantized-V skip (sparse-V, #546) is a MEASURED lossless decode win on
interleaved-SWA models (Gemma-4: the small, concentrated global V-cache lets a
peak-relative static tau=0.05 harvest the sub-mean V tail — RULER niah 100 at
64k/128k/256k on A4B-62e AND official Gemma-4-31B), but a no-win on full-attention
models (Qwen: diffuse weights cluster near the mean, nothing safely skippable). The
kernel reads its threshold from the TURBO_SPARSE_V_TAU env (ggml-cuda is a runtime
DSO, so env is the cross-DSO channel). Until now a user had to know to set that env.

This puts the intelligence in the BINARY so a STANDALONE `llamafile --server` self-
configures: at llama_context construction, when the user has NOT set TURBO_SPARSE_V_TAU
and the model is iSWA (hparams.swa_type != NONE) with a quantized V cache, default the
env to 0.05 (overwrite=0 — an explicit env still wins). Full-attention models leave it
unset -> kernel tau=0 -> dense, byte-identical. f16/bf16 V is inert via the kernel's own
`if constexpr type_V != F16/BF16` guard. Host-only (no DSO change); the existing kernel
getenv reads the pre-populated value. Announced once at WARN (the llamafile server
suppresses the model-load INFO block, and a binary that auto-changes inference should
say so + tell the user how to override).

Verified (3090): decision table PASS (policy fires iff iSWA+quant-V+no-env: A4B q8_0
auto=engaged, override/f16/Qwen-full-attn=off) + A/B at 24k lossless + non-slower.
---
diff --git a/llama.cpp/src/llama-context.cpp b/llama.cpp/src/llama-context.cpp
index 722bd7f..2d87414 100644
--- a/llama.cpp/src/llama-context.cpp
+++ b/llama.cpp/src/llama-context.cpp
@@ -54,6 +54,34 @@ llama_context::llama_context(
 
     const auto & hparams = model.hparams;
 
+    // opencoti-hook: sparse-v auto-policy (#565) — see docs/features/sparse_attn.md
+    // Self-configure the FA-VEC quantized-V skip (#546) from the model's OWN architecture, so a
+    // STANDALONE `llamafile --server` gets the win with no flags (the intelligence lives in the binary,
+    // not the opencoti TS adapter). The skip is a MEASURED lossless decode win on interleaved-SWA models
+    // (Gemma-4: the tiny, concentrated global V-cache lets a peak-relative static τ=0.05 harvest the
+    // sub-mean V tail — niah 100 at 64k/128k/256k on A4B-62e AND official Gemma-4-31B), but a no-win on
+    // full-attention models (Qwen: diffuse weights cluster near the mean, nothing safely skippable). It
+    // only engages for a QUANTIZED V cache (the kernel's `if constexpr type_V != F16/BF16` guard; -ctv
+    // f16 is inert). ggml-cuda is a runtime-loaded DSO, so the kernel reads its threshold from the
+    // TURBO_SPARSE_V_TAU env (the existing cross-DSO channel) — we just pre-populate a smart default when
+    // the user has not, with overwrite=0 so an explicit TURBO_SPARSE_V_TAU (or eps/sink/recent) still
+    // wins. Set as early as possible (before any graph_reserve / decode) so the kernel's lazy getenv
+    // caches the policy value. Full-attention models leave the env unset → kernel τ=0 → dense, byte-identical.
+    if (!hparams.vocab_only && getenv("TURBO_SPARSE_V_TAU") == nullptr) {
+        const bool is_iswa = hparams.swa_type != LLAMA_SWA_TYPE_NONE;
+        // Mirror the kernel's engage guard exactly: any non-float V cache (q8_0/q4_0/turbo*/tcq) is eligible.
+        const bool quant_v = params.type_v != GGML_TYPE_F16 && params.type_v != GGML_TYPE_BF16 &&
+                             params.type_v != GGML_TYPE_F32;
+        if (is_iswa && quant_v) {
+            setenv("TURBO_SPARSE_V_TAU", "0.05", /*overwrite=*/0);
+            // WARN level (not INFO): the llamafile server suppresses the model-load INFO block, and a binary
+            // that auto-changes inference behavior should announce it audibly + tell the user how to override.
+            LLAMA_LOG_WARN("%s: sparse-v auto-policy: iSWA + quantized V (%s) -> TURBO_SPARSE_V_TAU=0.05 "
+                           "(lossless decode win; export TURBO_SPARSE_V_TAU to override)\n",
+                    __func__, ggml_type_name(params.type_v));
+        }
+    }
+
     cparams.n_seq_max = std::max(1u, params.n_seq_max);
     if (cparams.n_seq_max > LLAMA_MAX_SEQ) {
         throw std::runtime_error("n_seq_max must be <= " + std::to_string(LLAMA_MAX_SEQ));