patches/0101-kv-attn-rot-inherit.patch

opencoti patch 0101 — shared draft cache inherits attention-rotation state (bug-2120)

llamafile 0.10.3 attention rotation (Hadamard-rotate K/V rows before quantized
store when the cache type is quantized && head_dim%64==0; readers rotate Q and
un-rotate the output via attn_inp_k_rot/v_rot) is computed per-cache from its
OWN type_k/type_v and accumulated head dims. The dual-context assistant draft
cache is created with f16 types AND its shared layers skip the head-dim
accumulation, so rotation silently disabled: the drafter read the target's
ROTATED quant K/V with an unrotated Q -> decorrelated attention, flat finite
logits, accept ~0.000 with ANY quant target KV (f16/f16 unaffected; Qwen NextN
unaffected — it reads through the target's own cache object).

Fix: in the ctor share branch, inherit attn_rot_k/v, n_embd_head_*_all and the
Hadamard tables from the source cache, overriding the local computation. The
generic iswa build_attn then applies the rotation on the drafter's Q-only path.

Gate (bs2, A4B + assistant Q8_0 drafter, 32k, 24k prompt): accept
k8_v16 0.000 -> 0.641; k8_v4 (production PolyKV) 0.000 -> 0.729; f16 control
0.677 unchanged.

--- a/llama.cpp/src/llama-kv-cache.cpp
+++ b/llama.cpp/src/llama-kv-cache.cpp
@@ -1106,6 +1106,26 @@
         }
     }
 
+    // opencoti bug-2120: a shared (dual-context drafter) cache must READ with the same
+    // attention-rotation state the SOURCE cache used to WRITE. The local computation above
+    // sees this cache's own type_k/type_v (f16 for the drafter) and n_embd_head_*_all == 0
+    // (shared layers `continue` before accumulating), so it always disables rotation — the
+    // drafter then reads Hadamard-rotated quant K/V with an unrotated Q and un-unrotated
+    // output: decorrelated attention, accept ~0 with ANY quant target KV (f16/f16 unaffected
+    // because no rotation is applied at all). Inherit flags + head-dim extents + Hadamard
+    // tables from `other` so build_input_k_rot/v_rot emit the matching rotation inputs.
+    if (share && other) {
+        attn_rot_k        = other->attn_rot_k;
+        attn_rot_v        = other->attn_rot_v;
+        n_embd_head_k_all = other->n_embd_head_k_all;
+        n_embd_head_v_all = other->n_embd_head_v_all;
+        attn_rot_hadamard = other->attn_rot_hadamard;
+        if (attn_rot_k || attn_rot_v) {
+            LLAMA_LOG_INFO("%s: shared cache inherits attn_rot_k=%d attn_rot_v=%d from source (bug-2120)\n",
+                    __func__, (int) attn_rot_k, (int) attn_rot_v);
+        }
+    }
+
     const char * LLAMA_KV_CACHE_DEBUG = getenv("LLAMA_KV_CACHE_DEBUG");
     debug = LLAMA_KV_CACHE_DEBUG ? atoi(LLAMA_KV_CACHE_DEBUG) : 0;
 }