docs/evaluations/hal_kv_compression.md

# HAL #582 — per-layer KV compression study (Qwen2.5-14B-1M)

**Date:** 2026-07-05 · **Status:** DECIDED — route (b) resident compression wins; route (a)
host-stream-exact demoted to overflow fallback; per-layer type maps rejected.
**Vehicle:** Qwen2.5-14B-Instruct-1M-Q8_0 (48 layers, 8 KV heads, head_dim 128, native 1M,
no YaRN) — the full-attention 1M serving class. **Hosts:** bs2 RTX 6000 Blackwell 96 GB
(PCIe ~50 GB/s), solidPC RTX 3090 (PCIe ~6.5 GB/s).

## 0. The question

HAL asked: can a *frozen* full-attention model's KV footprint be cut for long-context
serving, and by which mechanism? Three candidate routes existed; two died in the 2026-07-01
probes (recorded here for closure), leaving compression — this study quantifies it in VRAM
size, quality, and throughput, and picks the implementation.

| route | mechanism | verdict |
|-------|-----------|---------|
| full→iSWA conversion | few global layers + sliding-window rest | **DEAD** — retrieval is smeared across ~35/48 layers (only L0–4 local); StreamingLLM wall |
| cross-layer sharing (CLA/YOCO) | share K/V across layers | **DEAD** — cross-layer K/V orthogonal on the frozen model (best pair cos ≈ 0.10, needle ≈ 0) |
| per-layer/-token compression | quantized KV, resident | **WINNER** — this document |

Route (a) "host-stream exact" (rolling-KV) and route (b) "compressed resident" were the two
implementations of the surviving lever; §3 settles (a) vs (b) with measurements.

## 1. Size model (measured, matches bpv arithmetic)

f16 KV on this model = **192 KiB/token** (2 × 48 layers × 8 heads × 128 dim × 2 B; confirmed
by boot logs: 43008 cells = 8064 MiB, and §3b `KV self size`).

| KV config | KiB/tok | vs f16 | max resident ctx @ 8 GiB KV budget (24 GB card) | @ ~80 GiB (RTX 6000 96 GB) |
|-----------|---------|--------|------------------------------------------------|-----------------------------|
| f16/f16 | 192 | 100 % | 43.7k | 437k |
| q8_0/q8_0 | 102 | 53.1 % | 82k | 822k |
| **q8_0K/q4_0V** | **78** | **40.6 %** | **107k** | **1.07M** |
| q5_0/q5_0 | 66 | 34.4 % | 127k | 1.27M |
| q4_0/q4_0 | 54 | 28.1 % | 155k | 1.55M |
| turbo3_tcq | 39 | 20.3 % | (quality-dead on this class, §2) | — |

Headline: the model's **native 1M context needs 192 GiB of f16 KV — impossible on any
single card; q8K/q4V brings it to 78 GiB, inside one 96 GB RTX 6000** (78 + 15.7 weights
≈ 94 GiB — tight; comfortable at 768k). Multi-session capacity scales with 1/bytes-per-token:
q8K/q4V = **2.46× sessions per card** at equal context.

## 2. Quality bar (from §5b-14B-1M-ctx, #581 — KLD vs f16, 2k→256k)

The frozen 14B is more quant-sensitive than the 8B and degradation grows with ctx even
sub-native (context.md §5b-14B-1M-ctx). The serving-viable set at 256k:

- **q8_0 / turbo8:** KLD ≤ 0.007 — transparent.
- **asym f16K/q4V ≈ q8K/q4V: 0.008–0.013** — the recommended recipe; hi-precision K is what
  matters, V takes q4 nearly free.
- q5-class ≤ 0.025; t8K/t4V 0.02–0.033.
- q4/q4 0.084 (usable, thin margin); turbo4 0.23 = low-bit floor.
- **turbo2/3 and TCQ tiers: no safe regime on this class** (0.25–2.9) — 2–3-bit history is
  not available here, capping route-(b) compression at ~2.5× (4-bit V, 8-bit K).

niah retrieval: 100 % for q4-class through 512k (§5b full matrix, #575); needle PRESENT in
every cell of §3.

## 3. Throughput — the route decider (bs2 GPU0, greedy needle prompt, 64-tok decode)

### 3a. Route (a) host-stream exact — simulated 24 GB card (8 GiB KV budget, 2026-07-01)

| cell | resident / host tail | decode tps | needle |
|------|----------------------|------------|--------|
| f16 @32k | all / 0 | **67.3** | ✓ |
| f16 @49k | 43k cells / 1152 MiB | 57.6 | ✓ |
| f16 @64k | 43k / **4224 MiB** | **0.99** | ✓ |
| f16 @98k | 43k / 10368 MiB | 0.71 | ✓ |
| q4_0 @64k | all / 0 | **27.1** | ✓ |
| q4_0 @128k | all / 0 | **16.2** | ✓ |

The spill cliff is catastrophic once the tail exceeds ~2 GiB — **0.99 tps at 64k, on bs2's
50 GB/s PCIe**; a desktop 3090's 6.5 GB/s bus is ~8× worse (#584). Every decoded token must
stream the whole tail. Compression at the same context is **27×** faster. Route (a) is dead
as a primary mechanism; it remains the *beyond-max-compression* overflow fallback (rolling-KV
S1–S3, #586–#588, still worth finishing for that regime).

### 3b. Route (b) resident compression — same ctx, everything resident (96 GB card, 2026-07-05)

| ctx | KV | KV MiB | prefill tps | decode tps | needle |
|-----|----|--------|-------------|------------|--------|
| 64k | f16/f16 | 12288 | 3506 | **53.5** | ✓ |
| 64k | q8K/q4V | 4992 | 3396 (−3 %) | 26.7 | ✓ |
| 64k | q4/q4 | 3456 | 3383 (−4 %) | 26.8 | ✓ |
| 128k | f16/f16 | 24576 | 1508 | **39.2** | ✓ |
| 128k | q8K/q4V | 9984 | 1319 (−13 %) | 15.9 | ✓ |
| 128k | q4/q4 | 6912 | 1315 (−13 %) | 16.0 | ✓ |

Two findings:

1. **Prefill tax is small on this class** (−3…−13 %) — the Gemma D512 3–4× MMA-dequant tax
   (poly_kv M6-S0) does not generalize to D128 dense.
2. **Quant decode WAS kernel-bound, NOT bandwidth-bound — and that gap is now CLOSED (#620).**
   q4/q4 reads 31 % fewer bytes than q8K/q4V yet decoded at *identical* tps at both contexts
   (26.8 vs 26.7; 16.0 vs 15.9) — throughput was insensitive to bytes moved, so the quant
   FA-VEC read path (not memory) was the limiter, costing ~2× vs f16. The diagnosis predicted
   that a quant reader reaching f16 efficiency would let q8K/q4V ceiling *above* f16 (it moves
   2.5× fewer bytes). **#620 (#582-P0) realized exactly this:** it gives quant DECODE the same
   tensor-core MMA route f16 takes at high-GQA/long-KV, reading the raw quant blocks in-register.
   Measured (bs2, Qwen2.5-14B-1M Q8_0, 65k, DCA-off): q8K/q4V decode **30.6 tps on the old FA-VEC
   reader → 70.45 tps on the new MMA path (2.30×)**, vs f16 60.99 tps (**1.15× — faster than f16**,
   as predicted), and **logit-identical** (real_frac=0 — the change is lossless). So the "~2× tax /
   ~6× gap" characterization below is the pre-#620 state; it no longer holds for the served scalar
   pairs at D128. Default-on (`OPENCOTI_QUANT_MMA_DECODE`); env=0 reverts to the native FA-VEC
   reader. Generalized to the full scalar-D128 pair set (q8q8/q8q6/q8q5/q8q4/q6q6/q6q4/q5q5/
   q51q51/q4q4/q41q41), mirroring the #644 DCA-on set. See §5 P0 note.

### 3c. Contrast: Gemma-4 26B-A4B (iSWA) — same cells, same day

iSWA is the *architectural* version of this compression: only 5/30 layers scale KV with ctx
(~20 KiB/token vs the 14B's 192 — context.md §3), so there is little left for quant-KV to save
at these contexts:

| ctx | KV | prefill tps | decode tps | needle | total VRAM |
|-----|----|-------------|------------|--------|------------|
| 64k | f16/f16 | 9107 | **174.8** | ✓ | 19005 MiB |
| 64k | q8K/q4V | 9123 (±0 %) | 125.0 | ✓ | 18267 (−0.7 GiB) |
| 64k | q4/q4 | 9103 (±0 %) | 123.2 | ✓ | 18071 (−0.9 GiB) |
| 128k | f16/f16 | 7453 | **157.0** | ✓ | 20285 MiB |
| 128k | q8K/q4V | 7250 (−3 %) | 102.0 | ✓ | 18989 (−1.3 GiB) |
| 128k | q4/q4 | 7218 (−3 %) | 99.4 | ✓ | 18633 (−1.6 GiB) |

- **Quant-KV on iSWA at ≤128k WAS a bad trade: −28…−35 % decode for <1.7 GiB saved.** The
  numbers above are on the pre-#620 binary, where Gemma's D256/D512 scalar decode ran the
  kernel-bound FA-VEC reader. The break-even without a kernel fix moves to ~512k–1M (global KV
  10.6→21 GiB f16, quant saves 6–12.6 GiB) or multi-session KV stacking.
- **The kernel-bound signature reproduces exactly on D512/iSWA:** q4/q4 == q8K/q4V decode
  (123.2 vs 125.0; 99.4 vs 102.0) — bytes don't matter, the quant reader does. This was the
  P0 prediction: fix the reader once and it erases the tax on both head-dim classes at once.
- **✅ ERASED by #674 (0127) — the D256/D512 generalization of P0/#620.** Re-measured on the
  fixed binary (bs2 GPU1, same Gemma-4 A4B native, DCA-off, logit-equiv `real_frac` vs f16
  since Gemma A4B PPL is garbage — never greedy needle):

  | ctx | f16 | q8K/q4V (was VEC) | q4/q4 | q8/q8 | vs f16 | real_frac | env=0 VEC |
  |-----|-----|-------------------|-------|-------|--------|-----------|-----------|
  | 64k  | 162.13 | **174.94** | 170.17 | 173.86 | **+7.9 %** | 0.0 (lossless) | 122.17 (MMA +43 %) |
  | 128k | 152.15 | **154.46** | 152.95 | — | **+1.5 %** | 0.0 (lossless) | — |

  The −28 % (64k) / −35 % (128k) quant tax is not just gone — quant decode now **beats f16**
  (fewer KV bytes moved), and it is **top-token-lossless** to f16 at both depths. The env=0
  line (122.17 = the old VEC path) is the mechanism proof: the MMA fast-path is +43 % over the
  VEC reader that produced the −28 % row. The win narrows at 128k because A4B's MoE-FFN +
  per-step launch overhead dilute the KV-bandwidth delta (and iSWA caps SWA-layer KV at the
  1024 window, so only the D512 globals grow), but scalar stays ≥ f16 throughout. **This makes
  scalar-quant KV the preferred high-VRAM Gemma serving path over turbo** (turbo is slower AND
  decisively lower quality); P1's policy no longer needs the "iSWA + resident → stay f16" rule
  as a *quality/perf* hedge — the only remaining reason to stay f16 is when the VRAM saving
  isn't needed. See UPSTREAM_SYNC `0127`.
- Prefill tax ≈ 0 (the 3–4× M6-S0 D512 dequant tax is gone on the current fused/asym paths);
  needle clean in all six cells (chat endpoint per the Gemma-it greedy-completion gotcha).

### 3d. Contrast: Qwen3.6-35B-A3B (hybrid DeltaNet+attention, NextN MoE) — same cells

The third architecture class: most layers are DeltaNet (constant-size recurrent state —
untouched by `-ctk/-ctv`), only the few full-attention layers grow KV. Measured f16 KV growth
≈ 20 KiB/token (VRAM 64k→128k: +1280 MiB) — same order as A4B's iSWA globals, 10× under the
pure-full-attention 14B:

| ctx | KV | prefill tps | decode tps | needle | total VRAM |
|-----|----|-------------|------------|--------|------------|
| 64k | f16/f16 | 6926 | **178.9** | ✓ | 30595 MiB |
| 64k | q8K/q4V | 6986 (+1 %) | 126.5 | ✓ | 29937 (−0.6 GiB) |
| 64k | q4/q4 | 6989 (+1 %) | 126.9 | ✓ | 29777 (−0.8 GiB) |
| 128k | f16/f16 | 6371 | **158.9** | ✓ | 31875 MiB |
| 128k | q8K/q4V | 6243 (−2 %) | 90.5 | ✓ | 30557 (−1.3 GiB) |
| 128k | q4/q4 | 6244 (−2 %) | 91.0 | ✓ | 30237 (−1.6 GiB) |

Same shape as iSWA, sharper: **−29 % (64k) → −43 % (128k) decode for ≤1.6 GiB saved**, prefill
free, needle clean, and the kernel-bound signature for the third time (q4/q4 == q8K/q4V:
126.9 vs 126.5; 91.0 vs 90.5).

> **Spill IS a graceful fallback here (DCA-off) — but compression is still the primary
> resident lever.** The fine 256k tail ladder on this same qw35a3 (`rolling_kv.md` §5c,
> #631/#633) shows **DCA-off** spill is bandwidth-bound (~57–62 GiB/s, needle ✓) and holds
> within ~15 % of resident to a ~680 MiB tail — the correct rolling-KV shape. **DCA-on** is
> penalty-free to ~800 MiB then cliffs ~38× at ~1 GiB (bug-2144: DCA bypasses the streaming
> window path → full-KV concat round-trip; ~20× slower than DCA-off at a matched 1.1 GiB
> tail). Either way, `turbo2` resident (46.4 / 27.3 tps @256k/512k, KV 0.68 / 1.36 GiB) keeps
> qw35a3 fully on-device and its KV is small enough (5–10 GiB) to never need spilling on a
> 96 GB card — so **compression/residency is the capacity lever; DCA-off spill is a valid
> overflow fallback for smaller cards / multi-session pressure**. (Engaging spill for hybrid
> models at all required fixing bug-2142 — the residency knobs were dropped by the
> `llama_memory_hybrid` wrapper; #633.)

**Cross-class summary — quant-KV decode tax vs VRAM saved @128k, all measured 2026-07-05:**

| class | model | KV growth | decode tax | VRAM saved | quant verdict at ≤128k |
|-------|-------|-----------|------------|------------|------------------------|
| full attention | Qwen2.5-14B-1M | 192 KiB/tok | −59 % | 14.6 GiB | **worth it** (capacity is binding) |
| iSWA | Gemma-4 26B-A4B | ~20 KiB/tok | −35 % | 1.3 GiB | not worth it (stay f16) |
| hybrid recurrent | Qwen3.6-35B-A3B | ~20 KiB/tok | −43 % | 1.3 GiB | not worth it (stay f16) |

The P1 policy rule generalizes: **quantize KV only when residency at the target context
demands it** — which in practice means pure-full-attention models, very long ctx (≥512k) on
the bounded architectures, or multi-session KV stacking. P0 (kernel-bound fix) benefits all
three classes and would collapse the tax column toward zero.

## 4. Verdict

**Resident compression (route b) is the per-layer KV-compression mechanism, with q8_0K/q4_0V
as the default tier for the full-attention 1M class:** 2.46× KV capacity, KLD ≤ 0.013 @256k,
niah clean, prefill ≈ f16, and 22–27× faster than host-streaming at equal context. Its one
cost — the ~2× quant decode-kernel tax — was an engineering gap with measured headroom, not a
structural limit, and **#620 has now closed it** (q8K/q4V decode 30.6→70.45 tps, 1.15× *over*
f16, lossless; see §3b finding 2 + §5 P0 note). Per-layer *type maps* (layer 3 at q8, layer 30 at q4) are rejected: the
locality probe showed retrieval smeared across ~35/48 layers, so there is no cheap layer
subset to downgrade selectively.

## 5. Proposed implementation (ordered; P0 gates P2's value)

**P0 — make quant-KV decode bandwidth-bound (the 2× reclaim).** Profile the scalar-quant
FA-VEC decode at n_kv 64k–128k on Blackwell (nsys, `--cuda-graph-trace=node`); suspects:
vec-kernel occupancy/ILP on long KV, kernel-selection cliff (f16 taking a better path at
D128), per-block dequant overhead. Same playbook as #533/#542 (smem/vectorized V-read fixed
turbo tiers). Gate: q8K/q4V decode ≥ f16 decode at 128k (it reads 2.5× less — it should win).
Cheapest step, largest payoff, benefits every quant tier everywhere.

> **P0 — DCA-on SHIPPED; DCA-off now SHIPPED too (#620).** The two decode paths
> are separate kernels and shipped separately.
> **DCA-off** (#620, patch `0126`, 2026-07-14): the plain (non-DCA) scalar-quant decode now
> takes the tensor-core MMA route via typed instances
> (`fattn-mma-f16-scalar-<K>-<V>-d128.cu`) that read the raw quant blocks in-register
> (`load_tile_dequant`/`get_dequantize_V`) — the same machinery the turbo tiers and the #644
> DCA-on set use, no new kernel math. Selector-escaped for the served scalar-D128 pairs
> (`octi_scalar_mma_pair_d128`), default-on (`OPENCOTI_QUANT_MMA_DECODE`), env=0 → native
> FA-VEC. **Measured** (bs2, Qwen2.5-14B-1M Q8_0, 65k): q8K/q4V decode **70.45 tps vs f16
> 60.99 (1.15×) and 2.30× over the native FA-VEC baseline (30.6)**, **logit-identical**
> (real_frac=0 vs both f16 and native VEC). Generalized to the full scalar set (q8q8/q8q6/
> q8q5/q8q4/q6q6/q6q4/q5q5/q51q51/q4q4/q41q41). This also lifts **P2's throughput half**:
> P2's quant-tail read runs through the non-DCA streaming path, which now shares the same
> in-register readers.
> **DCA-on** (#643 beachhead + #644 full matrix, patch `0119`, 2026-07-12): the multichunk
> `dca_fused` MMA kernel previously lifted the WHOLE cache to f16 via `to_fp16_nc`
> every decode step (§3b ~29.5 s/decode); scalar q4_0/q4_1/q5_0/q5_1/q6_0/q8_0 K/V
> are now read IN-REGISTER via the same `load_tile_dequant`/`get_dequantize_V` path
> the turbo tiers use (no new kernel math). **#643 beachhead** q8_0-K/q4_0-V @D128
> ctx128k: `real_frac 0.0` (bit-identical to the shipped materialize) + **2.13×
> decode (14.72→31.33 tps)**. **#644 full matrix** @D128 32k multichunk DCA-on: all
> 10 pairs `real_frac 0.0` — symmetric diagonal `{q4_0,q4_1,q5_0,q5_1,q6_0,q8_0}`
> K==V + hi-K/cheap-V `{q8K/{q4,q5,q6}V, q6K/q4V}`. Decode-gated (`Q->ne[1] ≤
> WS2_NQ_MAX`=16, env `WS2_DCA_SCALAR_KV`); large-n_q prefill keeps the fast
> in-launch f16 lift (bug-720). D256/D512 scalar deferred (Gemma KV is turbo,
> already in-register). This unblocks P2.

**P1 — auto KV-tier policy at boot (capacity without user knobs). — SHIPPED (#621, patch
0116).** Given `-c`, free VRAM, and the model's KiB/token, pick the *least* compression that
keeps KV fully resident: f16 → q8/q8 → q8K/q4V → (last resort) q5K/q4V; never auto-pick below
the §2 quality bar for the detected class; spill only if even the floor doesn't fit (then
rolling-KV takes the excess). Anchors: in-binary auto-policy precedent `llama-context.cpp`
(#567 sparse-V auto), bounded-defaults precedent in `buildServerArgs` (#564); log the choice
at boot like the `position window` line. Small, additive, immediately useful for the
multi-session target.

*Implementation:* `opencoti_auto_select_kv_tier` (`llama-kv-cache.cpp`), run in the
`llama_kv_cache` ctor BEFORE any residency/allocation math consumes `type_k`/`type_v`. It
reuses the EXACT budget arithmetic of `opencoti_compute_resident_window_cells` (free −
already-resident − 1.5 GiB compute-reserve, capped by `--vram-target`) and the same per-layer
per-cell byte sum, then walks the ladder `{f16, q8_0/q8_0, q8_0/q4_0, q5_1/q4_0}` and picks the
first (least-compressing) rung whose whole-cache cost fits; floor if none fit. **Strictly
opt-in** (`OPENCOTI_KV_AUTO_TIER=1`) and a no-op — byte-identical — when: no GPU offload; the
user set `-ctk`/`-ctv` off the f16 default (override respected); iSWA (`swa_type != NONE`,
Gemma stays f16/resident per §3c); or f16 already fits. Announced at WARN with a
`set -ctk/-ctv to override` hint (mirrors #567). Host adapter opt-in: `kvAutoTier` config →
`OPENCOTI_KV_AUTO_TIER` env in `buildServerArgs`.

*Validated (bs2, Qwen2.5-14B-1M Q8_0, `-c 32768`, RTX 6000):* the ladder walks correctly as
`--vram-target` tightens — 20000 MiB → q8_0/q8_0 (KV 3264 MiB fits budget 3328), 19400 →
q8_0/q4_0 (2496 fits 2728), 18800 → q5_1/q4_0 (2016 fits 2128); every rung logged `position
window = FULLY RESIDENT … no host tail` and returned the passkey needle cleanly. Config-
equivalence: auto q8_0/q8_0 output was byte-identical to an explicit `-ctk q8_0 -ctv q8_0`
boot, which emitted zero auto-tier WARN (opt-in path confirmed off).

**P2 — position-axis mixed KV: f16 recent window ⊕ compressed VRAM tail.** The natural
extension once P0 lands, built entirely on existing seams — this is the "low-bit history ⊕
exact recent" HAL route, kept in VRAM:
- *Storage:* reuse the POSITION_WINDOW split (`llama-kv-cache.h:294-296` get_k/v_window +
  get_k/v_tail accessors, per-layer budgets `headinfer_window_cells`, S0 #585) — but the
  tail tensor becomes a **device** buffer of quantized type instead of pinned-host f16.
- *Write path:* tokens quantize once on eviction from the rolling f16 window, batched through
  the existing GPU quantize-on-write kernels (`set-rows.cu` k_set_rows_* family, #375/#380).
- *Read path:* the two-region streaming FA op (`ggml.h:2481`
  `ggml_streaming_flash_attn_window`) already merges window+tail with exact online-softmax
  combine; each region launches its own typed kernel, so mixed f16-window/quant-tail needs
  no new FA instances — only supports_op relaxation on the tail type.
- *Value over uniform quant:* recent tokens (the ones agentic loops re-read hardest) stay
  exact; fresh-token writes skip the encode in the hot path; quality strictly ≥ uniform tier
  at equal tail type, opening the door to a q4-class or t8K/t4V tail (78→~60 KiB/tok) that
  uniform quant can't risk. Gate with KLD @256k + niah ladder vs uniform q8K/q4V.
- *Ordering:* only worth building after P0 — today the tail's quant kernel tax would dominate
  and mask the win.

**P3 — per-layer type maps: rejected** (probe evidence above). Revisit only with a genuinely
new selection mechanism, per the #551 lesson.

## 6. Provenance

- Resident cells (§3b): `bs2:/srv/ml/opencoti-c1/hal-resident.sh` → `hal582-out/` (per-cell
  logs + `run.log`), binary `llamafile.dualctx` c-series, 2026-07-05.
- A4B contrast (§3c): `bs2:/srv/ml/opencoti-c1/hal-resident-a4b.sh` → `hal582a4b-out/`,
  `google_gemma-4-26B-A4B-it-Q4_K_M.gguf` (registry sha e718536f), chat endpoint, 2026-07-05.
- 35B contrast (§3d): `bs2:/srv/ml/opencoti-c1/hal-resident-35b.sh` → `hal58235b-out/`,
  `Qwen3.6-35B-A3B-UD-Q6_K.gguf` (registry sha 49935b04), chat endpoint, 2026-07-05.
- Overspill cells (§3a): `bs2:/srv/ml/opencoti-c1/bs2-overspill.sh` → `overspill-out/`,
  2026-07-01 (VT=9600 simulating 24 GB).
- Quality: context.md §5b-14B-1M-ctx (#581), §5b full matrix (#529), DCA niah matrix (#575).
- Probes: `.opencoti/hal/window_sim.py` (locality), cross-layer K/V cosine probe (#582
  2026-07-01, pgvector `project_qwen_retrieval_distributed_hal`).
- Cliff curves: #583/#584 (3090 + bs2), PCIe reference `reference_pcie_bandwidth_hosts`.