HAL #582 — per-layer KV compression study (Qwen2.5-14B-1M)
Date: 2026-07-05 · Status: DECIDED — route (b) resident compression wins; route (a) host-stream-exact demoted to overflow fallback; per-layer type maps rejected. Vehicle: Qwen2.5-14B-Instruct-1M-Q8_0 (48 layers, 8 KV heads, head_dim 128, native 1M, no YaRN) — the full-attention 1M serving class. Hosts: bs2 RTX 6000 Blackwell 96 GB (PCIe ~50 GB/s), solidPC RTX 3090 (PCIe ~6.5 GB/s).
0. The question
HAL asked: can a frozen full-attention model's KV footprint be cut for long-context serving, and by which mechanism? Three candidate routes existed; two died in the 2026-07-01 probes (recorded here for closure), leaving compression — this study quantifies it in VRAM size, quality, and throughput, and picks the implementation.
| route | mechanism | verdict |
|---|---|---|
| full→iSWA conversion | few global layers + sliding-window rest | DEAD — retrieval is smeared across ~35/48 layers (only L0–4 local); StreamingLLM wall |
| cross-layer sharing (CLA/YOCO) | share K/V across layers | DEAD — cross-layer K/V orthogonal on the frozen model (best pair cos ≈ 0.10, needle ≈ 0) |
| per-layer/-token compression | quantized KV, resident | WINNER — this document |
Route (a) "host-stream exact" (rolling-KV) and route (b) "compressed resident" were the two implementations of the surviving lever; §3 settles (a) vs (b) with measurements.
1. Size model (measured, matches bpv arithmetic)
f16 KV on this model = 192 KiB/token (2 × 48 layers × 8 heads × 128 dim × 2 B; confirmed
by boot logs: 43008 cells = 8064 MiB, and §3b KV self size).
| KV config | KiB/tok | vs f16 | max resident ctx @ 8 GiB KV budget (24 GB card) | @ ~80 GiB (RTX 6000 96 GB) |
|---|---|---|---|---|
| f16/f16 | 192 | 100 % | 43.7k | 437k |
| q8_0/q8_0 | 102 | 53.1 % | 82k | 822k |
| q8_0K/q4_0V | 78 | 40.6 % | 107k | 1.07M |
| q5_0/q5_0 | 66 | 34.4 % | 127k | 1.27M |
| q4_0/q4_0 | 54 | 28.1 % | 155k | 1.55M |
| turbo3_tcq | 39 | 20.3 % | (quality-dead on this class, §2) | — |
Headline: the model's native 1M context needs 192 GiB of f16 KV — impossible on any single card; q8K/q4V brings it to 78 GiB, inside one 96 GB RTX 6000 (78 + 15.7 weights ≈ 94 GiB — tight; comfortable at 768k). Multi-session capacity scales with 1/bytes-per-token: q8K/q4V = 2.46× sessions per card at equal context.
2. Quality bar (from §5b-14B-1M-ctx, #581 — KLD vs f16, 2k→256k)
The frozen 14B is more quant-sensitive than the 8B and degradation grows with ctx even sub-native (context.md §5b-14B-1M-ctx). The serving-viable set at 256k:
- q8_0 / turbo8: KLD ≤ 0.007 — transparent.
- asym f16K/q4V ≈ q8K/q4V: 0.008–0.013 — the recommended recipe; hi-precision K is what matters, V takes q4 nearly free.
- q5-class ≤ 0.025; t8K/t4V 0.02–0.033.
- q4/q4 0.084 (usable, thin margin); turbo4 0.23 = low-bit floor.
- turbo2/3 and TCQ tiers: no safe regime on this class (0.25–2.9) — 2–3-bit history is not available here, capping route-(b) compression at ~2.5× (4-bit V, 8-bit K).
niah retrieval: 100 % for q4-class through 512k (§5b full matrix, #575); needle PRESENT in every cell of §3.
3. Throughput — the route decider (bs2 GPU0, greedy needle prompt, 64-tok decode)
3a. Route (a) host-stream exact — simulated 24 GB card (8 GiB KV budget, 2026-07-01)
| cell | resident / host tail | decode tps | needle |
|---|---|---|---|
| f16 @32k | all / 0 | 67.3 | ✓ |
| f16 @49k | 43k cells / 1152 MiB | 57.6 | ✓ |
| f16 @64k | 43k / 4224 MiB | 0.99 | ✓ |
| f16 @98k | 43k / 10368 MiB | 0.71 | ✓ |
| q4_0 @64k | all / 0 | 27.1 | ✓ |
| q4_0 @128k | all / 0 | 16.2 | ✓ |
The spill cliff is catastrophic once the tail exceeds ~2 GiB — 0.99 tps at 64k, on bs2's 50 GB/s PCIe; a desktop 3090's 6.5 GB/s bus is ~8× worse (#584). Every decoded token must stream the whole tail. Compression at the same context is 27× faster. Route (a) is dead as a primary mechanism; it remains the beyond-max-compression overflow fallback (rolling-KV S1–S3, #586–#588, still worth finishing for that regime).
3b. Route (b) resident compression — same ctx, everything resident (96 GB card, 2026-07-05)
| ctx | KV | KV MiB | prefill tps | decode tps | needle |
|---|---|---|---|---|---|
| 64k | f16/f16 | 12288 | 3506 | 53.5 | ✓ |
| 64k | q8K/q4V | 4992 | 3396 (−3 %) | 26.7 | ✓ |
| 64k | q4/q4 | 3456 | 3383 (−4 %) | 26.8 | ✓ |
| 128k | f16/f16 | 24576 | 1508 | 39.2 | ✓ |
| 128k | q8K/q4V | 9984 | 1319 (−13 %) | 15.9 | ✓ |
| 128k | q4/q4 | 6912 | 1315 (−13 %) | 16.0 | ✓ |
Two findings:
- Prefill tax is small on this class (−3…−13 %) — the Gemma D512 3–4× MMA-dequant tax (poly_kv M6-S0) does not generalize to D128 dense.
- Quant decode WAS kernel-bound, NOT bandwidth-bound — and that gap is now CLOSED (#620).
q4/q4 reads 31 % fewer bytes than q8K/q4V yet decoded at identical tps at both contexts
(26.8 vs 26.7; 16.0 vs 15.9) — throughput was insensitive to bytes moved, so the quant
FA-VEC read path (not memory) was the limiter, costing
2× vs f16. The diagnosis predicted that a quant reader reaching f16 efficiency would let q8K/q4V ceiling above f16 (it moves 2.5× fewer bytes). #620 (#582-P0) realized exactly this: it gives quant DECODE the same tensor-core MMA route f16 takes at high-GQA/long-KV, reading the raw quant blocks in-register. Measured (bs2, Qwen2.5-14B-1M Q8_0, 65k, DCA-off): q8K/q4V decode 30.6 tps on the old FA-VEC reader → 70.45 tps on the new MMA path (2.30×), vs f16 60.99 tps (1.15× — faster than f16, as predicted), and logit-identical (real_frac=0 — the change is lossless). So the "2× tax / ~6× gap" characterization below is the pre-#620 state; it no longer holds for the served scalar pairs at D128. Default-on (OPENCOTI_QUANT_MMA_DECODE); env=0 reverts to the native FA-VEC reader. Generalized to the full scalar-D128 pair set (q8q8/q8q6/q8q5/q8q4/q6q6/q6q4/q5q5/ q51q51/q4q4/q41q41), mirroring the #644 DCA-on set. See §5 P0 note.
3c. Contrast: Gemma-4 26B-A4B (iSWA) — same cells, same day
iSWA is the architectural version of this compression: only 5/30 layers scale KV with ctx (~20 KiB/token vs the 14B's 192 — context.md §3), so there is little left for quant-KV to save at these contexts:
| ctx | KV | prefill tps | decode tps | needle | total VRAM |
|---|---|---|---|---|---|
| 64k | f16/f16 | 9107 | 174.8 | ✓ | 19005 MiB |
| 64k | q8K/q4V | 9123 (±0 %) | 125.0 | ✓ | 18267 (−0.7 GiB) |
| 64k | q4/q4 | 9103 (±0 %) | 123.2 | ✓ | 18071 (−0.9 GiB) |
| 128k | f16/f16 | 7453 | 157.0 | ✓ | 20285 MiB |
| 128k | q8K/q4V | 7250 (−3 %) | 102.0 | ✓ | 18989 (−1.3 GiB) |
| 128k | q4/q4 | 7218 (−3 %) | 99.4 | ✓ | 18633 (−1.6 GiB) |
Quant-KV on iSWA at ≤128k WAS a bad trade: −28…−35 % decode for <1.7 GiB saved. The numbers above are on the pre-#620 binary, where Gemma's D256/D512 scalar decode ran the kernel-bound FA-VEC reader. The break-even without a kernel fix moves to ~512k–1M (global KV 10.6→21 GiB f16, quant saves 6–12.6 GiB) or multi-session KV stacking.
The kernel-bound signature reproduces exactly on D512/iSWA: q4/q4 == q8K/q4V decode (123.2 vs 125.0; 99.4 vs 102.0) — bytes don't matter, the quant reader does. This was the P0 prediction: fix the reader once and it erases the tax on both head-dim classes at once.
✅ ERASED by #674 (0127) — the D256/D512 generalization of P0/#620. Re-measured on the fixed binary (bs2 GPU1, same Gemma-4 A4B native, DCA-off, logit-equiv
real_fracvs f16 since Gemma A4B PPL is garbage — never greedy needle):ctx f16 q8K/q4V (was VEC) q4/q4 q8/q8 vs f16 real_frac env=0 VEC 64k 162.13 174.94 170.17 173.86 +7.9 % 0.0 (lossless) 122.17 (MMA +43 %) 128k 152.15 154.46 152.95 — +1.5 % 0.0 (lossless) — The −28 % (64k) / −35 % (128k) quant tax is not just gone — quant decode now beats f16 (fewer KV bytes moved), and it is top-token-lossless to f16 at both depths. The env=0 line (122.17 = the old VEC path) is the mechanism proof: the MMA fast-path is +43 % over the VEC reader that produced the −28 % row. The win narrows at 128k because A4B's MoE-FFN + per-step launch overhead dilute the KV-bandwidth delta (and iSWA caps SWA-layer KV at the 1024 window, so only the D512 globals grow), but scalar stays ≥ f16 throughout. This makes scalar-quant KV the preferred high-VRAM Gemma serving path over turbo (turbo is slower AND decisively lower quality); P1's policy no longer needs the "iSWA + resident → stay f16" rule as a quality/perf hedge — the only remaining reason to stay f16 is when the VRAM saving isn't needed. See UPSTREAM_SYNC
0127.Prefill tax ≈ 0 (the 3–4× M6-S0 D512 dequant tax is gone on the current fused/asym paths); needle clean in all six cells (chat endpoint per the Gemma-it greedy-completion gotcha).
3d. Contrast: Qwen3.6-35B-A3B (hybrid DeltaNet+attention, NextN MoE) — same cells
The third architecture class: most layers are DeltaNet (constant-size recurrent state —
untouched by -ctk/-ctv), only the few full-attention layers grow KV. Measured f16 KV growth
≈ 20 KiB/token (VRAM 64k→128k: +1280 MiB) — same order as A4B's iSWA globals, 10× under the
pure-full-attention 14B:
| ctx | KV | prefill tps | decode tps | needle | total VRAM |
|---|---|---|---|---|---|
| 64k | f16/f16 | 6926 | 178.9 | ✓ | 30595 MiB |
| 64k | q8K/q4V | 6986 (+1 %) | 126.5 | ✓ | 29937 (−0.6 GiB) |
| 64k | q4/q4 | 6989 (+1 %) | 126.9 | ✓ | 29777 (−0.8 GiB) |
| 128k | f16/f16 | 6371 | 158.9 | ✓ | 31875 MiB |
| 128k | q8K/q4V | 6243 (−2 %) | 90.5 | ✓ | 30557 (−1.3 GiB) |
| 128k | q4/q4 | 6244 (−2 %) | 91.0 | ✓ | 30237 (−1.6 GiB) |
Same shape as iSWA, sharper: −29 % (64k) → −43 % (128k) decode for ≤1.6 GiB saved, prefill free, needle clean, and the kernel-bound signature for the third time (q4/q4 == q8K/q4V: 126.9 vs 126.5; 91.0 vs 90.5).
Spill IS a graceful fallback here (DCA-off) — but compression is still the primary resident lever. The fine 256k tail ladder on this same qw35a3 (
rolling_kv.md§5c, #631/#633) shows DCA-off spill is bandwidth-bound (~57–62 GiB/s, needle ✓) and holds within ~15 % of resident to a ~680 MiB tail — the correct rolling-KV shape. DCA-on is penalty-free to ~800 MiB then cliffs ~38× at ~1 GiB (bug-2144: DCA bypasses the streaming window path → full-KV concat round-trip; ~20× slower than DCA-off at a matched 1.1 GiB tail). Either way,turbo2resident (46.4 / 27.3 tps @256k/512k, KV 0.68 / 1.36 GiB) keeps qw35a3 fully on-device and its KV is small enough (5–10 GiB) to never need spilling on a 96 GB card — so compression/residency is the capacity lever; DCA-off spill is a valid overflow fallback for smaller cards / multi-session pressure. (Engaging spill for hybrid models at all required fixing bug-2142 — the residency knobs were dropped by thellama_memory_hybridwrapper; #633.)
Cross-class summary — quant-KV decode tax vs VRAM saved @128k, all measured 2026-07-05:
| class | model | KV growth | decode tax | VRAM saved | quant verdict at ≤128k |
|---|---|---|---|---|---|
| full attention | Qwen2.5-14B-1M | 192 KiB/tok | −59 % | 14.6 GiB | worth it (capacity is binding) |
| iSWA | Gemma-4 26B-A4B | ~20 KiB/tok | −35 % | 1.3 GiB | not worth it (stay f16) |
| hybrid recurrent | Qwen3.6-35B-A3B | ~20 KiB/tok | −43 % | 1.3 GiB | not worth it (stay f16) |
The P1 policy rule generalizes: quantize KV only when residency at the target context demands it — which in practice means pure-full-attention models, very long ctx (≥512k) on the bounded architectures, or multi-session KV stacking. P0 (kernel-bound fix) benefits all three classes and would collapse the tax column toward zero.
4. Verdict
Resident compression (route b) is the per-layer KV-compression mechanism, with q8_0K/q4_0V as the default tier for the full-attention 1M class: 2.46× KV capacity, KLD ≤ 0.013 @256k, niah clean, prefill ≈ f16, and 22–27× faster than host-streaming at equal context. Its one cost — the ~2× quant decode-kernel tax — was an engineering gap with measured headroom, not a structural limit, and #620 has now closed it (q8K/q4V decode 30.6→70.45 tps, 1.15× over f16, lossless; see §3b finding 2 + §5 P0 note). Per-layer type maps (layer 3 at q8, layer 30 at q4) are rejected: the locality probe showed retrieval smeared across ~35/48 layers, so there is no cheap layer subset to downgrade selectively.
5. Proposed implementation (ordered; P0 gates P2's value)
P0 — make quant-KV decode bandwidth-bound (the 2× reclaim). Profile the scalar-quant
FA-VEC decode at n_kv 64k–128k on Blackwell (nsys, --cuda-graph-trace=node); suspects:
vec-kernel occupancy/ILP on long KV, kernel-selection cliff (f16 taking a better path at
D128), per-block dequant overhead. Same playbook as #533/#542 (smem/vectorized V-read fixed
turbo tiers). Gate: q8K/q4V decode ≥ f16 decode at 128k (it reads 2.5× less — it should win).
Cheapest step, largest payoff, benefits every quant tier everywhere.
P0 — DCA-on SHIPPED; DCA-off now SHIPPED too (#620). The two decode paths are separate kernels and shipped separately. DCA-off (#620, patch
0126, 2026-07-14): the plain (non-DCA) scalar-quant decode now takes the tensor-core MMA route via typed instances (fattn-mma-f16-scalar-<K>-<V>-d128.cu) that read the raw quant blocks in-register (load_tile_dequant/get_dequantize_V) — the same machinery the turbo tiers and the #644 DCA-on set use, no new kernel math. Selector-escaped for the served scalar-D128 pairs (octi_scalar_mma_pair_d128), default-on (OPENCOTI_QUANT_MMA_DECODE), env=0 → native FA-VEC. Measured (bs2, Qwen2.5-14B-1M Q8_0, 65k): q8K/q4V decode 70.45 tps vs f16 60.99 (1.15×) and 2.30× over the native FA-VEC baseline (30.6), logit-identical (real_frac=0 vs both f16 and native VEC). Generalized to the full scalar set (q8q8/q8q6/ q8q5/q8q4/q6q6/q6q4/q5q5/q51q51/q4q4/q41q41). This also lifts P2's throughput half: P2's quant-tail read runs through the non-DCA streaming path, which now shares the same in-register readers. DCA-on (#643 beachhead + #644 full matrix, patch0119, 2026-07-12): the multichunkdca_fusedMMA kernel previously lifted the WHOLE cache to f16 viato_fp16_ncevery decode step (§3b ~29.5 s/decode); scalar q4_0/q4_1/q5_0/q5_1/q6_0/q8_0 K/V are now read IN-REGISTER via the sameload_tile_dequant/get_dequantize_Vpath the turbo tiers use (no new kernel math). #643 beachhead q8_0-K/q4_0-V @D128 ctx128k:real_frac 0.0(bit-identical to the shipped materialize) + 2.13× decode (14.72→31.33 tps). #644 full matrix @D128 32k multichunk DCA-on: all 10 pairsreal_frac 0.0— symmetric diagonal{q4_0,q4_1,q5_0,q5_1,q6_0,q8_0}K==V + hi-K/cheap-V{q8K/{q4,q5,q6}V, q6K/q4V}. Decode-gated (Q->ne[1] ≤ WS2_NQ_MAX=16, envWS2_DCA_SCALAR_KV); large-n_q prefill keeps the fast in-launch f16 lift (bug-720). D256/D512 scalar deferred (Gemma KV is turbo, already in-register). This unblocks P2.
P1 — auto KV-tier policy at boot (capacity without user knobs). — SHIPPED (#621, patch
0116). Given -c, free VRAM, and the model's KiB/token, pick the least compression that
keeps KV fully resident: f16 → q8/q8 → q8K/q4V → (last resort) q5K/q4V; never auto-pick below
the §2 quality bar for the detected class; spill only if even the floor doesn't fit (then
rolling-KV takes the excess). Anchors: in-binary auto-policy precedent llama-context.cpp
(#567 sparse-V auto), bounded-defaults precedent in buildServerArgs (#564); log the choice
at boot like the position window line. Small, additive, immediately useful for the
multi-session target.
Implementation: opencoti_auto_select_kv_tier (llama-kv-cache.cpp), run in the
llama_kv_cache ctor BEFORE any residency/allocation math consumes type_k/type_v. It
reuses the EXACT budget arithmetic of opencoti_compute_resident_window_cells (free −
already-resident − 1.5 GiB compute-reserve, capped by --vram-target) and the same per-layer
per-cell byte sum, then walks the ladder {f16, q8_0/q8_0, q8_0/q4_0, q5_1/q4_0} and picks the
first (least-compressing) rung whose whole-cache cost fits; floor if none fit. Strictly
opt-in (OPENCOTI_KV_AUTO_TIER=1) and a no-op — byte-identical — when: no GPU offload; the
user set -ctk/-ctv off the f16 default (override respected); iSWA (swa_type != NONE,
Gemma stays f16/resident per §3c); or f16 already fits. Announced at WARN with a
set -ctk/-ctv to override hint (mirrors #567). Host adapter opt-in: kvAutoTier config →
OPENCOTI_KV_AUTO_TIER env in buildServerArgs.
Validated (bs2, Qwen2.5-14B-1M Q8_0, -c 32768, RTX 6000): the ladder walks correctly as
--vram-target tightens — 20000 MiB → q8_0/q8_0 (KV 3264 MiB fits budget 3328), 19400 →
q8_0/q4_0 (2496 fits 2728), 18800 → q5_1/q4_0 (2016 fits 2128); every rung logged position window = FULLY RESIDENT … no host tail and returned the passkey needle cleanly. Config-
equivalence: auto q8_0/q8_0 output was byte-identical to an explicit -ctk q8_0 -ctv q8_0
boot, which emitted zero auto-tier WARN (opt-in path confirmed off).
P2 — position-axis mixed KV: f16 recent window ⊕ compressed VRAM tail. The natural extension once P0 lands, built entirely on existing seams — this is the "low-bit history ⊕ exact recent" HAL route, kept in VRAM:
- Storage: reuse the POSITION_WINDOW split (
llama-kv-cache.h:294-296get_k/v_window + get_k/v_tail accessors, per-layer budgetsheadinfer_window_cells, S0 #585) — but the tail tensor becomes a device buffer of quantized type instead of pinned-host f16. - Write path: tokens quantize once on eviction from the rolling f16 window, batched through
the existing GPU quantize-on-write kernels (
set-rows.cuk_set_rows_* family, #375/#380). - Read path: the two-region streaming FA op (
ggml.h:2481ggml_streaming_flash_attn_window) already merges window+tail with exact online-softmax combine; each region launches its own typed kernel, so mixed f16-window/quant-tail needs no new FA instances — only supports_op relaxation on the tail type. - Value over uniform quant: recent tokens (the ones agentic loops re-read hardest) stay exact; fresh-token writes skip the encode in the hot path; quality strictly ≥ uniform tier at equal tail type, opening the door to a q4-class or t8K/t4V tail (78→~60 KiB/tok) that uniform quant can't risk. Gate with KLD @256k + niah ladder vs uniform q8K/q4V.
- Ordering: only worth building after P0 — today the tail's quant kernel tax would dominate and mask the win.
P3 — per-layer type maps: rejected (probe evidence above). Revisit only with a genuinely new selection mechanism, per the #551 lesson.
6. Provenance
- Resident cells (§3b):
bs2:/srv/ml/opencoti-c1/hal-resident.sh→hal582-out/(per-cell logs +run.log), binaryllamafile.dualctxc-series, 2026-07-05. - A4B contrast (§3c):
bs2:/srv/ml/opencoti-c1/hal-resident-a4b.sh→hal582a4b-out/,google_gemma-4-26B-A4B-it-Q4_K_M.gguf(registry sha e718536f), chat endpoint, 2026-07-05. - 35B contrast (§3d):
bs2:/srv/ml/opencoti-c1/hal-resident-35b.sh→hal58235b-out/,Qwen3.6-35B-A3B-UD-Q6_K.gguf(registry sha 49935b04), chat endpoint, 2026-07-05. - Overspill cells (§3a):
bs2:/srv/ml/opencoti-c1/bs2-overspill.sh→overspill-out/, 2026-07-01 (VT=9600 simulating 24 GB). - Quality: context.md §5b-14B-1M-ctx (#581), §5b full matrix (#529), DCA niah matrix (#575).
- Probes:
.opencoti/hal/window_sim.py(locality), cross-layer K/V cosine probe (#582 2026-07-01, pgvectorproject_qwen_retrieval_distributed_hal). - Cliff curves: #583/#584 (3090 + bs2), PCIe reference
reference_pcie_bandwidth_hosts.