# Ornith-1.0-35B GGUF Quant KLD Probe

Date: 2026-06-27

Hardware: single RTX PRO 6000 Blackwell Max-Q 96GB, `tp=1`.

Backend: local `llama.cpp` CUDA server from `vendor/llama.cpp/build-cuda/bin/llama-server`.

## Method

This corrected probe compares each quant against the BF16 GGUF baseline using
approximate `KL(P_bf16 || P_quant)` over next-token top-64 distributions.

Prompts: first 32 rows of `data/eval/coding_sft_eval_messages.jsonl`, rendered
with the Ornith chat template, final assistant answer removed, and generation
prompt added.

API path: native llama.cpp `/completion` with `n_predict=1`, `temperature=-1`,
`n_probs=64`, and `cache_prompt=false`.

Matching: distributions are keyed by token ID, not token text.

Approximation: missing candidate top-K tokens receive
`candidate_tail_mass / (vocab_size - K)`, and one explicit tail bucket is
included. In this run, the BF16 top-64 distribution covered mean probability
mass 0.999965, so the top-64 approximation captured essentially all next-token
mass for these prompts. This is still a fast next-token quant-regression signal,
not a full-vocab or full-sequence evaluation.

BF16 baseline loaded VRAM: 66,118 MiB / 64.57 GiB.

## Results

| Quant | GGUF GiB | KLD profile VRAM GiB | Mean KLD nats | Mean KLD bits | P50 nats | P95 nats | Max nats | Top-1 match |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| Q3_K_M | 15.61 | 16.32 | 0.3620 | 0.5223 | 0.2548 | 1.1077 | 1.3730 | 27/32 (84.4%) |
| IQ4_XS | 17.64 | 19.34 | 0.1426 | 0.2057 | 0.0868 | 0.3195 | 0.6586 | 27/32 (84.4%) |
| Q4_K_M | 19.71 | 20.35 | 0.0864 | 0.1247 | 0.0379 | 0.2877 | 0.4503 | 29/32 (90.6%) |
| Q5_K_M | 23.03 | 23.61 | 0.0354 | 0.0510 | 0.0235 | 0.0943 | 0.2497 | 30/32 (93.8%) |
| Q6_K | 26.56 | 27.07 | 0.0165 | 0.0238 | 0.0092 | 0.0513 | 0.0586 | 32/32 (100.0%) |
| Q8_0 | 34.37 | 34.77 | 0.0108 | 0.0156 | 0.0052 | 0.0440 | 0.0590 | 31/32 (96.9%) |

## Interpretation

Q8_0 has the lowest mean and median KLD in this corrected top-64 probe. Q6_K is
nearly as close and has perfect top-1 agreement across all 32 prompts, making it
the best quality/footprint tradeoff among the higher-precision quants here.

Q5_K_M is a strong middle option: much lower drift than Q4_K_M and Q3_K_M while
still staying materially smaller than Q6_K and Q8_0. IQ4_XS sits between Q3_K_M
and Q4_K_M: materially lower KLD than Q3_K_M at a smaller footprint than Q4_K_M,
but not as close to BF16 as Q4_K_M. Q4_K_M remains the speed-focused serving
recommendation from throughput benchmarks, but it drifts more than Q5/Q6/Q8.
Q3_K_M is the lowest-footprint option and passed behavior checks, but it has the
largest KLD drift versus BF16 in this probe.

## Raw Artifacts

- `runs/kld-native-bf16-top64-32prompts.json`
- `runs/kld-native-q3-k-m-top64-32prompts.json`
- `runs/kld-native-q3-k-m-vs-bf16.json`
- `runs/kld-native-iq4-xs-top64-32prompts.json`
- `runs/kld-native-iq4-xs-vs-bf16.json`
- `runs/kld-native-q4-k-m-top64-32prompts.json`
- `runs/kld-native-q4-k-m-vs-bf16.json`
- `runs/kld-native-q5-k-m-top64-32prompts.json`
- `runs/kld-native-q5-k-m-vs-bf16.json`
- `runs/kld-native-q6-k-top64-32prompts.json`
- `runs/kld-native-q6-k-vs-bf16.json`
- `runs/kld-native-q8-0-top64-32prompts.json`
- `runs/kld-native-q8-0-vs-bf16.json`
- `runs/kld-native-vram.csv`