# Ornith-1.0-35B GGUF Quant Benchmarks

Date: 2026-06-27

Hardware: single RTX PRO 6000 Blackwell Max-Q 96GB, `tp=1` / one model copy per GPU.

Backend: local `llama.cpp` CUDA server from `vendor/llama.cpp/build-cuda/bin/llama-server`, launched through `scripts/serve_llamacpp_gpu1.sh`.

Benchmark profile:

```bash
CUDA_VISIBLE_DEVICES=1 \
HOST=127.0.0.1 PORT=<port> QUANT=<quant> \
CTX_SIZE=8192 PARALLEL=16 BATCH_SIZE=4096 UBATCH_SIZE=512 \
CONT_BATCHING=1 METRICS=1 \
scripts/serve_llamacpp_gpu1.sh
```

Important: with llama.cpp server, `-c` is split across `-np`, so this benchmark profile exposes `n_ctx = 512` per slot (`8192 / 16`). It is a short-context throughput profile for the fixed 256-token generation sweep. For a practical coding serving profile with 16 concurrent slots and about 8k tokens per slot, use `CTX_SIZE=131072 PARALLEL=16`.

Correctness gate: each quant was checked through the OpenAI-compatible `/v1/models`, `/v1/chat/completions`, and `/v1/completions` paths. The probe required coherent text plus regex matches for `def add(` and `return a + b`.

## Summary

All tested GGUF quants are usable through llama.cpp. vLLM's GGUF path remains unusable for this model family in our earlier probe: Q4_K_M loaded but produced corrupted/gibberish output, while the same GGUF was coherent through llama.cpp.

Q4_K_M is the best speed quant in this profile. Q3_K_M is a custom smaller quant and is usable, but it trades throughput for memory: about 20.8% smaller on disk and about 19.0% lower loaded VRAM than Q4_K_M, while trailing Q4_K_M at higher concurrency. IQ4_XS is a local middle-footprint quant: about 10.5% smaller on disk and about 9.2% lower loaded VRAM than Q4_K_M, with lower KLD drift than Q3_K_M but lower throughput than Q4_K_M. Q5_K_M is the closest higher-precision alternative and nearly ties Q4 at c16, but costs about 3.31 GiB more disk and about 2.99 GiB more loaded VRAM. Q6_K and Q8_0 add footprint without winning throughput in this profile.

Note: Q3_K_M was benchmarked after the serving script default changed to `REASONING=off`; the Q4_K_M through Q8_0 rows are preserved from the original sweep. The usable serving profile is `REASONING=off` because reasoning-on/auto can return empty final `content` on short coding requests.

MTP note: llama.cpp exposes `--spec-type draft-mtp`, but it requires a separate draft GGUF with NextN/MTP tensors. The released Ornith HF weights and GGUF files do not contain those tensors. This lab created an experimental bootstrap MTP draft by copying trunk layer 39 into `model.mtp.layers.0`, adding the missing MTP projection/norm tensors, converting with `convert_hf_to_gguf.py --mtp`, and quantizing the draft to IQ4_XS. It then trained a chain-corrected MTP draft from target hidden-state captures and exported Q6_K, Q5_K_M, and IQ4_XS draft GGUFs. The trained Q6_K draft reaches 67.22% draft-token acceptance, but the current llama.cpp profile is still slower than target-only decoding. See `benchmarks/llamacpp-iq4-xs-mtp-bootstrap-profile.md` and `benchmarks/llamacpp-iq4-xs-mtp-chaincorr-profile.md`.

## Footprint

| Quant | GGUF bytes | GGUF GiB | Loaded VRAM MiB | Loaded VRAM GiB | Correctness |
|---|---:|---:|---:|---:|---|
| Q3_K_M | 16,764,763,712 | 15.61 | 17,684 | 17.27 | pass |
| IQ4_XS | 18,939,311,680 | 17.64 | 19,806 | 19.34 | pass |
| Q4_K_M | 21,166,757,760 | 19.71 | 21,818 | 21.31 | pass |
| Q5_K_M | 24,729,130,848 | 23.03 | 25,238 | 24.65 | pass |
| Q6_K | 28,514,152,288 | 26.56 | 28,700 | 28.03 | pass |
| Q8_0 | 36,903,138,880 | 34.37 | 36,582 | 35.72 | pass |

Loaded VRAM was captured with `nvidia-smi` after model load on the benchmark profile.

## Quant Provenance

All files are GGUF artifacts for `deepreinforce-ai/Ornith-1.0-35B`.

| File | Quant | Provenance | Quantization detail |
|---|---|---|---|
| `ornith-1.0-35b-Q3_K_M.gguf` | Q3_K_M | Locally produced by this lab from the upstream BF16 GGUF | `llama-quantize ornith-1.0-35b-bf16.gguf ornith-1.0-35b-Q3_K_M.gguf Q3_K_M`; quantizer reported source 66,152.24 MiB / 16.01 BPW and quantized size 15,977.65 MiB / 3.87 BPW |
| `ornith-1.0-35b-IQ4_XS.gguf` | IQ4_XS | Locally produced by this lab from the upstream BF16 GGUF | `llama-quantize ornith-1.0-35b-bf16.gguf ornith-1.0-35b-IQ4_XS.gguf IQ4_XS 32`; no importance matrix; quantizer reported source 66,152.24 MiB / 16.01 BPW and quantized size 18,051.46 MiB / 4.37 BPW |
| `ornith-1.0-35b-mtp-bootstrap-layer39-IQ4_XS.gguf` | IQ4_XS draft MTP | Locally produced by this lab from the released HF safetensors | `build_bootstrap_mtp_checkpoint.py --source-layer 39 --eh-proj-init hidden`, then `convert_hf_to_gguf.py --mtp --outtype bf16`, then `llama-quantize ... IQ4_XS 32`; draft quantizer reported source 3,552.05 MiB / 16.00 BPW and quantized size 1,085.37 MiB / 4.89 BPW |
| `ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-Q6_K.gguf` | Q6_K draft MTP | Locally trained by this lab from target hidden-state captures, then converted to GGUF | Chain-corrected Qwen3.5 MoE MTP draft; `llama-quantize ... Q6_K`; source draft 8,388.17 MiB / 16.01 BPW; estimated draft load 3,176.45 MiB |
| `ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-Q5_K_M.gguf` | Q5_K_M draft MTP | Locally trained by this lab from target hidden-state captures, then converted to GGUF | Same trained draft as above, quantized with `llama-quantize ... Q5_K_M`; quantized size 3,090.01 MiB / 5.90 BPW; estimated draft load 2,885.59 MiB |
| `ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-IQ4_XS.gguf` | IQ4_XS draft MTP | Locally trained by this lab from target hidden-state captures, then converted to GGUF | Same trained draft as above, quantized with `llama-quantize ... IQ4_XS`; quantized size 2,374.93 MiB / 4.53 BPW; estimated draft load 2,246.29 MiB |
| `ornith-1.0-35b-Q4_K_M.gguf` | Q4_K_M | Mirrored from `deepreinforce-ai/Ornith-1.0-35B-GGUF`, then validated here | Upstream llama.cpp GGUF K-quant artifact; exact upstream quantization command was not republished in this repo |
| `ornith-1.0-35b-Q5_K_M.gguf` | Q5_K_M | Mirrored from `deepreinforce-ai/Ornith-1.0-35B-GGUF`, then validated here | Upstream llama.cpp GGUF K-quant artifact; exact upstream quantization command was not republished in this repo |
| `ornith-1.0-35b-Q6_K.gguf` | Q6_K | Mirrored from `deepreinforce-ai/Ornith-1.0-35B-GGUF`, then validated here | Upstream llama.cpp GGUF K-quant artifact; exact upstream quantization command was not republished in this repo |
| `ornith-1.0-35b-Q8_0.gguf` | Q8_0 | Mirrored from `deepreinforce-ai/Ornith-1.0-35B-GGUF`, then validated here | Upstream llama.cpp GGUF Q8_0 artifact; exact upstream quantization command was not republished in this repo |

## Throughput

Values are aggregate output tokens/second from `scripts/bench_openai.py --stream --max-tokens 256`; each row had 0 failed requests.

| Quant | c1 tok/s | c1 p95 TTFT ms | c4 tok/s | c4 p95 TTFT ms | c8 tok/s | c8 p95 TTFT ms | c16 tok/s | c16 p95 TTFT ms |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| Q3_K_M | 240.5 | 77.9 | 422.0 | 170.7 | 464.4 | 344.9 | 493.0 | 493.7 |
| IQ4_XS | 234.1 | 75.1 | 297.7 | 159.7 | 411.5 | 330.2 | 476.0 | 541.5 |
| Q4_K_M | 243.3 | 76.3 | 458.3 | 192.3 | 615.0 | 361.8 | 655.6 | 650.0 |
| Q5_K_M | 236.7 | 75.1 | 311.0 | 198.8 | 439.0 | 383.6 | 638.6 | 620.4 |
| Q6_K | 225.9 | 76.8 | 295.8 | 194.2 | 409.6 | 394.8 | 603.3 | 657.4 |
| Q8_0 | 208.5 | 76.9 | 281.5 | 190.8 | 405.8 | 389.1 | 601.4 | 725.8 |

## Experimental MTP Throughput

The target model for these rows is `ornith-1.0-35b-IQ4_XS.gguf`. The original
bootstrap MTP draft made `draft-mtp` loadable but had very low acceptance. The
latest trained chain-corrected MTP draft reaches the expected acceptance band,
but current llama.cpp scheduling is still slower than target-only decoding on
the c16 short-context throughput profile.

| Profile | Loaded VRAM GiB | c1 tok/s | c1 p95 TTFT ms | c4 tok/s | c4 p95 TTFT ms | c8 tok/s | c8 p95 TTFT ms | c16 tok/s | c16 p95 TTFT ms | Draft acceptance |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| IQ4_XS target only | 19.34 | 234.1 | 75.1 | 297.7 | 159.7 | 411.5 | 330.2 | 476.0 | 541.5 | n/a |
| IQ4_XS + BF16 MTP draft, n_max=3 | 25.12 | 95.5 | 84.3 | 97.0 | 179.2 | 102.1 | 408.1 | 108.2 | 726.8 | 0.47% |
| IQ4_XS + BF16 MTP draft, n_max=1 | 23.03 | 153.6 | 81.9 | 147.2 | 168.4 | 154.3 | 390.7 | 159.5 | 701.3 | 1.07% |
| IQ4_XS + IQ4_XS MTP draft, n_max=1 | 21.32 | 155.1 | 82.8 | 151.6 | 213.9 | 155.8 | 377.8 | 162.7 | 670.9 | 1.21% |
| IQ4_XS + Q6_K trained MTP draft, n_max=2 | 22.44 | n/t | n/t | n/t | n/t | n/t | n/t | 286.77 | 847.5 | 67.22% |
| IQ4_XS + Q5_K_M trained MTP draft, n_max=2 | 22.16 | n/t | n/t | n/t | n/t | n/t | n/t | 285.32 | 834.1 | 66.61% |
| IQ4_XS + IQ4_XS trained MTP draft, n_max=2 | 21.53 | n/t | n/t | n/t | n/t | n/t | n/t | 265.06 | 821.4 | 59.20% |
| IQ4_XS + Q6_K trained MTP draft, n_max=1 | 22.44 | n/t | n/t | n/t | n/t | n/t | n/t | 262.76 | 801.9 | 68.12% |

`n/t` means not tested in the latest trained-MTP loop; those rows were measured
only on the c16 profile used for the MTP fix loop. The refreshed IQ4_XS
target-only comparator from that loop was 482.68 tok/s at c16. The remaining MTP
fix is scheduler/runtime efficiency, not acceptance alone: the Q6_K draft
accepts 67.22% of speculative tokens, but draft and target verification are
still mostly sequential and host-synchronized.

## BF16 KLD Probe

Each quant was also compared against the upstream BF16 GGUF baseline with a
corrected native llama.cpp next-token top-64 `KL(P_bf16 || P_quant)` probe over
32 coding prompts. The probe used `/completion` with `n_predict=1`,
`temperature=-1`, `n_probs=64`, `cache_prompt=false`, and token-ID based
matching. This is a fast quant-regression signal, not a full-vocab or
full-sequence eval. See `benchmarks/kld-quant-vs-bf16-top64.md` for details.

| Quant | KLD profile VRAM GiB | Mean KLD nats | Mean KLD bits | P50 nats | P95 nats | Max nats | Top-1 match |
|---|---:|---:|---:|---:|---:|---:|---:|
| Q3_K_M | 16.32 | 0.3620 | 0.5223 | 0.2548 | 1.1077 | 1.3730 | 84.4% |
| IQ4_XS | 19.34 | 0.1426 | 0.2057 | 0.0868 | 0.3195 | 0.6586 | 84.4% |
| Q4_K_M | 20.35 | 0.0864 | 0.1247 | 0.0379 | 0.2877 | 0.4503 | 90.6% |
| Q5_K_M | 23.61 | 0.0354 | 0.0510 | 0.0235 | 0.0943 | 0.2497 | 93.8% |
| Q6_K | 27.07 | 0.0165 | 0.0238 | 0.0092 | 0.0513 | 0.0586 | 100.0% |
| Q8_0 | 34.77 | 0.0108 | 0.0156 | 0.0052 | 0.0440 | 0.0590 | 96.9% |

## Raw Artifacts

Correctness probes:

- `runs/behavior-llamacpp-q3-k-m-parallel16.json`
- `runs/gguf-materialization-q3-k-m.json`
- `runs/probe-llamacpp-iq4-xs-correctness.json`
- `runs/behavior-llamacpp-iq4-xs-parallel16.json`
- `runs/probe-llamacpp-iq4-xs-mtp-bootstrap-correctness.json`
- `runs/probe-llamacpp-iq4-xs-mtp-bootstrap-nmax1-correctness.json`
- `runs/probe-llamacpp-iq4-xs-mtp-bootstrap-iq4draft-nmax1-correctness.json`
- `runs/mtp-support-ornith-iq4-xs.json`
- `runs/probe-llamacpp-q4-correctness.json`
- `runs/probe-llamacpp-q5-correctness.json`
- `runs/probe-llamacpp-q6-correctness.json`
- `runs/probe-llamacpp-q8-correctness.json`

Benchmark JSONL:

- `runs/llamacpp-q3-k-m-c1-256.jsonl`
- `runs/llamacpp-q3-k-m-c4-256.jsonl`
- `runs/llamacpp-q3-k-m-c8-256.jsonl`
- `runs/llamacpp-q3-k-m-c16-256.jsonl`
- `runs/llamacpp-iq4-xs-c1-256.jsonl`
- `runs/llamacpp-iq4-xs-c4-256.jsonl`
- `runs/llamacpp-iq4-xs-c8-256.jsonl`
- `runs/llamacpp-iq4-xs-c16-256.jsonl`
- `runs/llamacpp-iq4-xs-mtp-bootstrap-c1-256.jsonl`
- `runs/llamacpp-iq4-xs-mtp-bootstrap-c4-256.jsonl`
- `runs/llamacpp-iq4-xs-mtp-bootstrap-c8-256.jsonl`
- `runs/llamacpp-iq4-xs-mtp-bootstrap-c16-256.jsonl`
- `runs/llamacpp-iq4-xs-mtp-bootstrap-nmax1-c1-256.jsonl`
- `runs/llamacpp-iq4-xs-mtp-bootstrap-nmax1-c4-256.jsonl`
- `runs/llamacpp-iq4-xs-mtp-bootstrap-nmax1-c8-256.jsonl`
- `runs/llamacpp-iq4-xs-mtp-bootstrap-nmax1-c16-256.jsonl`
- `runs/llamacpp-iq4-xs-mtp-bootstrap-iq4draft-nmax1-c1-256.jsonl`
- `runs/llamacpp-iq4-xs-mtp-bootstrap-iq4draft-nmax1-c4-256.jsonl`
- `runs/llamacpp-iq4-xs-mtp-bootstrap-iq4draft-nmax1-c8-256.jsonl`
- `runs/llamacpp-iq4-xs-mtp-bootstrap-iq4draft-nmax1-c16-256.jsonl`
- `runs/llamacpp-iq4-xs-target-only-refresh-c16-256.jsonl`
- `runs/llamacpp-iq4-xs-mtp-nmax2-fast-backend-topk1-c16-128-temp0.jsonl`
- `runs/llamacpp-iq4-xs-mtp-nmax2-draftq5-fast-topk1-c16-128-temp0.jsonl`
- `runs/llamacpp-iq4-xs-mtp-nmax2-draftiq4-fast-topk1-c16-128-temp0.jsonl`
- `runs/llamacpp-iq4-xs-mtp-nmax1-q6-fast-topk1-c16-128-temp0.jsonl`
- `runs/llamacpp-q4-c1-256.jsonl`
- `runs/llamacpp-q4-c4-256.jsonl`
- `runs/llamacpp-q4-c8-256.jsonl`
- `runs/llamacpp-q4-c16-256.jsonl`
- `runs/llamacpp-q5-c1-256.jsonl`
- `runs/llamacpp-q5-c4-256.jsonl`
- `runs/llamacpp-q5-c8-256.jsonl`
- `runs/llamacpp-q5-c16-256.jsonl`
- `runs/llamacpp-q6-c1-256.jsonl`
- `runs/llamacpp-q6-c4-256.jsonl`
- `runs/llamacpp-q6-c8-256.jsonl`
- `runs/llamacpp-q6-c16-256.jsonl`
- `runs/llamacpp-q8-c1-256.jsonl`
- `runs/llamacpp-q8-c4-256.jsonl`
- `runs/llamacpp-q8-c8-256.jsonl`
- `runs/llamacpp-q8-c16-256.jsonl`

Corrected native top-64 KLD JSON:

- `runs/kld-native-bf16-top64-32prompts.json`
- `runs/kld-native-q3-k-m-top64-32prompts.json`
- `runs/kld-native-q3-k-m-vs-bf16.json`
- `runs/kld-native-iq4-xs-top64-32prompts.json`
- `runs/kld-native-iq4-xs-vs-bf16.json`
- `runs/kld-native-q4-k-m-top64-32prompts.json`
- `runs/kld-native-q4-k-m-vs-bf16.json`
- `runs/kld-native-q5-k-m-top64-32prompts.json`
- `runs/kld-native-q5-k-m-vs-bf16.json`
- `runs/kld-native-q6-k-top64-32prompts.json`
- `runs/kld-native-q6-k-vs-bf16.json`
- `runs/kld-native-q8-0-top64-32prompts.json`
- `runs/kld-native-q8-0-vs-bf16.json`
- `runs/kld-native-vram.csv`

## Recommended Commands

Fast short-context throughput profile:

```bash
QUANT=Q4_K_M PORT=8002 CTX_SIZE=8192 PARALLEL=16 \
  scripts/serve_llamacpp_gpu0.sh
```

Coding-serving profile with 16 slots and about 8k context per slot:

```bash
QUANT=Q4_K_M PORT=8002 CTX_SIZE=131072 PARALLEL=16 \
  scripts/serve_llamacpp_gpu0.sh
```

Use `scripts/serve_llamacpp_gpu1.sh` for the second physical GPU. The scripts also support `MODEL_PATH`, `ALIAS`, `BATCH_SIZE`, `UBATCH_SIZE`, `CONT_BATCHING`, `METRICS`, `BACKEND_SAMPLING`, `SPEC_TYPE`, `SPEC_DRAFT_MODEL`, `SPEC_DRAFT_HF`, `SPEC_DRAFT_NGL`, `SPEC_DRAFT_N_MAX`, `SPEC_DRAFT_N_MIN`, `SPEC_DRAFT_P_MIN`, `SPEC_DRAFT_DEVICE`, `SPEC_DRAFT_BACKEND_SAMPLING`, and any of the tested `QUANT` values.

Lowest-memory custom quant:

```bash
QUANT=Q3_K_M PORT=8002 CTX_SIZE=8192 PARALLEL=16 \
  scripts/serve_llamacpp_gpu0.sh
```

Middle-footprint custom quant:

```bash
QUANT=IQ4_XS PORT=8002 CTX_SIZE=8192 PARALLEL=16 \
  scripts/serve_llamacpp_gpu0.sh
```

Experimental trained-MTP profile:

```bash
SPEC_DRAFT_BACKEND_SAMPLING=1 \
LLAMA_MTP_FAST_BACKEND_SAMPLE=1 \
LLAMA_MTP_DRAFT_TOP_K=1 LLAMA_MTP_DRAFT_TOP_P=1 LLAMA_MTP_DRAFT_TEMP=1 \
QUANT=IQ4_XS PORT=8002 CTX_SIZE=8192 PARALLEL=16 \
SPEC_TYPE=draft-mtp \
SPEC_DRAFT_MODEL=ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-Q6_K.gguf \
SPEC_DRAFT_N_MAX=2 SPEC_DRAFT_N_MIN=0 \
  scripts/serve_llamacpp_gpu0.sh
```