--- license: mit license_link: https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B/blob/main/LICENSE base_model: deepreinforce-ai/Ornith-1.0-35B pipeline_tag: text-generation library_name: gguf tags: - gguf - llama-cpp - quantized - text-generation - coding - mtp - qwen3_5_moe - single-gpu - tp1 --- # Ornith-1.0-35B — GGUF (llama.cpp, single-GPU `tp=1`) Single-GPU llama.cpp GGUF package for [`deepreinforce-ai/Ornith-1.0-35B`](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B). The supported serving policy is `tp=1` — one model copy per GPU. Multi-GPU tensor-parallel serving is intentionally out of scope for this version. This release ships six body quants (Q3_K_M → Q8_0) plus an **integrated IQ4_XS-MTP graft** that adds a native multi-token-prediction (MTP) draft head for low-concurrency speculative decode. Quant quality is measured against the upstream BF16 GGUF with a native llama.cpp next-token top-64 KL-divergence probe over 32 coding prompts. ## TL;DR — pick an artifact | Use case | Recommended artifact | Key numbers | |---|---|---| | Default serving speed | `ornith-1.0-35b-Q4_K_M.gguf` | 19.71 GiB on disk, 21.31 GiB loaded VRAM, 243.3 tok/s c1, 655.6 tok/s c16 | | Lowest memory | `ornith-1.0-35b-Q3_K_M.gguf` | 15.61 GiB on disk, 17.27 GiB loaded VRAM, 240.5 tok/s c1, 493.0 tok/s c16 | | Middle footprint | `ornith-1.0-35b-IQ4_XS.gguf` | 17.64 GiB on disk, 19.34 GiB loaded VRAM, 0.1426 mean top-64 KLD nats | | Highest fidelity / footprint | `ornith-1.0-35b-Q6_K.gguf` | 26.56 GiB on disk, 28.03 GiB loaded VRAM, 0.0165 mean top-64 KLD nats, 32/32 top-1 | | Native low-concurrency MTP | `ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf` | 19.6 GB decimal, **0.0731 mean top-64 KLD (29/32)**, c1/128 adaptive MTP 319.53 tok/s, acceptance `(0.953, 0.865)` | > **MTP graft note:** active MTP is byte-for-byte identical to target-only at the > next-token (API-visible) level, and gives ~1.3× single-stream decode. It is > **not** bit-exact to target-only over long deterministic sequences — see > [MTP](#multi-token-prediction-mtp). Use target-only for strict reproduction, > MTP for throughput on open-ended work. ## Quantization & fidelity Fidelity is `KL(P_bf16 || P_candidate)` over next-token top-64 distributions, 32 coding prompts, `n_predict=1, temperature=-1, n_probs=64`, keyed by token ID. Lower KLD and higher greedy top-1 agreement are better. ### Primary fidelity table | Artifact | Mean KLD (nats) | Greedy top-1 † | |---|---:|---:| | `Q8_0` | 0.0108 | 31/32 (96.9%) | | `Q6_K` | 0.0165 | 32/32 (100.0%) | | `Q5_K_M` | 0.0354 | 30/32 (93.8%) | | **`IQ4_XS-MTP` (graft)** | **0.0731382442** | **29/32 (90.6%)** | | `Q4_K_M` | 0.0864 | 29/32 (90.6%) | | `IQ4_XS` | 0.1426 | 27/32 (84.4%) | | `Q3_K_M` | 0.3620 | 27/32 (84.4%) | ![Next-token fidelity ladder: mean top-64 KLD vs BF16 for the six body quants plus the IQ4_XS-MTP graft, lower is better.](assets/02_fidelity_ladder.png) *Fidelity ladder — the integrated IQ4_XS-MTP graft (purple) lands between Q5_K_M and Q4_K_M, i.e. better next-token fidelity than its Q4_K_M neighbor.* > **† Top-1 vs mean KLD are different axes.** *Top-1* counts how often the single > most-likely (argmax) token matches BF16's, over the 32 prompts; *mean KLD* > measures divergence across the full top-64 distribution. Two quants can share a > top-1 count yet differ a lot in KLD — e.g. `Q3_K_M` and `IQ4_XS` are both > 27/32 on top-1, but `Q3_K_M`'s distribution diverges ~2.5× more (0.3620 vs > 0.1426 nats). Neither value is a typo. - The **integrated IQ4_XS-MTP graft is 48.70% lower mean KLD than the plain IQ4_XS body** (0.0731382442 vs 0.1425748206), and improves greedy top-1 from 27/32 to 29/32. - **Active `draft-mtp` ≡ target-only at the next-token level:** their top-64 next-token distributions are identical (KLD 0.0, 32/32) for the one-token API-visible probe. ### Body-quant distribution detail (secondary) Full per-quant distribution from the corrected top-64 probe (same 32 prompts). `Mean / P50 / P95 / Max` are explicit nats statistics of the per-prompt top-64 KLD; in this run the BF16 top-64 distribution captured mean probability mass 0.999965, so the top-64 approximation captured essentially all next-token mass. | Quant | GGUF GiB | KLD-probe VRAM GiB | Mean nats | P50 nats | P95 nats | Max nats | Top-1 † | |---|---:|---:|---:|---:|---:|---:|---:| | Q3_K_M | 15.61 | 16.32 | 0.3620 | 0.2548 | 1.1077 | 1.3730 | 27/32 (84.4%) | | IQ4_XS | 17.64 | 19.34 | 0.1426 | 0.0868 | 0.3195 | 0.6586 | 27/32 (84.4%) | | Q4_K_M | 19.71 | 20.35 | 0.0864 | 0.0379 | 0.2877 | 0.4503 | 29/32 (90.6%) | | Q5_K_M | 23.03 | 23.61 | 0.0354 | 0.0235 | 0.0943 | 0.2497 | 30/32 (93.8%) | | Q6_K | 26.56 | 27.07 | 0.0165 | 0.0092 | 0.0513 | 0.0586 | 32/32 (100.0%) | | Q8_0 | 34.37 | 34.77 | 0.0108 | 0.0052 | 0.0440 | 0.0590 | 31/32 (96.9%) | The IQ4_XS-MTP graft uses the same probe on a different breakpoint context; its distribution detail (`P50 0.0492811974 / P95 0.1593743642 / Max 0.3348258511`) is in the [MTP](#multi-token-prediction-mtp) section. Source: [benchmarks/kld-quant-vs-bf16-top64.md](benchmarks/kld-quant-vs-bf16-top64.md). ## Serving performance Aggregate decode throughput and p95 time-to-first-token across concurrency, llama.cpp `tp=1`, from `scripts/bench_openai.py --stream --max-tokens 256` (0 failed requests per row). Q4_K_M is the speed pick; Q5_K_M nearly ties it at c16. | Quant | c1 tok/s | c1 p95 TTFT ms | c4 tok/s | c4 p95 TTFT ms | c8 tok/s | c8 p95 TTFT ms | c16 tok/s | c16 p95 TTFT ms | |---|---:|---:|---:|---:|---:|---:|---:|---:| | Q3_K_M | 240.5 | 77.9 | 422.0 | 170.7 | 464.4 | 344.9 | 493.0 | 493.7 | | IQ4_XS | 234.1 | 75.1 | 297.7 | 159.7 | 411.5 | 330.2 | 476.0 | 541.5 | | Q4_K_M | 243.3 | 76.3 | 458.3 | 192.3 | 615.0 | 361.8 | 655.6 | 650.0 | | Q5_K_M | 236.7 | 75.1 | 311.0 | 198.8 | 439.0 | 383.6 | 638.6 | 620.4 | | Q6_K | 225.9 | 76.8 | 295.8 | 194.2 | 409.6 | 394.8 | 603.3 | 657.4 | | Q8_0 | 208.5 | 76.9 | 281.5 | 190.8 | 405.8 | 389.1 | 601.4 | 725.8 | ![Aggregate decode throughput vs concurrency (c1–c16) for the six quants; Q4_K_M leads at c16.](assets/03_throughput_tps.png) ![p95 time-to-first-token vs concurrency (c1–c16) for the six quants; lower is better.](assets/04_ttft_p95.png) > **Profile note:** the table above is a **short-context** profile — `CTX_SIZE=8192 > PARALLEL=16` exposes `n_ctx = 512`/slot, with ~23-token prompts and 256-token > generations. Source: [benchmarks/llamacpp-quant-benchmarks.md](benchmarks/llamacpp-quant-benchmarks.md). > For how prefill latency scales with prompt length, see long-context TTFT below. ### Long-context TTFT (single stream) p95 time-to-first-token vs prompt/context length, single GPU `tp=1`, single stream (`CTX_SIZE=131072 PARALLEL=1`, `n_ctx/slot=131072`, exact prompt token counts, 140 rows/quant, 0 failures). | Context tokens | Q4_K_M p50/p95 ms | Q4_K_M tok/s | IQ4_XS p50/p95 ms | IQ4_XS tok/s | MTP-graft p50/p95 ms | MTP-graft tok/s | |---:|---:|---:|---:|---:|---:|---:| | 512 | 91.8 / 94.1 | 188.3 | 87.1 / 88.7 | 177.4 | 86.0 / 87.8 | 184.0 | | 1024 | 169.6 / 172.2 | 145.8 | 159.9 / 161.6 | 139.9 | 157.6 / 159.0 | 155.9 | | 2048 | 341.8 / 346.1 | 97.9 | 316.1 / 318.7 | 112.0 | 307.6 / 308.9 | 113.9 | | 4096 | 699.2 / 702.2 | 67.2 | 647.0 / 653.4 | 70.7 | 625.7 / 631.3 | 69.0 | | 8192 | 1447.5 / 1458.0 | 33.7 | 1342.9 / 1348.5 | 39.8 | 1297.4 / 1304.0 | 40.9 | | 16384 | 3009.7 / 3030.4 | 19.5 | 2793.2 / 2804.0 | 20.9 | 2709.8 / 2737.3 | 21.5 | | 32768 | 6302.4 / 6313.0 | 9.7 | 5829.9 / 5853.1 | 10.5 | 5673.5 / 5696.6 | 10.7 | ![Long-context p95 TTFT vs prompt length (512–32768 tokens) for Q4_K_M, IQ4_XS, and the MTP graft; single stream, tp=1.](assets/07_longctx_ttft.png) p95 TTFT rises with prompt length (prefill cost) to ~6.3 s at 32k tokens for Q4_K_M; the IQ4_XS body and IQ4_XS-MTP graft prefill slightly faster at every length. Decode throughput falls from ~180–190 tok/s at 512 tokens to ~10 tok/s at 32k as the KV cache grows. Source: [benchmarks/llamacpp-longctx-ttft.md](benchmarks/llamacpp-longctx-ttft.md). ## Multi-token prediction (MTP) The integrated `IQ4_XS-MTP-graft-headQ6` artifact carries a native MTP draft head on the IQ4_XS body. The release recommendation is **adaptive**: use MTP for low-concurrency / single-user requests and keep `LLAMA_SPEC_MAX_DRAFTING_SLOTS=1` so saturated batches fall back to target-only throughput. The MTP catalog validates the published MTP profiles. ### Next-token quality Active MTP **does not change** the one-token, API-visible next-token distribution. Mean BF16 KLD is **0.0731382442 nats** over the 32-prompt top-64 probe (P50 0.0492811974 / P95 0.1593743642 / Max 0.3348258511), top-1 29/32. The active `draft-mtp` vs target-only graft next-token KLD is **0.0 (32/32 identical)**. ### Sequence-level behavior A deterministic **8 prompt × 64 token** probe (`n_predict=64, temperature=-1, n_probs=64`) compares the same GGUF running active native `draft-mtp` against target-only. | Runtime | Client agg tok/s | Server decode tok/s | Draft acceptance | Exact 64-tok seqs | Token-position match | |---|---:|---|---|---:|---| | Target-only graft | 172.57 | ~210 (per-request timing) | n/a | 8/8 baseline | n/a | | Fast active `draft-mtp`, `n_max=2` | 233.81 | 325.70 (/metrics) | 310/378 = 82.01%; per-position `(0.884, 0.747)` | 6/8 | 478/512 = 93.36% | | `LLAMA_SPEC_VERIFY_SEQUENTIAL=1` | 164.34 | 204.88 (/metrics) | 313/375 = 83.47%; per-position `(0.884, 0.772)` | 5/8 | 455/512 = 88.87% | ![MTP single-stream decode speedup on the deterministic 8x64 probe: target-only vs active MTP graft, client and server decode.](assets/06_mtp_tps.png) - Fast active MTP gives **~1.35× client throughput** (233.81 vs 172.57) and server decode 325.70 vs ~210, with **82.01%** draft acceptance. - It matches **6/8** sequences exactly and **478/512 = 93.36%** of token positions; the two non-exact sequences first diverge at positions **60** and **34**. - The `LLAMA_SPEC_VERIFY_SEQUENTIAL=1` variant is both **slower** (164.34 / 204.88) **and** matches less (5/8, 88.87%; divergences at 25, 43, 60) — i.e. the fast verifier is here both faster and more target-matching. > **Note:** MTP self-speculation is **not bit-exact to target-only over long > deterministic generations**. For workloads that require exact target > reproduction, run target-only; for throughput on open-ended work, MTP wins. > MTP is a single-user / low-concurrency win — it is not faster on saturated > batches, which is why the adaptive throttle falls back to target-only. ### Sequence logprob comparability (API limitation) llama.cpp omits candidate `top_logprobs` for most accepted speculative tokens, so a naive "sequence KLD" looks **enormous** — that is an **API/logprob-availability artifact, not model quality**. On the comparable subset where the emitted token also matches, KLD is **≈ 0.001 nats** (near zero): | Runtime | Paired positions | Comparable top-logprobs | Missing | Comparable same-token mean KLD (nats) | Max (nats) | |---|---:|---:|---:|---:|---:| | Fast active `draft-mtp` vs target-only | 512 | 12 | 500 | 0.0011215739 | 0.0040239103 | | Sequential verifier vs target-only | 512 | 10 | 502 | 0.0012293172 | 0.0040239103 | Full sequence KLD cannot be computed without changing the runtime to return target verifier logprobs for accepted speculative tokens. ### Draft-head distillation provenance The meaningful KL-style metric for a standalone MTP draft head is **teacher KL against cached target hidden states**, not standard base-model next-token KLD. The draft head was trained and **measurably improved** across checkpoints — teacher KL fell 1.92 → 1.50 → 0.76 and teacher argmax top-1 rose 0.54 → 0.69 → 0.89. | Draft checkpoint | Tokens | Windows | Teacher KL | Teacher argmax top-1 | Teacher argmax top-5 | Gold top-1 | |---|---:|---:|---:|---:|---:|---:| | `mtp-distill-kl-step500` | 22,637 | 128 | 1.9222025748 | 0.5360250917 | 0.7881786456 | 0.4425939833 | | `snapshots-iq4-live-accepted-a/step-1000` | 58,754 | 413 | 1.5038724942 | 0.6876808388 | 0.8368451510 | 0.6764816013 | | `snapshots-iq4-live-allrows-b/step-1000` | 32,264 | 2,048 | 0.7591610373 | 0.8908070915 | 0.9389722291 | 0.8076183982 | ![MTP draft-head distillation progress: teacher KL falls (left) while top-1 agreement rises (right) across three checkpoints.](assets/05_mtp_draft_head_distill.png) *Teacher KL is measured against cached target hidden states (not base next-token KLD). These training artifacts live under `artifacts/mtp/` locally and are referenced by name as provenance — they are not committed in this HF repo.* ## Run with llama.cpp Serve a body quant on GPU0 (`tp=1`): ```bash QUANT=Q4_K_M PORT=8000 CTX_SIZE=8192 PARALLEL=16 REASONING=off \ scripts/serve_llamacpp_gpu0.sh ``` Serve the integrated MTP graft for low-concurrency speculative decode (keep adaptive throttling so saturated batches fall back to target-only): ```bash QUANT=IQ4_XS-MTP-graft-headQ6 PORT=8000 CTX_SIZE=8192 PARALLEL=1 \ CACHE_RAM=0 REASONING=off LLAMA_SPEC_MAX_DRAFTING_SLOTS=1 \ scripts/serve_llamacpp_gpu0.sh ``` `REASONING=off` is the default because the model otherwise spends simple coding prompts in `reasoning_content` before producing final `content`. See [benchmarks/llamacpp-q4-reasoning-off-fix.md](benchmarks/llamacpp-q4-reasoning-off-fix.md). Serving / quant / MTP profile catalogs: [configs/serving_profiles.yaml](configs/serving_profiles.yaml), [configs/quant_artifacts.yaml](configs/quant_artifacts.yaml), [configs/mtp_profiles.yaml](configs/mtp_profiles.yaml). ## Benchmark environment **Hardware:** NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 97,887 MiB VRAM, single GPU (`tp=1`). Driver 580.159.03, CUDA runtime reported by driver 13.0. **Dedicated MTP KLD eval config (2026-06-28):** local llama.cpp CUDA server, single **RTX PRO 6000 Blackwell 96GB**, `tp=1`, `CTX_SIZE=8192`, `PARALLEL=1`, `CACHE_RAM=0`, `REASONING=off`. - **llama.cpp build/commit:** `050ee92d04c2e1f639025786dea701c70e7d4204` (pinned for the long-context TTFT run; earlier KLD/throughput sweeps were not separately pinned). - **Reasoning mode:** all benchmarks in this card ran with `REASONING=off`. ## Provenance & reproducibility - **Figures:** regenerate with [`scripts/make_charts.py`](scripts/make_charts.py) → `assets/02_fidelity_ladder.png`, `assets/03_throughput_tps.png`, `assets/04_ttft_p95.png`, `assets/05_mtp_draft_head_distill.png`, `assets/06_mtp_tps.png`. - **Serving throughput / TTFT:** [benchmarks/llamacpp-quant-benchmarks.md](benchmarks/llamacpp-quant-benchmarks.md). - **Body-quant fidelity:** [benchmarks/kld-quant-vs-bf16-top64.md](benchmarks/kld-quant-vs-bf16-top64.md). - **Dedicated MTP KLD / sequence eval (2026-06-28):** [benchmarks/mtp-kld-eval-2026-06-28.md](benchmarks/mtp-kld-eval-2026-06-28.md) and [benchmarks/mtp-dedicated-kld-evaluation.md](benchmarks/mtp-dedicated-kld-evaluation.md), with raw evidence under [benchmarks/raw/](benchmarks/raw/). Includes the sequence-level probe and its logprob-availability limitation (full sequence KLD is not computable from the current llama.cpp API response). - **Adaptive MTP serving profile:** [benchmarks/llamacpp-iq4-xs-mtp-graft-adaptive-profile.md](benchmarks/llamacpp-iq4-xs-mtp-graft-adaptive-profile.md). - **Long-context TTFT:** [benchmarks/llamacpp-longctx-ttft.md](benchmarks/llamacpp-longctx-ttft.md).