# Ornith-1.0-35B GGUF Release Report

Date: 2026-06-28

Hardware for reported GPU numbers: one NVIDIA RTX PRO 6000 Blackwell Max-Q
96GB, `tp=1`, llama.cpp CUDA server. The release policy is single-GPU serving
only: one model copy per GPU, no tensor-parallel serving profile.

## Executive Summary

This release turns Ornith-1.0-35B into a practical llama.cpp GGUF stack with
validated serving profiles, locally produced smaller quants, quantitative KLD
quality checks, and a working native MTP path.

Recommended use:

| Use case | Artifact | Why |
|---|---|---|
| Default serving speed | `ornith-1.0-35b-Q4_K_M.gguf` | Fastest validated c16 profile, 655.6 tok/s, 21.31 GiB loaded VRAM |
| Lowest memory | `ornith-1.0-35b-Q3_K_M.gguf` | 15.61 GiB on disk, 17.27 GiB loaded VRAM, behavior gate passed |
| Middle footprint | `ornith-1.0-35b-IQ4_XS.gguf` | 17.64 GiB on disk, lower KLD drift than Q3_K_M |
| Quality/footprint | `ornith-1.0-35b-Q6_K.gguf` | 0.0165 mean top-64 KLD nats and 32/32 top-1 match |
| Native low-concurrency MTP | `ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf` | c1 MTP speedup with high acceptance; adaptive policy avoids saturated c16 loss |

Key result: Q4_K_M is the production default. The integrated IQ4_XS-MTP graft is
the MTP release candidate for low-concurrency/single-user experiments. Always-on
MTP is not recommended for saturated c16 server traffic because target-only
decoding already saturates the GPU more efficiently.

## Quantization Work

### What Was Built

Two main-body GGUF quants were produced locally from the upstream BF16 GGUF:

| Artifact | Quant | Quantization detail |
|---|---|---|
| `ornith-1.0-35b-Q3_K_M.gguf` | Q3_K_M | `llama-quantize ornith-1.0-35b-bf16.gguf ornith-1.0-35b-Q3_K_M.gguf Q3_K_M`; source 66,152.24 MiB / 16.01 BPW, output 15,977.65 MiB / 3.87 BPW |
| `ornith-1.0-35b-IQ4_XS.gguf` | IQ4_XS | `llama-quantize ornith-1.0-35b-bf16.gguf ornith-1.0-35b-IQ4_XS.gguf IQ4_XS 32`; no importance matrix; source 66,152.24 MiB / 16.01 BPW, output 18,051.46 MiB / 4.37 BPW |

The existing upstream GGUFs were mirrored and validated:

| Artifact | Quant | Release status |
|---|---|---|
| `ornith-1.0-35b-Q4_K_M.gguf` | Q4_K_M | validated, production default |
| `ornith-1.0-35b-Q5_K_M.gguf` | Q5_K_M | validated, higher-quality mid option |
| `ornith-1.0-35b-Q6_K.gguf` | Q6_K | validated, quality/footprint option |
| `ornith-1.0-35b-Q8_0.gguf` | Q8_0 | validated, closest to BF16 by mean KLD |

MTP-specific artifacts were also produced or published:

| Artifact | Type | Purpose |
|---|---|---|
| `ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf` | integrated target + MTP block | native `draft-mtp` low-concurrency profile |
| `ornith-1.0-35b-mtp-bootstrap-layer39-IQ4_XS.gguf` | bootstrap external draft | proved MTP plumbing, not recommended |
| `ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-Q6_K.gguf` | trained external draft | MTP development and regression testing |
| `ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-Q5_K_M.gguf` | trained external draft | smaller draft variant |
| `ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-IQ4_XS.gguf` | trained external draft | smallest trained draft, lower acceptance |

### Footprint Chart

Loaded VRAM is from the short-context throughput profile
`CTX_SIZE=8192 PARALLEL=16`.

| Quant | GGUF GiB | Loaded VRAM GiB | Disk vs Q4_K_M | VRAM vs Q4_K_M |
|---|---:|---:|---:|---:|
| Q3_K_M | 15.61 | 17.27 | -20.8% | -19.0% |
| IQ4_XS | 17.64 | 19.34 | -10.5% | -9.2% |
| Q4_K_M | 19.71 | 21.31 | baseline | baseline |
| Q5_K_M | 23.03 | 24.65 | +16.8% | +15.7% |
| Q6_K | 26.56 | 28.03 | +34.8% | +31.5% |
| Q8_0 | 34.37 | 35.72 | +74.4% | +67.6% |

Footprint bar, loaded VRAM:

| Quant | Loaded VRAM chart |
|---|---|
| Q3_K_M | `################# 17.27 GiB` |
| IQ4_XS | `################### 19.34 GiB` |
| Q4_K_M | `##################### 21.31 GiB` |
| Q5_K_M | `######################### 24.65 GiB` |
| Q6_K | `############################ 28.03 GiB` |
| Q8_0 | `#################################### 35.72 GiB` |

### Throughput Chart

Aggregate output tokens/second from `scripts/bench_openai.py --stream
--max-tokens 256`; all rows had 0 failed requests.

| Quant | c1 tok/s | c4 tok/s | c8 tok/s | c16 tok/s | c16 p95 TTFT ms |
|---|---:|---:|---:|---:|---:|
| Q3_K_M | 240.5 | 422.0 | 464.4 | 493.0 | 493.7 |
| IQ4_XS | 234.1 | 297.7 | 411.5 | 476.0 | 541.5 |
| Q4_K_M | 243.3 | 458.3 | 615.0 | 655.6 | 650.0 |
| Q5_K_M | 236.7 | 311.0 | 439.0 | 638.6 | 620.4 |
| Q6_K | 225.9 | 295.8 | 409.6 | 603.3 | 657.4 |
| Q8_0 | 208.5 | 281.5 | 405.8 | 601.4 | 725.8 |

c16 throughput bar:

| Quant | c16 tok/s chart |
|---|---|
| Q4_K_M | `################################################################ 655.6` |
| Q5_K_M | `############################################################## 638.6` |
| Q6_K | `########################################################### 603.3` |
| Q8_0 | `########################################################### 601.4` |
| Q3_K_M | `################################################ 493.0` |
| IQ4_XS | `############################################### 476.0` |

### KLD Quality Chart

KLD method: next-token top-64 approximate `KL(P_bf16 || P_quant)` over 32
coding prompts, native llama.cpp `/completion`, `n_predict=1`, `temperature=-1`,
`n_probs=64`, token-ID matching. Lower is better.

| Quant | Mean KLD nats | Mean KLD bits | P95 nats | Max nats | Top-1 match |
|---|---:|---:|---:|---:|---:|
| Q3_K_M | 0.3620 | 0.5223 | 1.1077 | 1.3730 | 27/32 (84.4%) |
| IQ4_XS | 0.1426 | 0.2057 | 0.3195 | 0.6586 | 27/32 (84.4%) |
| Q4_K_M | 0.0864 | 0.1247 | 0.2877 | 0.4503 | 29/32 (90.6%) |
| Q5_K_M | 0.0354 | 0.0510 | 0.0943 | 0.2497 | 30/32 (93.8%) |
| Q6_K | 0.0165 | 0.0238 | 0.0513 | 0.0586 | 32/32 (100.0%) |
| Q8_0 | 0.0108 | 0.0156 | 0.0440 | 0.0590 | 31/32 (96.9%) |

Mean KLD drift bar, lower is better:

| Quant | Mean KLD chart |
|---|---|
| Q8_0 | `# 0.0108` |
| Q6_K | `## 0.0165` |
| Q5_K_M | `#### 0.0354` |
| Q4_K_M | `######### 0.0864` |
| IQ4_XS | `############## 0.1426` |
| Q3_K_M | `#################################### 0.3620` |

Interpretation:

- Q4_K_M is the speed release target.
- Q3_K_M is valid for minimum memory, but has the largest KLD drift.
- IQ4_XS is useful as a middle-footprint body and as the MTP graft base.
- Q5_K_M, Q6_K, and Q8_0 improve distribution fidelity but cost more VRAM.

### SWE-bench Agent Scores

Full report: `benchmarks/swebench-agent-evals.md`.

The full 100-task SWE-bench Verified run used Q6_K with reasoning enabled,
`temperature=1.0`, `top_p=1.0`, and no explicit thinking budget:

| Profile | Dataset slice | Reasoning | Resolved | Score |
|---|---|---|---:|---:|
| Q6_K GGUF | `SWE-bench/SWE-bench_Verified` test `0:100` | on | 69/100 | 69.0% |

The multi-quant sweep used `MariusHobbhahn/swe-bench-verified-mini`, split
`test`, 50 tasks:

| Quant/profile | Resolved | Score | Completed | Empty patch | Errors |
|---|---:|---:|---:|---:|---:|
| Q6_K | 33/50 | 66.0% | 48/50 | 2 | 0 |
| Q3_K_M | 32/50 | 64.0% | 47/50 | 3 | 0 |
| Q5_K_M | 32/50 | 64.0% | 49/50 | 1 | 0 |
| Q4_K_M | 31/50 | 62.0% | 48/50 | 2 | 0 |
| IQ4_XS | 30/50 | 60.0% | 45/50 | 5 | 0 |
| IQ4_XS-MTP-graft-headQ6 | 30/50 | 60.0% | 48/50 | 2 | 0 |
| Q8_0 | 29/50 | 58.0% | 47/50 | 3 | 0 |

## MTP Work

### Starting Problem

The upstream Ornith config advertises one MTP/NextN layer, but the released HF
weights and plain GGUF files do not contain a loadable MTP head. In practice,
the plain model cannot simply be started with llama.cpp `--spec-type draft-mtp`.
The work therefore split into two lines:

1. Make a loadable MTP draft path for experimentation.
2. Find a release artifact and serving policy that gives real single-user speed.

### Bootstrap Draft

The first bootstrap created a minimal external MTP draft by cloning trunk layer
39 into the expected MTP layout, adding the missing MTP projection/norm tensors,
converting with `convert_hf_to_gguf.py --mtp`, and quantizing the draft to
IQ4_XS.

Result:

| Profile | Loaded VRAM GiB | c16 tok/s | Acceptance | Verdict |
|---|---:|---:|---:|---|
| IQ4_XS target only | 19.34 | 476.0 | n/a | baseline |
| Bootstrap IQ4_XS MTP draft, `n_max=1` | 21.32 | 162.7 | 1.21% | not recommended |

This proved the server plumbing and artifact layout but did not provide useful
acceptance.

### Trained External Drafts

The second line trained chain-corrected Qwen3.5 MoE MTP drafts from target
hidden-state captures. The goal was to train a draft that predicts future tokens
from the target hidden state and can be exported as a separate MTP GGUF.

Best external draft family:

| Draft artifact | Quant | Loaded VRAM GiB | c16/128 tok/s | Acceptance | Acceptance by position |
|---|---|---:|---:|---:|---|
| `...h2r9fw125...Q6_K.gguf` | Q6_K | 22.44 | 286.77 | 67.22% | `(0.719, 0.621)` |
| `...h2r9fw125...Q5_K_M.gguf` | Q5_K_M | 22.16 | 285.32 | 66.61% | `(0.715, 0.613)` |
| `...h2r9fw125...IQ4_XS.gguf` | IQ4_XS | 21.53 | 265.06 | 59.20% | `(0.640, 0.541)` |

The Q6_K draft reached the requested development acceptance band, but the
serving path remained slower than the target-only matched baseline:

| Profile | c1 tok/s | c4 tok/s | c8 tok/s | c16 tok/s |
|---|---:|---:|---:|---:|
| IQ4_XS target only, no prompt cache | 239.91 | 431.95 | 470.81 | 595.89 |
| IQ4_XS + Q6_K trained MTP draft, `n_max=2`, per-head contexts | 235.99 | 245.28 | 247.82 | 288.67 |
| IQ4_XS + Q6_K trained MTP draft, sequential verify | 192.48 | 165.63 | 167.47 | 169.24 |

Conclusion: the external draft quality became good enough for development, but
the separate-draft runtime path was not a production speedup.

### Integrated IQ4_XS-MTP Graft

The release MTP path is the integrated graft:

`ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf`

This file was made by starting from the MIT-licensed
`wang-yang/Ornith-1.0-35B-MTP-GGUF` Q6_K graft and requantizing the body to
IQ4_XS while preserving the appended `blk.40.*` MTP block at Q6_K/F16/F32:

```bash
llama-quantize --allow-requantize \
  --tensor-type-file runs/tensor-types-mtp-blk40-preserve.txt \
  Ornith-1.0-35B-Q6_K-MTP.gguf \
  ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf \
  IQ4_XS 24
```

Structural verification:

| Key | Value |
|---|---|
| `general.architecture` | `qwen35moe` |
| `qwen35moe.block_count` | `41` |
| `qwen35moe.nextn_predict_layers` | `1` |
| MTP tensors | appended `blk.40.*` block with `blk.40.nextn.*` tensors |
| Body quant | IQ4_XS |
| MTP block quant | preserved Q6_K/F16/F32 |
| File size | 19.6 GB decimal, about 0.69 GB over plain IQ4_XS |

The integrated graft reproduced the high-acceptance behavior reported by the
wang-yang reference and avoided the heavy external draft context.

Low-concurrency results:

| Profile | Shape | tok/s | Speedup | Draft acceptance |
|---|---|---:|---:|---:|
| Integrated graft, target-only AR | c1/128 | 221.80 | 1.00x | n/a |
| Integrated graft, `draft-mtp n_max=2` | c1/128 | 279.36 | 1.26x | high |
| Integrated graft, adaptive MTP throttle | c1/128 | 319.53 | 1.44x | `(0.953, 0.865)` |

Adaptive c1 server counters:

| Counter | Value |
|---|---:|
| Generated speculative tokens | 339 |
| Accepted speculative tokens | 309 |
| Mean accepted length | 2.82 |
| Acceptance by position | `(0.953, 0.865)` |

Saturated c16 results:

| Profile | Shape | tok/s | Ratio vs AR | Notes |
|---|---|---:|---:|---|
| Target-only AR | c16/64 | 568.57 | 1.00x | matched control |
| Always-on active MTP, `n_max=2` | c16/64 | 323.12 | 0.57x | high acceptance but slower |
| Adaptive MTP throttle, `LLAMA_SPEC_MAX_DRAFTING_SLOTS=1` | c16/64 | 564.48 | 0.99x | MTP disabled while saturated |

The final serving policy is therefore adaptive MTP: enable MTP for a single
active request and disable drafting when the server is saturated.

Threshold sweep:

| `LLAMA_SPEC_MAX_DRAFTING_SLOTS` | c1 tok/s | c2 tok/s | c4 tok/s | c8 tok/s | c16 tok/s |
|---:|---:|---:|---:|---:|---:|
| 1 | 271.12 | 316.46 | 424.14 | 564.24 | 592.45 |
| 2 | 267.04 | 313.25 | 422.88 | 558.17 | 592.39 |
| 4 | 269.23 | 313.06 | 328.45 | 547.56 | 591.50 |

Recommended default: `LLAMA_SPEC_MAX_DRAFTING_SLOTS=1`.

### Dedicated MTP KLD and Sequence Evaluation

See [benchmarks/mtp-dedicated-kld-evaluation.md](benchmarks/mtp-dedicated-kld-evaluation.md)
for raw paths and full details.

Next-token top-64 KLD over 32 coding prompts:

| Candidate | Mean KLD nats | Mean KLD bits | P95 nats | Max nats | Top-1 match |
|---|---:|---:|---:|---:|---:|
| Integrated IQ4_XS-MTP graft, target-only | 0.0731382442 | 0.1055161823 | 0.1593743642 | 0.3348258511 | 29/32 |
| Integrated IQ4_XS-MTP graft, active `draft-mtp` | 0.0731382442 | 0.1055161823 | 0.1593743642 | 0.3348258511 | 29/32 |
| Active `draft-mtp` vs target-only graft | 0.0 | 0.0 | 0.0 | 0.0 | 32/32 |

The one-token API-visible distribution is clean: active `draft-mtp` returns the
same next-token distribution as the graft running target-only. The integrated
graft also measured lower drift than the prior plain IQ4_XS capture
(0.0731382442 vs 0.1425748206 mean KLD nats).

The longer 8 prompt x 64 token single-user sequence probe exercised real native
MTP drafting:

| Runtime | Client aggregate tok/s | Draft acceptance | Exact 64-token sequences | Token-position match |
|---|---:|---:|---:|---:|
| Target-only graft | 172.57 | n/a | 8/8 baseline | n/a |
| Fast active `draft-mtp`, `n_max=2` | 233.81 | 310/378 = 82.01%; `(0.884, 0.747)` by position | 6/8 | 478/512 = 93.36% |
| `LLAMA_SPEC_VERIFY_SEQUENTIAL=1` | 164.34 | 313/375 = 83.47%; `(0.884, 0.772)` by position | 5/8 | 455/512 = 88.87% |

This means the native graft remains an experimental low-concurrency speed
profile, not a strict target-equivalent deterministic serving mode. Also,
llama.cpp omitted candidate top-logprobs for most accepted speculative tokens
in sequence responses: 500/512 positions were missing candidate top-logprobs in
the fast active run. Comparable same-emitted-token positions had near-zero KLD
(0.0011215739 mean nats, 0.0040239103 max), but full sequence KLD is not valid
without runtime support for returning verifier logprobs on accepted draft tokens.

### Runtime Engineering Done

The MTP work included artifact surgery and runtime work:

| Area | Work completed | Impact |
|---|---|---|
| Missing MTP tensors | Built bootstrap MTP draft from trunk layer 39 and added required projection/norm tensors | Made llama.cpp `draft-mtp` loadable for local experiments |
| Hidden-state capture | Added/used capture paths for target hidden rows and token positions | Enabled trained external MTP draft experiments |
| Chain-corrected training | Trained h2/h1/h3 chain-corrected drafts and exported Q6_K/Q5_K_M/IQ4_XS GGUFs | Raised external-draft acceptance from about 1% to about 67% |
| Integrated graft requantization | Requantized a Q6_K integrated MTP graft to IQ4_XS while preserving `blk.40.*` MTP tensors | Produced a 19.6 GB native MTP artifact |
| Fast backend sampling | Added a fast path for unambiguous backend top-k=1 draft sampling | Reduced sampling overhead in the native graft path |
| Qwen35MoE recurrent row index | Patched recurrent-state writes and rollback snapshots to use runtime row-index tensors rather than graph-baked head offsets | Fixed graph/recurrent mismatch across MTP head changes |
| Per-head MTP contexts | Added optional `LLAMA_MTP_PER_HEAD_CONTEXT=1` path for trained external drafts | Improved graph reuse but did not make external drafts faster than target-only |
| Sequential verification | Added `LLAMA_SPEC_VERIFY_SEQUENTIAL=1` correctness guard | Made temperature-0 MTP output match target-only in deterministic probes |
| Adaptive throttling | Added/validated `LLAMA_SPEC_MAX_DRAFTING_SLOTS=1` policy | Keeps c1 MTP speedup while avoiding saturated c16 regression |

Important correctness note: the external trained-draft path can diverge from
target-only output without sequential verification. The sequential verifier
matches target-only output but is slower, so it is a debug/correctness guard, not
the production profile. The integrated graft's release value is low-concurrency
speed with adaptive throttling.

### MTP Recommendation

Use this single-GPU command for the integrated MTP profile:

```bash
export CUDA_VISIBLE_DEVICES=0
export LLAMA_SPEC_MAX_DRAFTING_SLOTS=1
export LLAMA_MTP_FAST_BACKEND_SAMPLE=1
export LLAMA_MTP_DRAFT_TOP_K=1
export LLAMA_MTP_DRAFT_TOP_P=1
export LLAMA_MTP_DRAFT_TEMP=1

llama-server \
  -m ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf \
  --alias Ornith-1.0-35B-GGUF-IQ4_XS-MTP \
  --host 127.0.0.1 --port 8002 \
  -c 131072 -np 1 -b 4096 -ub 512 -ngl 99 \
  --flash-attn on \
  --reasoning off --reasoning-format deepseek \
  --spec-type draft-mtp \
  --spec-draft-ngl 99 \
  --spec-draft-n-max 2 \
  --spec-draft-n-min 0 \
  --spec-draft-backend-sampling \
  --cont-batching \
  --metrics
```

For multi-slot server traffic, keep `LLAMA_SPEC_MAX_DRAFTING_SLOTS=1` so MTP is
used only when the server has a small active batch. For a single-user agentic
workload, this is the preferred MTP profile.

## Pi-Agent Coding Smoke

A Pi-agent smoke was run against the single-user integrated MTP profile with
`CTX_SIZE=131072`, `PARALLEL=1`, and model alias `ornith-pi-agent-mtp`.

| Test | Result |
|---|---|
| Calculator tool smoke | pass |
| Ledger parser/summary repo-edit task | pass, 3/3 tests |
| TTL cache repo-edit task | pass, 3/3 tests |
| Router dispatch repo-edit task | pass, 4/4 tests |

Serving counters from that Pi-agent run:

| Metric | Value |
|---|---:|
| Prompt throughput | 3460.27 tok/s |
| Decode throughput | 339.23 tok/s |
| MTP generated draft tokens | 1234 |
| MTP accepted draft tokens | 1114 |
| Aggregate draft acceptance | 90.3% |
| Acceptance by position | `(0.929, 0.877)` |
| Mean accepted length | 2.81 |
| Peak observed loaded VRAM | about 22.65 GiB |

## Pi-Agent Quant Flappy Bird Test

Each published serving profile was loaded as a single-user llama.cpp server and
attached to Pi-agent with the same vanilla HTML/CSS/JS Flappy Bird prompt. The
generated games are not hand-edited; Q3_K_M required three Pi-agent repair
passes, and the other profiles passed on the first agent run.

Full report: `benchmarks/pi-agent-flappy-quant-test.md`.
Raw evidence: `probes/pi-agent-flappy/`.

| Quant/Profile | Result | Decode TPS on Pi-agent task |
|---|---:|---:|
| Q3_K_M | pass after 3 Pi-agent repair passes | 250.88 |
| IQ4_XS | pass first run | 242.44 |
| Q4_K_M | pass first run | 249.59 |
| Q5_K_M | pass first run | 239.67 |
| Q6_K | pass first run | 226.51 |
| Q8_0 | pass first run | 212.98 |
| IQ4_XS-MTP-graft-headQ6 | pass first run | 320.91 |

Interaction checks loaded every generated page, verified one canvas, pressed
Space, captured an after-input screenshot, and observed zero page errors for all
7 profiles.

The MTP profile's cumulative server log for this agent workload reports 3,452
accepted draft tokens out of 3,882 generated draft tokens, 88.92% draft-token
acceptance, mean acceptance length 2.78, and per-position acceptance
`(0.936, 0.843)`.

## Evidence Index

- Quant benchmarks: `benchmarks/llamacpp-quant-benchmarks.md`
- Quant KLD probe: `benchmarks/kld-quant-vs-bf16-top64.md`
- Integrated MTP profile: `benchmarks/llamacpp-iq4-xs-mtp-graft-adaptive-profile.md`
- SWE-bench agent evaluations: `benchmarks/swebench-agent-evals.md`
- Pi-agent quant Flappy Bird test: `benchmarks/pi-agent-flappy-quant-test.md`
  with raw logs and metrics in `probes/pi-agent-flappy/`
- Trained MTP profile: `benchmarks/llamacpp-iq4-xs-mtp-chaincorr-profile.md`
- MTP profile catalog: `configs/mtp_profiles.yaml`
- Quant artifact catalog: `configs/quant_artifacts.yaml`
- Serving profile catalog: `configs/serving_profiles.yaml`
- Runtime patches: `patches/llamacpp-qwen35moe-mtp-recurrent-row-index.patch`,
  `patches/llamacpp-mtp-adaptive-serving-and-recurrent-rollback.patch`, and
  `patches/llamacpp-server-mtp-seqverify-and-perhead.patch`