--- base_model: - deepreinforce-ai/Ornith-1.0-35B tags: - gguf - llama.cpp - coding - quantized - single-gpu - openai-compatible library_name: gguf --- # Ornith-1.0-35B GGUF Single-GPU Serving Package This repo packages validated GGUF quants of `deepreinforce-ai/Ornith-1.0-35B` with a single-GPU llama.cpp serving profile and benchmark artifacts. Validated on 2026-06-26 with one RTX PRO 6000 Blackwell Max-Q 96GB GPU, `tp=1`, using `llama.cpp` CUDA server. Updated on 2026-06-27 with a local IQ4_XS quant, KLD probe, and trained experimental MTP drafts. ## Files - `ornith-1.0-35b-Q4_K_M.gguf` - `ornith-1.0-35b-Q3_K_M.gguf` - `ornith-1.0-35b-IQ4_XS.gguf` - `ornith-1.0-35b-mtp-bootstrap-layer39-IQ4_XS.gguf` - `ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-Q6_K.gguf` - `ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-Q5_K_M.gguf` - `ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-IQ4_XS.gguf` - `ornith-1.0-35b-Q5_K_M.gguf` - `ornith-1.0-35b-Q6_K.gguf` - `ornith-1.0-35b-Q8_0.gguf` - `scripts/serve_llamacpp_gpu0.sh` - `scripts/serve_llamacpp_gpu1.sh` - `scripts/build_bootstrap_mtp_checkpoint.py` - `benchmarks/llamacpp-quant-benchmarks.md` - `benchmarks/kld-quant-vs-bf16-top64.md` - `benchmarks/sft-lora-smoke-bigcode.md` - `benchmarks/sft-prod-20k-preflight.md` - `benchmarks/llamacpp-q3-k-m-profile.md` - `benchmarks/llamacpp-iq4-xs-profile.md` - `benchmarks/llamacpp-iq4-xs-mtp-bootstrap-profile.md` - `benchmarks/llamacpp-iq4-xs-mtp-chaincorr-profile.md` - `benchmarks/llamacpp-q4-reasoning-off-fix.md` - `probes/behavior-llamacpp-q3-k-m-parallel16.json` - `probes/probe-llamacpp-iq4-xs-correctness.json` - `probes/behavior-llamacpp-iq4-xs-parallel16.json` - `probes/probe-llamacpp-iq4-xs-mtp-bootstrap-correctness.json` - `probes/probe-llamacpp-iq4-xs-mtp-bootstrap-nmax1-correctness.json` - `probes/probe-llamacpp-iq4-xs-mtp-bootstrap-iq4draft-nmax1-correctness.json` - `probes/mtp-support-ornith-iq4-xs.json` - `probes/gguf-materialization-q3-k-m.json` - `probes/behavior-llamacpp-q4.json` - `probes/behavior-llamacpp-q4-script-default.json` - `probes/investigation-ornith-q4-live.md` - `probes/investigation-ornith-q4-live.json` ## Recommended Quant Use `Q4_K_M` for speed-focused serving. It remains the fastest validated GGUF in the benchmark profile. Use `Q3_K_M` when footprint matters more than peak throughput. It is a custom quant generated from the upstream BF16 GGUF, validated through the materialization audit and behavior suite, and is about 20.8% smaller on disk than Q4_K_M. Use `IQ4_XS` when you want a middle-footprint quant: smaller than Q4_K_M with better next-token KLD than Q3_K_M, but slower than Q4_K_M in the short-context throughput profile. Use the trained Q6_K MTP draft only for MTP experimentation and regression work. It makes llama.cpp `draft-mtp` operational with 67.22% draft-token acceptance at `n_max=2`, but it is not yet a production throughput speedup: the measured IQ4_XS target plus Q6_K draft profile reached 286.77 tok/s at c16 versus 482.68 tok/s for refreshed IQ4_XS target-only. The smaller Q5_K_M and IQ4_XS MTP drafts are included as experimental footprint variants but are not recommended. ## Quant Provenance All files are GGUF artifacts for `deepreinforce-ai/Ornith-1.0-35B`. | File | Quant | Provenance | Quantization detail | |---|---|---|---| | `ornith-1.0-35b-Q3_K_M.gguf` | Q3_K_M | Locally produced by this lab from the upstream BF16 GGUF | `llama-quantize ornith-1.0-35b-bf16.gguf ornith-1.0-35b-Q3_K_M.gguf Q3_K_M`; quantizer reported source 66,152.24 MiB / 16.01 BPW and quantized size 15,977.65 MiB / 3.87 BPW | | `ornith-1.0-35b-IQ4_XS.gguf` | IQ4_XS | Locally produced by this lab from the upstream BF16 GGUF | `llama-quantize ornith-1.0-35b-bf16.gguf ornith-1.0-35b-IQ4_XS.gguf IQ4_XS 32`; no importance matrix; quantizer reported source 66,152.24 MiB / 16.01 BPW and quantized size 18,051.46 MiB / 4.37 BPW | | `ornith-1.0-35b-mtp-bootstrap-layer39-IQ4_XS.gguf` | IQ4_XS draft MTP | Locally produced by this lab from the released HF safetensors | `build_bootstrap_mtp_checkpoint.py --source-layer 39 --eh-proj-init hidden`, then `convert_hf_to_gguf.py --mtp --outtype bf16`, then `llama-quantize ... IQ4_XS 32`; draft quantizer reported source 3,552.05 MiB / 16.00 BPW and quantized size 1,085.37 MiB / 4.89 BPW | | `ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-Q6_K.gguf` | Q6_K draft MTP | Locally trained by this lab from target hidden-state captures, then converted to GGUF | Chain-corrected Qwen3.5 MoE MTP draft; `llama-quantize ... Q6_K`; source draft 8,388.17 MiB / 16.01 BPW; estimated draft load 3,176.45 MiB | | `ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-Q5_K_M.gguf` | Q5_K_M draft MTP | Locally trained by this lab from target hidden-state captures, then converted to GGUF | Same trained draft as above, quantized with `llama-quantize ... Q5_K_M`; quantized size 3,090.01 MiB / 5.90 BPW; estimated draft load 2,885.59 MiB | | `ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-IQ4_XS.gguf` | IQ4_XS draft MTP | Locally trained by this lab from target hidden-state captures, then converted to GGUF | Same trained draft as above, quantized with `llama-quantize ... IQ4_XS`; quantized size 2,374.93 MiB / 4.53 BPW; estimated draft load 2,246.29 MiB | | `ornith-1.0-35b-Q4_K_M.gguf` | Q4_K_M | Mirrored from `deepreinforce-ai/Ornith-1.0-35B-GGUF`, then validated here | Upstream llama.cpp GGUF K-quant artifact; exact upstream quantization command was not republished in this repo | | `ornith-1.0-35b-Q5_K_M.gguf` | Q5_K_M | Mirrored from `deepreinforce-ai/Ornith-1.0-35B-GGUF`, then validated here | Upstream llama.cpp GGUF K-quant artifact; exact upstream quantization command was not republished in this repo | | `ornith-1.0-35b-Q6_K.gguf` | Q6_K | Mirrored from `deepreinforce-ai/Ornith-1.0-35B-GGUF`, then validated here | Upstream llama.cpp GGUF K-quant artifact; exact upstream quantization command was not republished in this repo | | `ornith-1.0-35b-Q8_0.gguf` | Q8_0 | Mirrored from `deepreinforce-ai/Ornith-1.0-35B-GGUF`, then validated here | Upstream llama.cpp GGUF Q8_0 artifact; exact upstream quantization command was not republished in this repo | The locally produced Q3_K_M and IQ4_XS files were generated with the CPU `llama-quantize` binary built from the checked-out `llama.cpp` source in this lab. The input BF16 GGUF was downloaded from `deepreinforce-ai/Ornith-1.0-35B-GGUF`: `ornith-1.0-35b-bf16.gguf`. ## KLD Probe Each quant was compared to the upstream BF16 GGUF baseline with a corrected native llama.cpp next-token top-64 `KL(P_bf16 || P_quant)` probe over 32 coding prompts. The probe used `/completion` with `n_predict=1`, `temperature=-1`, `n_probs=64`, `cache_prompt=false`, and token-ID based matching. The BF16 baseline loaded 64.57 GiB VRAM in this KLD profile. | Quant | KLD profile VRAM GiB | Mean KLD nats | P50 nats | P95 nats | Max nats | Top-1 match | |---|---:|---:|---:|---:|---:|---:| | Q3_K_M | 16.32 | 0.3620 | 0.2548 | 1.1077 | 1.3730 | 84.4% | | IQ4_XS | 19.34 | 0.1426 | 0.0868 | 0.3195 | 0.6586 | 84.4% | | Q4_K_M | 20.35 | 0.0864 | 0.0379 | 0.2877 | 0.4503 | 90.6% | | Q5_K_M | 23.61 | 0.0354 | 0.0235 | 0.0943 | 0.2497 | 93.8% | | Q6_K | 27.07 | 0.0165 | 0.0092 | 0.0513 | 0.0586 | 100.0% | | Q8_0 | 34.77 | 0.0108 | 0.0052 | 0.0440 | 0.0590 | 96.9% | Q8_0 had the lowest mean KLD in this corrected top-64 probe. Q6_K was nearly as close and had perfect top-1 agreement across all prompts. IQ4_XS landed between Q3_K_M and Q4_K_M: materially closer to BF16 than Q3_K_M, but not as close as Q4_K_M. See `benchmarks/kld-quant-vs-bf16-top64.md` for method caveats and raw artifact paths. ## Serving Fast short-context throughput profile: ```bash QUANT=Q4_K_M PORT=8002 CTX_SIZE=8192 PARALLEL=16 \ scripts/serve_llamacpp_gpu0.sh ``` Lowest-memory profile: ```bash QUANT=Q3_K_M PORT=8002 CTX_SIZE=8192 PARALLEL=16 \ scripts/serve_llamacpp_gpu0.sh ``` Middle-footprint profile: ```bash QUANT=IQ4_XS PORT=8002 CTX_SIZE=8192 PARALLEL=16 \ scripts/serve_llamacpp_gpu0.sh ``` Experimental trained-MTP profile: ```bash SPEC_DRAFT_BACKEND_SAMPLING=1 \ LLAMA_MTP_FAST_BACKEND_SAMPLE=1 \ LLAMA_MTP_DRAFT_TOP_K=1 LLAMA_MTP_DRAFT_TOP_P=1 LLAMA_MTP_DRAFT_TEMP=1 \ QUANT=IQ4_XS PORT=8002 CTX_SIZE=8192 PARALLEL=16 \ SPEC_TYPE=draft-mtp \ SPEC_DRAFT_MODEL=ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-Q6_K.gguf \ SPEC_DRAFT_N_MAX=2 SPEC_DRAFT_N_MIN=0 \ scripts/serve_llamacpp_gpu0.sh ``` Coding-serving profile with 16 slots and about 8k tokens per slot: ```bash QUANT=Q4_K_M PORT=8002 CTX_SIZE=131072 PARALLEL=16 \ scripts/serve_llamacpp_gpu0.sh ``` Use `scripts/serve_llamacpp_gpu1.sh` for the second physical GPU. The wrappers accept `QUANT`, `MODEL_PATH`, `PORT`, `ALIAS`, `CTX_SIZE`, `PARALLEL`, `BATCH_SIZE`, `UBATCH_SIZE`, `CONT_BATCHING`, `METRICS`, `BACKEND_SAMPLING`, `REASONING`, `REASONING_FORMAT`, `REASONING_BUDGET`, `REASONING_BUDGET_MESSAGE`, `SPEC_TYPE`, `SPEC_DRAFT_MODEL`, `SPEC_DRAFT_HF`, `SPEC_DRAFT_NGL`, `SPEC_DRAFT_N_MAX`, `SPEC_DRAFT_N_MIN`, `SPEC_DRAFT_P_MIN`, `SPEC_DRAFT_DEVICE`, and `SPEC_DRAFT_BACKEND_SAMPLING`. The default is `REASONING=off`. This is the usable coding-serving profile: with llama.cpp reasoning left on/auto, simple requests can spend the whole short response budget in parsed `reasoning_content` and return empty final `content`. Set `REASONING=on` only for explicit thinking-mode experiments. ## Correctness All uploaded quants passed an OpenAI-compatible correctness gate through llama.cpp: - `/v1/models` listed the served alias. - `/v1/chat/completions` returned coherent text. - `/v1/completions` returned coherent text. - Output matched `def add(` and `return a + b`. The same model's GGUF path was not usable in the tested vLLM build: Q4_K_M loaded but produced corrupted output. Use llama.cpp for these GGUF files unless vLLM GGUF support for this architecture is fixed. The Q4 llama.cpp profile also found and fixed a reasoning-mode serving issue. With reasoning left on/auto, short coding requests could finish by length with empty final `content`; the patched default `REASONING=off` profile passes the behavior suite with 14/14 checks and 0 issues. See `benchmarks/llamacpp-q4-reasoning-off-fix.md`. The custom Q3_K_M quant passed the same behavior suite on the exact 16-slot serving profile with 14/14 checks and 0 issues. See `benchmarks/llamacpp-q3-k-m-profile.md`. The local IQ4_XS quant also passed the OpenAI-compatible correctness gate and the same 16-slot behavior suite with 14/14 checks and 0 issues. See `benchmarks/llamacpp-iq4-xs-profile.md`. The MTP profiles also passed the OpenAI-compatible correctness gate. The bootstrap profile is preserved as a historical loadability baseline, while the trained chain-corrected Q6_K draft is the current MTP development profile. See `benchmarks/llamacpp-iq4-xs-mtp-bootstrap-profile.md` and `benchmarks/llamacpp-iq4-xs-mtp-chaincorr-profile.md`. ## Investigation Harness The reusable investigation harness combines metadata/tokenizer/config checks with live OpenAI-compatible behavior probes. A live Q4_K_M llama.cpp run passed 27/27 combined checks with 0 behavior issues. The only remaining issues were low-severity metadata notes: - `generation-eos-includes-pad`: generation config EOS list includes the PAD token. - `missing-config-torch-dtype`: model config does not declare `torch_dtype`, so serving/quantization commands should pass dtype explicitly. See `probes/investigation-ornith-q4-live.md` and `probes/investigation-ornith-q4-live.json`. ## MTP Status The released HF config advertises `mtp_num_hidden_layers=1`, but the released HF safetensors and GGUF artifacts do not contain a loadable MTP/NextN head. The source snapshot has no `mtp`, `nextn`, `draft`, or `eagle` tensor names, and the GGUF metadata has no `nextn` keys. A llama.cpp `convert_hf_to_gguf.py --mtp --dry-run` only planned embeddings, output head, and output norm, not the Qwen3.5 MoE MTP block tensors required by llama.cpp. Status of the released upstream checkpoint: `config_only_not_loadable`. To make MTP loadable, this lab first created `ornith-1.0-35b-mtp-bootstrap-layer39-IQ4_XS.gguf`, a separate llama.cpp MTP draft GGUF. It copies full-attention trunk layer 39 into `model.mtp.layers.0`, adds the missing `mtp.fc` and MTP input norms, converts with `convert_hf_to_gguf.py --mtp`, and quantizes the draft to IQ4_XS. That artifact loads with `--spec-type draft-mtp --spec-draft-model ...` and passes the OpenAI-compatible correctness gate, but only accepted 1.21% of draft tokens. This lab then trained a chain-corrected Qwen3.5 MoE MTP draft from target hidden-state captures and exported Q6_K, Q5_K_M, and IQ4_XS draft GGUFs. The current best MTP profile is IQ4_XS target plus Q6_K trained draft with `n_max=2`, draft backend sampling, and fast top-k=1 draft sampling: - c16 throughput: 286.77 tok/s - draft-token acceptance: 67.22% - acceptance by position: `(0.719, 0.621)` - approximate loaded VRAM: 22.44 GiB This is operational but still not a production MTP speedup. The refreshed IQ4_XS target-only profile reaches 482.68 tok/s at c16. The remaining blocker is runtime efficiency: llama.cpp currently executes draft and target verification mostly sequentially with host-side sampling/control overhead, and target-only c16 already saturates the GPU better. See `probes/mtp-support-ornith-iq4-xs.json`, `benchmarks/llamacpp-iq4-xs-profile.md`, `benchmarks/llamacpp-iq4-xs-mtp-bootstrap-profile.md`, and `benchmarks/llamacpp-iq4-xs-mtp-chaincorr-profile.md`. ## Fine-Tuning Smoke A one-step LoRA SFT smoke run succeeded on GPU1 using `bigcode/self-oss-instruct-sc2-exec-filter-50k` with license metadata required. This validates the local training stack and data pipeline only. No capability-bearing fine-tuned adapter is released in this repo. Smoke metrics: train loss 1.1419, eval loss 1.1048, train mean token accuracy 0.7574, eval mean token accuracy 0.7442. See `benchmarks/sft-lora-smoke-bigcode.md`. The production coding SFT run has been prepared and launched locally as a 20,000-step LoRA run over 44,000 train rows and 2,000 eval rows from `bigcode/self-oss-instruct-sc2-exec-filter-50k`. No adapter from that run is released in this repo until the run completes and passes eval gates. See `benchmarks/sft-prod-20k-preflight.md`. ## Benchmark Summary Benchmark profile: ```bash CUDA_VISIBLE_DEVICES=1 \ HOST=127.0.0.1 PORT= QUANT= \ CTX_SIZE=8192 PARALLEL=16 BATCH_SIZE=4096 UBATCH_SIZE=512 \ CONT_BATCHING=1 METRICS=1 \ scripts/serve_llamacpp_gpu1.sh ``` `CTX_SIZE=8192 PARALLEL=16` gives 512 context tokens per slot in llama.cpp server. Use `CTX_SIZE=131072 PARALLEL=16` for about 8k tokens per slot. | Quant | GGUF GiB | Loaded VRAM GiB | c1 tok/s | c1 p95 TTFT ms | c4 tok/s | c4 p95 TTFT ms | c8 tok/s | c8 p95 TTFT ms | c16 tok/s | c16 p95 TTFT ms | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | Q3_K_M | 15.61 | 17.27 | 240.5 | 77.9 | 422.0 | 170.7 | 464.4 | 344.9 | 493.0 | 493.7 | | IQ4_XS | 17.64 | 19.34 | 234.1 | 75.1 | 297.7 | 159.7 | 411.5 | 330.2 | 476.0 | 541.5 | | Q4_K_M | 19.71 | 21.31 | 243.3 | 76.3 | 458.3 | 192.3 | 615.0 | 361.8 | 655.6 | 650.0 | | Q5_K_M | 23.03 | 24.65 | 236.7 | 75.1 | 311.0 | 198.8 | 439.0 | 383.6 | 638.6 | 620.4 | | Q6_K | 26.56 | 28.03 | 225.9 | 76.8 | 295.8 | 194.2 | 409.6 | 394.8 | 603.3 | 657.4 | | Q8_0 | 34.37 | 35.72 | 208.5 | 76.9 | 281.5 | 190.8 | 405.8 | 389.1 | 601.4 | 725.8 | Full raw benchmark details are in `benchmarks/llamacpp-quant-benchmarks.md`. Experimental MTP profiles on the same IQ4_XS target: | Profile | Loaded VRAM GiB | c1 tok/s | c4 tok/s | c8 tok/s | c16 tok/s | Draft acceptance | |---|---:|---:|---:|---:|---:|---:| | IQ4_XS target only | 19.34 | 234.1 | 297.7 | 411.5 | 482.68 refreshed / 476.0 original | n/a | | IQ4_XS + BF16 MTP draft, n_max=3 | 25.12 | 95.5 | 97.0 | 102.1 | 108.2 | 0.47% | | IQ4_XS + BF16 MTP draft, n_max=1 | 23.03 | 153.6 | 147.2 | 154.3 | 159.5 | 1.07% | | IQ4_XS + IQ4_XS MTP draft, n_max=1 | 21.32 | 155.1 | 151.6 | 155.8 | 162.7 | 1.21% | | IQ4_XS + Q6_K trained MTP draft, n_max=2 | 22.44 | n/t | n/t | n/t | 286.77 | 67.22% | | IQ4_XS + Q5_K_M trained MTP draft, n_max=2 | 22.16 | n/t | n/t | n/t | 285.32 | 66.61% | | IQ4_XS + IQ4_XS trained MTP draft, n_max=2 | 21.53 | n/t | n/t | n/t | 265.06 | 59.20% | | IQ4_XS + Q6_K trained MTP draft, n_max=1 | 22.44 | n/t | n/t | n/t | 262.76 | 68.12% | `n/t` means not tested in the latest trained-MTP loop; those rows were measured only on the c16 profile used for the MTP fix loop. Note: Q3_K_M was benchmarked after the serving script default changed to `REASONING=off`; the original Q4_K_M through Q8_0 throughput rows are preserved from the earlier sweep. The usable serving profile in this repo defaults to `REASONING=off` because reasoning-on/auto can spend short responses in parsed `reasoning_content` and return empty final `content`.