---
base_model: deepreinforce-ai/Ornith-1.0-35B
library_name: gguf
pipeline_tag: text-generation
license: other
license_name: mit-apache-2.0-composite
tags:
  - gguf
  - llama.cpp
  - ornith
  - mixture-of-experts
  - speculative-decoding
  - mtp
  - vulkan
  - amd
  - strix-halo
  - cuda
  - nvidia
  - windows
---

# Ornith-1.0-35B MTP Strix Halo Hybrid GGUF

## Available optimized builds

| Build | Runtime target | Integrated MTP |
|---|---|---:|
| **[Full MTP build (this repository)](https://huggingface.co/petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF)** | Current llama.cpp with `draft-mtp`; patched Vulkan/CUDA deployments | Yes |
| **[LM Studio compatible build](https://huggingface.co/petr567/Ornith-1.0-35B-Strix-Halo-Hybrid-LMStudio-GGUF)** | LM Studio runtimes that reject integrated Qwen3.6/Ornith MTP metadata | No |

The two repositories contain the same selectively optimized 40-layer base.
Choose this repository when the runtime supports integrated MTP. Choose the
second repository for **LM Studio compatible, no integrated MTP** loading.

## Fast Windows local endpoint

For the tested RTX 4060 Laptop 8 GiB + 64 GiB RAM layout, use the complete
[Windows endpoint recipe](WINDOWS_ENDPOINT_QUICKSTART.md) or download and run
[`run-ornith-mtp-windows.ps1`](run-ornith-mtp-windows.ps1). It starts a
localhost-only OpenAI-compatible API with the exact llama.cpp b10066 image and
the measured CPU-MoE/GPU-dense MTP profile.

```powershell
.\run-ornith-mtp-windows.ps1 -ModelDir 'C:\Models\Ornith-MTP' -Port 18081
```

Endpoint: `http://127.0.0.1:18081/v1`. Do not substitute the no-MTP LM Studio
file if speculative acceleration is required.

The documented flow was re-tested from a fresh `C:\Models\Ornith-MTP`
directory using a real Hugging Face download (no hard links), SHA-256
verification, Docker startup, `/health`, `/v1/models`, and
`/v1/chat/completions`.

This repository contains a single, deployment-ready GGUF optimized and tested
for batch-1 agent workloads on AMD Ryzen AI MAX+ 395 / Radeon 8060S (`gfx1151`,
128 GiB UMA) with the Vulkan backend of llama.cpp b9994. The same artifact was
also A/B-tested on a Windows 11 laptop with an 8 GiB NVIDIA RTX 4060 Laptop GPU
using the official CUDA llama.cpp container. The two platforms and runtimes are
reported separately below; server Vulkan results are not mixed with laptop
CUDA results.

## File

| File | Size | Quantization | SHA-256 |
|---|---:|---|---|
| `ornith-1.0-35b-MTP-graft-down-Q4_0.gguf` | 19,377.09 MiB | Q4_K_M with 20 hot `ffn_down_exps` tensors overridden to Q4_0; integrated MTP layer | `365a7c02dfd320b9696f189d6dc12bd2b0eabb9f8e58ba9fc8cab3af93c0234b` |

The base model is [deepreinforce-ai/Ornith-1.0-35B](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B).
The compatible MTP-only tensors originate from
[a4lg/Qwen3.6-35B-A3B-MTP-ONLY-GGUF](https://huggingface.co/a4lg/Qwen3.6-35B-A3B-MTP-ONLY-GGUF).
The combined artifact therefore retains the base model's MIT terms and the
MTP donor's Apache-2.0 terms; review both linked source repositories before
redistribution or commercial use.

## What was changed

- grafted the compatible Qwen3.6 35B A3B MTP layer into the Ornith GGUF;
- selectively requantized 20 frequently streamed MoE down-projection tensors
  from Q6_K to Q4_0 while retaining the surrounding Q4_K_M plan;
- selected MTP depth 2 and probability floor 0.20 by measured sweep;
- combined verified MTP with exact n-gram drafting for repeated agent/code
  traffic;
- retained Q8_0/Q8_0 KV cache after a 64K boundary test showed Q4_0 KV slower.

No weights were trained or fine-tuned. The target model still verifies every
speculative token. Selective requantization does change numerical weights, so
quality was checked end-to-end on the same Orion benchmark slice.

## Benchmarks

### AMD Strix Halo server (Vulkan)

Hardware: Ryzen AI MAX+ 395, Radeon 8060S Vulkan (`gfx1151`), 128 GiB UMA,
batch/parallelism 1. Full machine-readable results are in
`benchmark-results.json`.

| Metric | Archived Ornith Q4_K_M | This hybrid | Change |
|---|---:|---:|---:|
| Weighted decode, identical 43-task Orion run | 61.446 tok/s | **79.661 tok/s** | **+29.6%** |
| Mean request decode | 65.681 tok/s | **83.944 tok/s** | **+27.8%** |
| P50 request decode | 68.45 tok/s | **84.84 tok/s** | **+23.9%** |
| Maximum request decode | 75.03 tok/s | **177.48 tok/s** | **+136.5%** |
| End-to-end wall time | 72m 42s | **36m 11s** | **-50.2%** |
| Aggregate quality | 40.8333 / 43 | **40.8333 / 43** | identical |

On a five-run repeated code stream, the combined MTP + n-gram profile averaged
**99.088 tok/s** (90.367 minimum, 116.945 maximum). This is workload-dependent,
not a universal 96 tok/s floor. At 62,630 prompt tokens plus 512 generated
tokens, Q8 KV decoded at 70.29 tok/s.

### Windows 11 laptop (CUDA): baseline vs this hybrid

This is a same-machine comparison, not a comparison against the server. The
test machine was a MACHENIKE L16P running Windows 11 Pro 24H2
(`26100.4652`), with an Intel Core i7-13650HX, 63.74 GiB RAM and an NVIDIA
GeForce RTX 4060 Laptop GPU (8,188 MiB, driver 591.74). Docker Desktop ran the
official `ghcr.io/ggml-org/llama.cpp:server-cuda` image, llama.cpp b10066
(`86a9c79f8`).

Both GGUF files used the same 16,384-token context, batch 2048, ubatch 512,
parallelism 1, Flash Attention, Q8_0 K/V cache, `--n-gpu-layers all`,
`--cpu-moe` and `--no-mmap`. This 8 GiB layout keeps dense/attention tensors
on the RTX GPU and MoE experts in host RAM. The baseline was
`ornith-1.0-35b-Q4_K_M.gguf` (SHA-256
`ff25291b2599fb927a835e624d2b3540106af61761c3fa57ac4264046dbec002`);
the hybrid was the file published in this repository. A naive 15-layer split
was rejected because a realistic 1K prompt fell to about 1.2 tok/s when MoE
expert traffic crossed the CPU/GPU boundary.

Raw `/completion` runs generated 128 tokens. Prompt rows were repeated twice;
the repeated-code row was repeated three times.

| Laptop profile | 1K decode | 8K decode | Repeated-code decode | 8K prompt processing |
|---|---:|---:|---:|---:|
| Baseline Q4_K_M, no speculation | 5.57 tok/s | 23.90 tok/s | 21.95 tok/s | 341.48 tok/s |
| This hybrid, speculation off | **29.70 tok/s** | **29.59 tok/s** | **26.75 tok/s** | **365.26 tok/s** |
| This hybrid, MTP + n-gram | 28.74 tok/s | **66.88 tok/s** | **67.23 tok/s** | 354.19 tok/s |

The baseline 1K row includes cold expert-cache/thrashing behavior and should
not be read as a universal 5.3x gain. The stable non-speculative hybrid gain
was +23.8% at 8K and +21.8% on repeated code. MTP + n-gram reached 67.23 tok/s
when draft acceptance was high; on less predictable short output it was 3.2%
slower than the hybrid without speculation. Observed draft acceptance was
about 45.5-51.9% for non-repetitive generations and 100% for repeated code.

| Laptop resource peak | Baseline Q4_K_M | Hybrid, no spec | Hybrid, MTP + n-gram |
|---|---:|---:|---:|
| GPU temperature | 71 C | 69 C | 69 C |
| GPU utilization | 98% | 98% | 98% |
| VRAM used | 2,342 MiB | 2,234 MiB | 2,672 MiB |
| GPU power | 49.34 W | 45.80 W | 46.47 W |
| Whole-system RAM used | 44.48 GiB | 42.64 GiB | 42.77 GiB |

The same four seeded Orion coding workspaces were then executed with
`approval=auto`, a 24-step cap and a 900-second per-task timeout. Deterministic
local verifiers evaluated the resulting files.

| Orion coding scenario | Baseline quality | Hybrid quality | Baseline time | Hybrid time | Hybrid time reduction |
|---|---:|---:|---:|---:|---:|
| Java service coverage | 80%, 8/10 | 80%, 8/10 | 323.9 s | 322.4 s | 0.4% |
| Kafka Node order pipeline | 75%, 6/8 | 75%, 6/8 | 282.6 s | 185.7 s | 34.3% |
| Rabbit retry / DLQ | 75%, 6/8 | 75%, 6/8 | 466.0 s | 324.7 s | 30.3% |
| Kafka Java outbox / idempotency | 80%, 8/10 | 80%, 8/10 | 382.4 s | 313.4 s | 18.1% |
| **Total / average** | **77.5%, 28/36** | **77.5%, 28/36** | **1,454.9 s** | **1,146.2 s** | **21.2%** |

The Windows laptop A/B therefore measured identical verifier quality and a
1.269x end-to-end speedup (308.7 seconds saved) for the hybrid. Both artifacts
missed the same Orion `cli_exit_zero` and `final_contract` checks: their
workspace changes passed identically, but neither produced the harness's
required final `changed / verified / not verified / risks` contract. This was
not an API, context or infrastructure failure and does not indicate a hybrid
quality regression.

These laptop numbers are intentionally modest in scope: one Windows machine,
16K context, 2/2/3 raw repetitions, 128 generated tokens, and four coding
scenarios. The 60-67 tok/s range is workload-dependent; a realistic production
expectation for ordinary agent traffic is roughly a 20-30% improvement.

## Recommended llama-server profile

Use llama.cpp b9994 or a compatible build with MTP support:

```bash
GGML_VK_ALLOW_GRAPHICS_QUEUE=1 \
GGML_VK_DISABLE_COOPMAT=1 \
GGML_VK_MAX_NODES_PER_SUBMIT=50 \
llama-server \
  --model ornith-1.0-35b-MTP-graft-down-Q4_0.gguf \
  --alias ornith-1.0-35b \
  --ctx-size 131072 --parallel 1 \
  --batch-size 2048 --ubatch-size 1024 \
  --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 \
  --n-gpu-layers all --device Vulkan0 --no-mmap \
  --spec-type ngram-mod,draft-mtp \
  --spec-draft-n-max 2 --spec-draft-n-min 1 --spec-draft-p-min 0.20 \
  --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 \
  --spec-ngram-mod-n-match 24
```

These environment variables and throughput figures are specific to the tested
AMD Strix Halo Vulkan system. Other backends and GPUs require their own sweep.

## Validation

- 790 Vulkan backend operation tests passed for every tested K-quant row-merge
  value (1, 2, 4, 8; 3,160 test cases total);
- identical aggregate quality on the same 43 tasks;
- Windows 11 / RTX 4060 Laptop A/B: identical 28/36 Orion checks with 21.2%
  lower hybrid wall time across the same four coding scenarios;
- GGUF checksum verified before publication;
- 131,072-token server context and OpenAI-compatible API verified in production.

## Reproducibility

The source fork, tensor manifest, production Compose profile, benchmark JSON,
and detailed research materials are maintained separately in the author's
private source repository. This Hugging Face repository intentionally contains
only the model, benchmark summary, and this concise deployment card.