Instructions to use petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF",
	filename="ornith-1.0-35b-MTP-graft-down-Q4_0.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF:Q4_0
# Run inference directly in the terminal:
llama cli -hf petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF:Q4_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF:Q4_0
# Run inference directly in the terminal:
llama cli -hf petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF:Q4_0

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF:Q4_0
# Run inference directly in the terminal:
./llama-cli -hf petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF:Q4_0

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF:Q4_0
# Run inference directly in the terminal:
./build/bin/llama-cli -hf petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF:Q4_0

Use Docker

docker model run hf.co/petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF:Q4_0

LM Studio
Jan

vLLM

How to use petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF:Q4_0

Ollama
How to use petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF with Ollama:
```
ollama run hf.co/petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF:Q4_0
```

Unsloth Studio

How to use petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF to start chatting

How to use petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF:Q4_0

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF:Q4_0"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF:Q4_0

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF:Q4_0

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF:Q4_0

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF:Q4_0" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF with Docker Model Runner:
```
docker model run hf.co/petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF:Q4_0
```

Lemonade

How to use petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF:Q4_0

Run and chat with the model

lemonade run user.Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF-Q4_0

List all available models

lemonade list

Ornith-1.0-35B MTP Strix Halo Hybrid GGUF

Available optimized builds

Build	Runtime target	Integrated MTP
Full MTP build (this repository)	Current llama.cpp with `draft-mtp`; patched Vulkan/CUDA deployments	Yes
LM Studio compatible build	LM Studio runtimes that reject integrated Qwen3.6/Ornith MTP metadata	No

The two repositories contain the same selectively optimized 40-layer base. Choose this repository when the runtime supports integrated MTP. Choose the second repository for LM Studio compatible, no integrated MTP loading.

Fast Windows local endpoint

For the tested RTX 4060 Laptop 8 GiB + 64 GiB RAM layout, use the complete Windows endpoint recipe or download and run run-ornith-mtp-windows.ps1. It starts a localhost-only OpenAI-compatible API with the exact llama.cpp b10066 image and the measured CPU-MoE/GPU-dense MTP profile.

.\run-ornith-mtp-windows.ps1 -ModelDir 'C:\Models\Ornith-MTP' -Port 18081

Endpoint: http://127.0.0.1:18081/v1. Do not substitute the no-MTP LM Studio file if speculative acceleration is required.

The documented flow was re-tested from a fresh C:\Models\Ornith-MTP directory using a real Hugging Face download (no hard links), SHA-256 verification, Docker startup, /health, /v1/models, and /v1/chat/completions.

This repository contains a single, deployment-ready GGUF optimized and tested for batch-1 agent workloads on AMD Ryzen AI MAX+ 395 / Radeon 8060S (gfx1151, 128 GiB UMA) with the Vulkan backend of llama.cpp b9994. The same artifact was also A/B-tested on a Windows 11 laptop with an 8 GiB NVIDIA RTX 4060 Laptop GPU using the official CUDA llama.cpp container. The two platforms and runtimes are reported separately below; server Vulkan results are not mixed with laptop CUDA results.

File

File	Size	Quantization	SHA-256
`ornith-1.0-35b-MTP-graft-down-Q4_0.gguf`	19,377.09 MiB	Q4_K_M with 20 hot `ffn_down_exps` tensors overridden to Q4_0; integrated MTP layer	`365a7c02dfd320b9696f189d6dc12bd2b0eabb9f8e58ba9fc8cab3af93c0234b`

The base model is deepreinforce-ai/Ornith-1.0-35B. The compatible MTP-only tensors originate from a4lg/Qwen3.6-35B-A3B-MTP-ONLY-GGUF. The combined artifact therefore retains the base model's MIT terms and the MTP donor's Apache-2.0 terms; review both linked source repositories before redistribution or commercial use.

What was changed

grafted the compatible Qwen3.6 35B A3B MTP layer into the Ornith GGUF;
selectively requantized 20 frequently streamed MoE down-projection tensors from Q6_K to Q4_0 while retaining the surrounding Q4_K_M plan;
selected MTP depth 2 and probability floor 0.20 by measured sweep;
combined verified MTP with exact n-gram drafting for repeated agent/code traffic;
retained Q8_0/Q8_0 KV cache after a 64K boundary test showed Q4_0 KV slower.

No weights were trained or fine-tuned. The target model still verifies every speculative token. Selective requantization does change numerical weights, so quality was checked end-to-end on the same Orion benchmark slice.

Benchmarks

AMD Strix Halo server (Vulkan)

Hardware: Ryzen AI MAX+ 395, Radeon 8060S Vulkan (gfx1151), 128 GiB UMA, batch/parallelism 1. Full machine-readable results are in benchmark-results.json.

Metric	Archived Ornith Q4_K_M	This hybrid	Change
Weighted decode, identical 43-task Orion run	61.446 tok/s	79.661 tok/s	+29.6%
Mean request decode	65.681 tok/s	83.944 tok/s	+27.8%
P50 request decode	68.45 tok/s	84.84 tok/s	+23.9%
Maximum request decode	75.03 tok/s	177.48 tok/s	+136.5%
End-to-end wall time	72m 42s	36m 11s	-50.2%
Aggregate quality	40.8333 / 43	40.8333 / 43	identical

On a five-run repeated code stream, the combined MTP + n-gram profile averaged 99.088 tok/s (90.367 minimum, 116.945 maximum). This is workload-dependent, not a universal 96 tok/s floor. At 62,630 prompt tokens plus 512 generated tokens, Q8 KV decoded at 70.29 tok/s.

Windows 11 laptop (CUDA): baseline vs this hybrid

This is a same-machine comparison, not a comparison against the server. The test machine was a MACHENIKE L16P running Windows 11 Pro 24H2 (26100.4652), with an Intel Core i7-13650HX, 63.74 GiB RAM and an NVIDIA GeForce RTX 4060 Laptop GPU (8,188 MiB, driver 591.74). Docker Desktop ran the official ghcr.io/ggml-org/llama.cpp:server-cuda image, llama.cpp b10066 (86a9c79f8).

Both GGUF files used the same 16,384-token context, batch 2048, ubatch 512, parallelism 1, Flash Attention, Q8_0 K/V cache, --n-gpu-layers all, --cpu-moe and --no-mmap. This 8 GiB layout keeps dense/attention tensors on the RTX GPU and MoE experts in host RAM. The baseline was ornith-1.0-35b-Q4_K_M.gguf (SHA-256 ff25291b2599fb927a835e624d2b3540106af61761c3fa57ac4264046dbec002); the hybrid was the file published in this repository. A naive 15-layer split was rejected because a realistic 1K prompt fell to about 1.2 tok/s when MoE expert traffic crossed the CPU/GPU boundary.

Raw /completion runs generated 128 tokens. Prompt rows were repeated twice; the repeated-code row was repeated three times.

Laptop profile	1K decode	8K decode	Repeated-code decode	8K prompt processing
Baseline Q4_K_M, no speculation	5.57 tok/s	23.90 tok/s	21.95 tok/s	341.48 tok/s
This hybrid, speculation off	29.70 tok/s	29.59 tok/s	26.75 tok/s	365.26 tok/s
This hybrid, MTP + n-gram	28.74 tok/s	66.88 tok/s	67.23 tok/s	354.19 tok/s

The baseline 1K row includes cold expert-cache/thrashing behavior and should not be read as a universal 5.3x gain. The stable non-speculative hybrid gain was +23.8% at 8K and +21.8% on repeated code. MTP + n-gram reached 67.23 tok/s when draft acceptance was high; on less predictable short output it was 3.2% slower than the hybrid without speculation. Observed draft acceptance was about 45.5-51.9% for non-repetitive generations and 100% for repeated code.

Laptop resource peak	Baseline Q4_K_M	Hybrid, no spec	Hybrid, MTP + n-gram
GPU temperature	71 C	69 C	69 C
GPU utilization	98%	98%	98%
VRAM used	2,342 MiB	2,234 MiB	2,672 MiB
GPU power	49.34 W	45.80 W	46.47 W
Whole-system RAM used	44.48 GiB	42.64 GiB	42.77 GiB

The same four seeded Orion coding workspaces were then executed with approval=auto, a 24-step cap and a 900-second per-task timeout. Deterministic local verifiers evaluated the resulting files.

Orion coding scenario	Baseline quality	Hybrid quality	Baseline time	Hybrid time	Hybrid time reduction
Java service coverage	80%, 8/10	80%, 8/10	323.9 s	322.4 s	0.4%
Kafka Node order pipeline	75%, 6/8	75%, 6/8	282.6 s	185.7 s	34.3%
Rabbit retry / DLQ	75%, 6/8	75%, 6/8	466.0 s	324.7 s	30.3%
Kafka Java outbox / idempotency	80%, 8/10	80%, 8/10	382.4 s	313.4 s	18.1%
Total / average	77.5%, 28/36	77.5%, 28/36	1,454.9 s	1,146.2 s	21.2%

The Windows laptop A/B therefore measured identical verifier quality and a 1.269x end-to-end speedup (308.7 seconds saved) for the hybrid. Both artifacts missed the same Orion cli_exit_zero and final_contract checks: their workspace changes passed identically, but neither produced the harness's required final changed / verified / not verified / risks contract. This was not an API, context or infrastructure failure and does not indicate a hybrid quality regression.

These laptop numbers are intentionally modest in scope: one Windows machine, 16K context, 2/2/3 raw repetitions, 128 generated tokens, and four coding scenarios. The 60-67 tok/s range is workload-dependent; a realistic production expectation for ordinary agent traffic is roughly a 20-30% improvement.

Recommended llama-server profile

Use llama.cpp b9994 or a compatible build with MTP support:

GGML_VK_ALLOW_GRAPHICS_QUEUE=1 \
GGML_VK_DISABLE_COOPMAT=1 \
GGML_VK_MAX_NODES_PER_SUBMIT=50 \
llama-server \
  --model ornith-1.0-35b-MTP-graft-down-Q4_0.gguf \
  --alias ornith-1.0-35b \
  --ctx-size 131072 --parallel 1 \
  --batch-size 2048 --ubatch-size 1024 \
  --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 \
  --n-gpu-layers all --device Vulkan0 --no-mmap \
  --spec-type ngram-mod,draft-mtp \
  --spec-draft-n-max 2 --spec-draft-n-min 1 --spec-draft-p-min 0.20 \
  --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 \
  --spec-ngram-mod-n-match 24

These environment variables and throughput figures are specific to the tested AMD Strix Halo Vulkan system. Other backends and GPUs require their own sweep.

Validation

790 Vulkan backend operation tests passed for every tested K-quant row-merge value (1, 2, 4, 8; 3,160 test cases total);
identical aggregate quality on the same 43 tasks;
Windows 11 / RTX 4060 Laptop A/B: identical 28/36 Orion checks with 21.2% lower hybrid wall time across the same four coding scenarios;
GGUF checksum verified before publication;
131,072-token server context and OpenAI-compatible API verified in production.

Reproducibility

The source fork, tensor manifest, production Compose profile, benchmark JSON, and detailed research materials are maintained separately in the author's private source repository. This Hugging Face repository intentionally contains only the model, benchmark summary, and this concise deployment card.

Downloads last month: -

GGUF

Model size

36B params

Architecture

qwen35moe

Hardware compatibility

4-bit

Model tree for petr567/Ornith-1.0-35B-MTP-Strix-Halo-Hybrid-GGUF

Base model

deepreinforce-ai/Ornith-1.0-35B

Quantized

(149)

this model