Instructions to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1",
	filename="ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Use Docker

docker model run hf.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

LM Studio
Jan

vLLM

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Ollama
How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Ollama:
```
ollama run hf.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
```

Unsloth Studio

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 to start chatting

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Docker Model Runner:
```
docker model run hf.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
```

Lemonade

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Run and chat with the model

lemonade run user.Ornith-1.0-35B-GGUF-llamacpp-tp1-Q4_K_M

List all available models

lemonade list

Ornith-1.0-35B — GGUF (llama.cpp, single-GPU `tp=1`)

Single-GPU llama.cpp GGUF package for deepreinforce-ai/Ornith-1.0-35B. The supported serving policy is tp=1 — one model copy per GPU. Multi-GPU tensor-parallel serving is intentionally out of scope for this version.

This release ships six body quants (Q3_K_M → Q8_0) plus an integrated IQ4_XS-MTP graft that adds a native multi-token-prediction (MTP) draft head for low-concurrency speculative decode. Quant quality is measured against the upstream BF16 GGUF with a native llama.cpp next-token top-64 KL-divergence probe over 32 coding prompts.

TL;DR — pick an artifact

Use case	Recommended artifact	Key numbers
Default serving speed	`ornith-1.0-35b-Q4_K_M.gguf`	19.71 GiB on disk, 21.31 GiB loaded VRAM, 243.3 tok/s c1, 655.6 tok/s c16
Lowest memory	`ornith-1.0-35b-Q3_K_M.gguf`	15.61 GiB on disk, 17.27 GiB loaded VRAM, 240.5 tok/s c1, 493.0 tok/s c16
Middle footprint	`ornith-1.0-35b-IQ4_XS.gguf`	17.64 GiB on disk, 19.34 GiB loaded VRAM, 0.1426 mean top-64 KLD nats
Highest fidelity / footprint	`ornith-1.0-35b-Q6_K.gguf`	26.56 GiB on disk, 28.03 GiB loaded VRAM, 0.0165 mean top-64 KLD nats, 32/32 top-1
Native low-concurrency MTP	`ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf`	19.6 GB decimal, 0.0731 mean top-64 KLD (29/32), c1/128 adaptive MTP 319.53 tok/s, acceptance `(0.953, 0.865)`

MTP graft note: active MTP is byte-for-byte identical to target-only at the next-token (API-visible) level, and gives ~1.3× single-stream decode. It is not bit-exact to target-only over long deterministic sequences — see MTP. Use target-only for strict reproduction, MTP for throughput on open-ended work.

Quantization & fidelity

Fidelity is KL(P_bf16 || P_candidate) over next-token top-64 distributions, 32 coding prompts, n_predict=1, temperature=-1, n_probs=64, keyed by token ID. Lower KLD and higher greedy top-1 agreement are better.

Primary fidelity table

Artifact	Mean KLD (nats)	Greedy top-1 †
`Q8_0`	0.0108	31/32 (96.9%)
`Q6_K`	0.0165	32/32 (100.0%)
`Q5_K_M`	0.0354	30/32 (93.8%)
`IQ4_XS-MTP` (graft)	0.0731382442	29/32 (90.6%)
`Q4_K_M`	0.0864	29/32 (90.6%)
`IQ4_XS`	0.1426	27/32 (84.4%)
`Q3_K_M`	0.3620	27/32 (84.4%)

Fidelity ladder — the integrated IQ4_XS-MTP graft (purple) lands between Q5_K_M and Q4_K_M, i.e. better next-token fidelity than its Q4_K_M neighbor.

† Top-1 vs mean KLD are different axes. Top-1 counts how often the single most-likely (argmax) token matches BF16's, over the 32 prompts; mean KLD measures divergence across the full top-64 distribution. Two quants can share a top-1 count yet differ a lot in KLD — e.g. Q3_K_M and IQ4_XS are both 27/32 on top-1, but Q3_K_M's distribution diverges ~2.5× more (0.3620 vs 0.1426 nats). Neither value is a typo.

The integrated IQ4_XS-MTP graft is 48.70% lower mean KLD than the plain IQ4_XS body (0.0731382442 vs 0.1425748206), and improves greedy top-1 from 27/32 to 29/32.
Active draft-mtp ≡ target-only at the next-token level: their top-64 next-token distributions are identical (KLD 0.0, 32/32) for the one-token API-visible probe.

Body-quant distribution detail (secondary)

Full per-quant distribution from the corrected top-64 probe (same 32 prompts). Mean / P50 / P95 / Max are explicit nats statistics of the per-prompt top-64 KLD; in this run the BF16 top-64 distribution captured mean probability mass 0.999965, so the top-64 approximation captured essentially all next-token mass.

Quant	GGUF GiB	KLD-probe VRAM GiB	Mean nats	P50 nats	P95 nats	Max nats	Top-1 †
Q3_K_M	15.61	16.32	0.3620	0.2548	1.1077	1.3730	27/32 (84.4%)
IQ4_XS	17.64	19.34	0.1426	0.0868	0.3195	0.6586	27/32 (84.4%)
Q4_K_M	19.71	20.35	0.0864	0.0379	0.2877	0.4503	29/32 (90.6%)
Q5_K_M	23.03	23.61	0.0354	0.0235	0.0943	0.2497	30/32 (93.8%)
Q6_K	26.56	27.07	0.0165	0.0092	0.0513	0.0586	32/32 (100.0%)
Q8_0	34.37	34.77	0.0108	0.0052	0.0440	0.0590	31/32 (96.9%)

The IQ4_XS-MTP graft uses the same probe on a different breakpoint context; its distribution detail (P50 0.0492811974 / P95 0.1593743642 / Max 0.3348258511) is in the MTP section. Source: benchmarks/kld-quant-vs-bf16-top64.md.

Serving performance

Aggregate decode throughput and p95 time-to-first-token across concurrency, llama.cpp tp=1, from scripts/bench_openai.py --stream --max-tokens 256 (0 failed requests per row). Q4_K_M is the speed pick; Q5_K_M nearly ties it at c16.

Quant	c1 tok/s	c1 p95 TTFT ms	c4 tok/s	c4 p95 TTFT ms	c8 tok/s	c8 p95 TTFT ms	c16 tok/s	c16 p95 TTFT ms
Q3_K_M	240.5	77.9	422.0	170.7	464.4	344.9	493.0	493.7
IQ4_XS	234.1	75.1	297.7	159.7	411.5	330.2	476.0	541.5
Q4_K_M	243.3	76.3	458.3	192.3	615.0	361.8	655.6	650.0
Q5_K_M	236.7	75.1	311.0	198.8	439.0	383.6	638.6	620.4
Q6_K	225.9	76.8	295.8	194.2	409.6	394.8	603.3	657.4
Q8_0	208.5	76.9	281.5	190.8	405.8	389.1	601.4	725.8

Profile note: the table above is a short-context profile — CTX_SIZE=8192 PARALLEL=16 exposes n_ctx = 512/slot, with ~23-token prompts and 256-token generations. Source: benchmarks/llamacpp-quant-benchmarks.md. For how prefill latency scales with prompt length, see long-context TTFT below.

Long-context TTFT (single stream)

p95 time-to-first-token vs prompt/context length, single GPU tp=1, single stream (CTX_SIZE=131072 PARALLEL=1, n_ctx/slot=131072, exact prompt token counts, 140 rows/quant, 0 failures).

Context tokens	Q4_K_M p50/p95 ms	Q4_K_M tok/s	IQ4_XS p50/p95 ms	IQ4_XS tok/s	MTP-graft p50/p95 ms	MTP-graft tok/s
512	91.8 / 94.1	188.3	87.1 / 88.7	177.4	86.0 / 87.8	184.0
1024	169.6 / 172.2	145.8	159.9 / 161.6	139.9	157.6 / 159.0	155.9
2048	341.8 / 346.1	97.9	316.1 / 318.7	112.0	307.6 / 308.9	113.9
4096	699.2 / 702.2	67.2	647.0 / 653.4	70.7	625.7 / 631.3	69.0
8192	1447.5 / 1458.0	33.7	1342.9 / 1348.5	39.8	1297.4 / 1304.0	40.9
16384	3009.7 / 3030.4	19.5	2793.2 / 2804.0	20.9	2709.8 / 2737.3	21.5
32768	6302.4 / 6313.0	9.7	5829.9 / 5853.1	10.5	5673.5 / 5696.6	10.7

p95 TTFT rises with prompt length (prefill cost) to ~6.3 s at 32k tokens for Q4_K_M; the IQ4_XS body and IQ4_XS-MTP graft prefill slightly faster at every length. Decode throughput falls from ~180–190 tok/s at 512 tokens to ~10 tok/s at 32k as the KV cache grows. Source: benchmarks/llamacpp-longctx-ttft.md.

Multi-token prediction (MTP)

The integrated IQ4_XS-MTP-graft-headQ6 artifact carries a native MTP draft head on the IQ4_XS body. The release recommendation is adaptive: use MTP for low-concurrency / single-user requests and keep LLAMA_SPEC_MAX_DRAFTING_SLOTS=1 so saturated batches fall back to target-only throughput. The MTP catalog validates the published MTP profiles.

Next-token quality

Active MTP does not change the one-token, API-visible next-token distribution. Mean BF16 KLD is 0.0731382442 nats over the 32-prompt top-64 probe (P50 0.0492811974 / P95 0.1593743642 / Max 0.3348258511), top-1 29/32. The active draft-mtp vs target-only graft next-token KLD is 0.0 (32/32 identical).

Sequence-level behavior

A deterministic 8 prompt × 64 token probe (n_predict=64, temperature=-1, n_probs=64) compares the same GGUF running active native draft-mtp against target-only.

Runtime	Client agg tok/s	Server decode tok/s	Draft acceptance	Exact 64-tok seqs	Token-position match
Target-only graft	172.57	~210 (per-request timing)	n/a	8/8 baseline	n/a
Fast active `draft-mtp`, `n_max=2`	233.81	325.70 (/metrics)	310/378 = 82.01%; per-position `(0.884, 0.747)`	6/8	478/512 = 93.36%
`LLAMA_SPEC_VERIFY_SEQUENTIAL=1`	164.34	204.88 (/metrics)	313/375 = 83.47%; per-position `(0.884, 0.772)`	5/8	455/512 = 88.87%

Fast active MTP gives ~1.35× client throughput (233.81 vs 172.57) and server decode 325.70 vs ~210, with 82.01% draft acceptance.
It matches 6/8 sequences exactly and 478/512 = 93.36% of token positions; the two non-exact sequences first diverge at positions 60 and 34.
The LLAMA_SPEC_VERIFY_SEQUENTIAL=1 variant is both slower (164.34 / 204.88) and matches less (5/8, 88.87%; divergences at 25, 43, 60) — i.e. the fast verifier is here both faster and more target-matching.

Note: MTP self-speculation is not bit-exact to target-only over long deterministic generations. For workloads that require exact target reproduction, run target-only; for throughput on open-ended work, MTP wins. MTP is a single-user / low-concurrency win — it is not faster on saturated batches, which is why the adaptive throttle falls back to target-only.

Sequence logprob comparability (API limitation)

llama.cpp omits candidate top_logprobs for most accepted speculative tokens, so a naive "sequence KLD" looks enormous — that is an API/logprob-availability artifact, not model quality. On the comparable subset where the emitted token also matches, KLD is ≈ 0.001 nats (near zero):

Runtime	Paired positions	Comparable top-logprobs	Missing	Comparable same-token mean KLD (nats)	Max (nats)
Fast active `draft-mtp` vs target-only	512	12	500	0.0011215739	0.0040239103
Sequential verifier vs target-only	512	10	502	0.0012293172	0.0040239103

Full sequence KLD cannot be computed without changing the runtime to return target verifier logprobs for accepted speculative tokens.

Draft-head distillation provenance

The meaningful KL-style metric for a standalone MTP draft head is teacher KL against cached target hidden states, not standard base-model next-token KLD. The draft head was trained and measurably improved across checkpoints — teacher KL fell 1.92 → 1.50 → 0.76 and teacher argmax top-1 rose 0.54 → 0.69 → 0.89.

Draft checkpoint	Tokens	Windows	Teacher KL	Teacher argmax top-1	Teacher argmax top-5	Gold top-1
`mtp-distill-kl-step500`	22,637	128	1.9222025748	0.5360250917	0.7881786456	0.4425939833
`snapshots-iq4-live-accepted-a/step-1000`	58,754	413	1.5038724942	0.6876808388	0.8368451510	0.6764816013
`snapshots-iq4-live-allrows-b/step-1000`	32,264	2,048	0.7591610373	0.8908070915	0.9389722291	0.8076183982

Teacher KL is measured against cached target hidden states (not base next-token KLD). These training artifacts live under artifacts/mtp/ locally and are referenced by name as provenance — they are not committed in this HF repo.

Run with llama.cpp

Serve a body quant on GPU0 (tp=1):

QUANT=Q4_K_M PORT=8000 CTX_SIZE=8192 PARALLEL=16 REASONING=off \
  scripts/serve_llamacpp_gpu0.sh

Serve the integrated MTP graft for low-concurrency speculative decode (keep adaptive throttling so saturated batches fall back to target-only):

QUANT=IQ4_XS-MTP-graft-headQ6 PORT=8000 CTX_SIZE=8192 PARALLEL=1 \
  CACHE_RAM=0 REASONING=off LLAMA_SPEC_MAX_DRAFTING_SLOTS=1 \
  scripts/serve_llamacpp_gpu0.sh

REASONING=off is the default because the model otherwise spends simple coding prompts in reasoning_content before producing final content. See benchmarks/llamacpp-q4-reasoning-off-fix.md. Serving / quant / MTP profile catalogs: configs/serving_profiles.yaml, configs/quant_artifacts.yaml, configs/mtp_profiles.yaml.

Benchmark environment

Hardware: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 97,887 MiB VRAM, single GPU (tp=1). Driver 580.159.03, CUDA runtime reported by driver 13.0.

Dedicated MTP KLD eval config (2026-06-28): local llama.cpp CUDA server, single RTX PRO 6000 Blackwell 96GB, tp=1, CTX_SIZE=8192, PARALLEL=1, CACHE_RAM=0, REASONING=off.

llama.cpp build/commit: 050ee92d04c2e1f639025786dea701c70e7d4204 (pinned for the long-context TTFT run; earlier KLD/throughput sweeps were not separately pinned).
Reasoning mode: all benchmarks in this card ran with REASONING=off.

Provenance & reproducibility

Figures: regenerate with scripts/make_charts.py → assets/02_fidelity_ladder.png, assets/03_throughput_tps.png, assets/04_ttft_p95.png, assets/05_mtp_draft_head_distill.png, assets/06_mtp_tps.png.
Serving throughput / TTFT: benchmarks/llamacpp-quant-benchmarks.md.
Body-quant fidelity: benchmarks/kld-quant-vs-bf16-top64.md.
Dedicated MTP KLD / sequence eval (2026-06-28): benchmarks/mtp-kld-eval-2026-06-28.md and benchmarks/mtp-dedicated-kld-evaluation.md, with raw evidence under benchmarks/raw/. Includes the sequence-level probe and its logprob-availability limitation (full sequence KLD is not computable from the current llama.cpp API response).
Adaptive MTP serving profile: benchmarks/llamacpp-iq4-xs-mtp-graft-adaptive-profile.md.
Long-context TTFT: benchmarks/llamacpp-longctx-ttft.md.

Downloads last month: 25,507

GGUF

Model size

35B params

Architecture

qwen35moe

Hardware compatibility

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Model tree for LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1

Base model

deepreinforce-ai/Ornith-1.0-35B

Quantized

(149)

this model

Ornith-1.0-35B — GGUF (llama.cpp, single-GPU tp=1)

TL;DR — pick an artifact

Quantization & fidelity

Primary fidelity table

Body-quant distribution detail (secondary)

Serving performance

Long-context TTFT (single stream)

Multi-token prediction (MTP)

Next-token quality

Sequence-level behavior

Sequence logprob comparability (API limitation)

Draft-head distillation provenance

Run with llama.cpp

Benchmark environment

Provenance & reproducibility

Model tree for LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1

Ornith-1.0-35B — GGUF (llama.cpp, single-GPU `tp=1`)