Instructions to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1",
	filename="ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Use Docker

docker model run hf.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

LM Studio
Jan

vLLM

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Ollama
How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Ollama:
```
ollama run hf.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
```

Unsloth Studio

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 to start chatting

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Docker Model Runner:
```
docker model run hf.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
```

Lemonade

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Run and chat with the model

lemonade run user.Ornith-1.0-35B-GGUF-llamacpp-tp1-Q4_K_M

List all available models

lemonade list

Ornith-1.0-35B-GGUF-llamacpp-tp1 / benchmarks /mtp-dedicated-kld-evaluation.md

LordNeel

Add dedicated MTP KLD and sequence evaluation

4a33943 verified 24 days ago

preview code

Raw

History Blame Contribute Delete

6.58 kB

Dedicated MTP KLD Evaluation

Date: 2026-06-28

Artifact under test: artifacts/quant/ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf

Backend: local llama.cpp CUDA server, tp=1, one RTX PRO 6000 Blackwell 96GB, CTX_SIZE=8192, PARALLEL=1, CACHE_RAM=0, REASONING=off.

What Was Measured

Two different checks were run.

Next-token top-64 KLD over 32 coding prompts, comparable with the existing BF16-vs-quant table. This uses n_predict=1, temperature=-1, n_probs=64.
Sequence-level 8 prompt x 64 token deterministic generations to exercise real native draft-mtp acceptance and compare active MTP against the same GGUF running target-only.

Important limitation: llama.cpp does not return full candidate top_logprobs for most accepted speculative tokens in the sequence-level response. Those positions are reported as missing candidate logprob positions, not as valid KLD measurements.

Next-Token KLD

Approximate KL(P_bf16 || P_candidate) over top-64 next-token distributions.

Candidate	Paired prompts	Mean KLD nats	Mean KLD bits	P50 nats	P95 nats	Max nats	Top-1 match
Integrated IQ4_XS-MTP graft, target-only	32	0.0731382442	0.1055161823	0.0492811974	0.1593743642	0.3348258511	29/32
Integrated IQ4_XS-MTP graft, active `draft-mtp`	32	0.0731382442	0.1055161823	0.0492811974	0.1593743642	0.3348258511	29/32
Active `draft-mtp` vs target-only graft	32	0.0	0.0	0.0	0.0	0.0	32/32

The active draft-mtp and target-only next-token distributions are byte-for-byte identical for this one-token API-visible probe. The integrated MTP graft's BF16 drift is also lower than the previous plain IQ4_XS body capture:

Candidate	Mean KLD nats	Top-1 match
Plain IQ4_XS vs BF16, prior body KLD run	0.1425748206	27/32
Integrated IQ4_XS-MTP graft vs BF16, this run	0.0731382442	29/32

That is a 48.70% lower mean KLD than the prior plain IQ4_XS capture.

Sequence-Level MTP Probe

Shape: first 8 coding eval prompts, n_predict=64, temperature=-1, n_probs=64, target-only baseline vs active native draft-mtp.

Runtime	Client aggregate tok/s	Server decode tok/s	Draft acceptance	Exact 64-token sequences	Token-position match
Target-only graft	172.57	about 210 from server per-request timing	n/a	8/8 baseline	n/a
Fast active `draft-mtp`, `n_max=2`	233.81	325.70 from `/metrics`	310/378 = 82.01%; per-position `(0.884, 0.747)`	6/8	478/512 = 93.36%
`LLAMA_SPEC_VERIFY_SEQUENTIAL=1`	164.34	204.88 from `/metrics`	313/375 = 83.47%; per-position `(0.884, 0.772)`	5/8	455/512 = 88.87%

Fast active MTP first divergence positions:

Source id	First mismatch position	Matched prefix tokens
`630213`	none	64
`373803`	none	64
`148184`	none	64
`631801`	none	64
`211640`	none	64
`90023`	none	64
`611646`	60	60
`502142`	34	34

Sequential verifier first divergence positions:

Source id	First mismatch position	Matched prefix tokens
`630213`	none	64
`373803`	none	64
`148184`	none	64
`631801`	none	64
`211640`	25	25
`90023`	43	43
`611646`	60	60
`502142`	none	64

Sequence Logprob Comparability

The active-MTP sequence response omitted candidate top-logprobs for most accepted draft tokens.

Runtime	Paired token positions	Positions with comparable top-logprobs	Missing candidate top-logprobs	Comparable same-token mean KLD nats	Comparable same-token max KLD nats
Fast active `draft-mtp` vs target-only	512	12	500	0.0011215739	0.0040239103
Sequential verifier vs target-only	512	10	502	0.0012293172	0.0040239103

If missing candidate top-logprobs are treated as tail mass, the apparent sequence KLD becomes extremely large. That number is not a valid model-quality KLD; it is an API/logprob availability artifact. The valid comparable subset is near zero when the emitted token also matches.

MTP Draft-Head KL Context

The meaningful KL-style metric for standalone MTP draft heads is teacher KL against cached target hidden states, not standard base-model next-token KLD. Previously measured draft-head evals:

Draft checkpoint	Cache	Tokens	Windows	Teacher KL	Teacher argmax top-1	Teacher argmax top-5	Gold top-1
`artifacts/mtp/ornith-1.0-35b-mtp-distill-kl-step500`	`artifacts/mtp/cache-coding-4096x512-merged`	22,637	128	1.9222025748	0.5360250917	0.7881786456	0.4425939833
`artifacts/mtp/snapshots-llamacpp-iq4-live-accepted-a/step-1000`	`artifacts/mtp/cache-llamacpp-iq4-live-accepted-selfgen1000-a`	58,754	413	1.5038724942	0.6876808388	0.8368451510	0.6764816013
`artifacts/mtp/snapshots-llamacpp-iq4-live-allrows-b/step-1000`	`artifacts/mtp/cache-llamacpp-iq4-live-allrows-livea1000-b`	32,264	2,048	0.7591610373	0.8908070915	0.9389722291	0.8076183982

Conclusion

The integrated IQ4_XS-MTP graft is clean on next-token distribution quality: active MTP does not change the API-visible one-token distribution, and its mean BF16 KLD is 0.0731382442 nats over the 32-prompt top-64 probe.

The longer sequence probe is more mixed. Fast active MTP gives real single-user speedup in this 8 x 64 test and high acceptance, but it is not fully target-equivalent over deterministic 64-token generations. The response also omits candidate top-logprobs for most accepted draft tokens, so full sequence KLD cannot be computed from the current llama.cpp API response without changing the runtime to return target verifier logprobs for accepted speculative tokens.

Raw files:

runs/kld-native-iq4-xs-mtp-graft-target-top64-32prompts.json
runs/kld-native-iq4-xs-mtp-graft-target-vs-bf16.json
runs/kld-native-iq4-xs-mtp-graft-active-top64-32prompts.json
runs/kld-native-iq4-xs-mtp-graft-active-vs-bf16.json
runs/kld-native-iq4-xs-mtp-graft-active-vs-target-only.json
runs/kld-seq-iq4-xs-mtp-graft-target-top64-8prompts-64tokens.json
runs/kld-seq-iq4-xs-mtp-graft-active-top64-8prompts-64tokens.json
runs/kld-seq-iq4-xs-mtp-graft-active-vs-target-only-8prompts-64tokens.json
runs/kld-seq-iq4-xs-mtp-graft-seqverify-top64-8prompts-64tokens.json
runs/kld-seq-iq4-xs-mtp-graft-seqverify-vs-target-only-8prompts-64tokens.json