Instructions to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1",
	filename="ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Use Docker

docker model run hf.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

LM Studio
Jan

vLLM

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Ollama
How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Ollama:
```
ollama run hf.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
```

Unsloth Studio

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 to start chatting

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Docker Model Runner:
```
docker model run hf.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
```

Lemonade

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Run and chat with the model

lemonade run user.Ornith-1.0-35B-GGUF-llamacpp-tp1-Q4_K_M

List all available models

lemonade list

Ornith-1.0-35B-GGUF-llamacpp-tp1 / MODEL_INVESTIGATION_HARNESS.md

LordNeel

Add machine-readable serving profile catalog

7c01a4f verified 26 days ago

preview code

Raw

History Blame

12.8 kB

Model Investigation Harness

This document is the reusable entry point for applying the Ornith 35B audit, serving, quantization, MTP, and SFT gates to another model.

Current Ornith Snapshot

Scope: single-GPU serving only, tp=1; use the two GPUs as independent lanes, not as one serving replica.
Published HF package: LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1.
Fast GGUF serving default: Q4_K_M through llama.cpp with REASONING=off.
Lowest-memory validated GGUF: local Q3_K_M, 15.61 GiB on disk, 17.27 GiB loaded VRAM in the short-context benchmark profile.
Middle-footprint validated GGUF: local IQ4_XS, 17.64 GiB on disk, 19.34 GiB loaded VRAM in the KLD profile.
Integrated MTP artifact: artifacts/quant/ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf.
Recommended MTP serving policy: LLAMA_SPEC_MAX_DRAFTING_SLOTS=1, SPEC_DRAFT_N_MAX=2, adaptive low concurrency only. This keeps the c1 MTP win and avoids the saturated c16 loss.
SFT status: production LoRA run stopped by user request. Latest durable checkpoint is checkpoint-8000; no SFT adapter is released.

Core Audit Flow

Run metadata/tokenizer/config checks first. This does not load full weights.

cd /home/ripper/ornith-35b-lab
.venv-lab/bin/python scripts/run_model_investigation.py \
  --model deepreinforce-ai/Ornith-1.0-35B \
  --gguf deepreinforce-ai/Ornith-1.0-35B-GGUF \
  --output runs/investigation-ornith-metadata.json

.venv-lab/bin/python scripts/summarize_investigation.py \
  runs/investigation-ornith-metadata.json \
  --output runs/investigation-ornith-metadata.md

.venv-lab/bin/python scripts/check_audit_gates.py \
  runs/investigation-ornith-metadata.json

When an OpenAI-compatible backend is running, add the behavior suite to the same combined report:

.venv-lab/bin/python scripts/run_model_investigation.py \
  --model deepreinforce-ai/Ornith-1.0-35B \
  --gguf deepreinforce-ai/Ornith-1.0-35B-GGUF \
  --base-url http://127.0.0.1:8002/v1 \
  --served-model Ornith-1.0-35B-GGUF-Q4_K_M \
  --output runs/investigation-ornith-live.json

Behavior checks live in configs/model_behavior_suite.yaml. They cover coding correctness, JSON formatting, chat-template leakage, decode corruption, and optional tool-call behavior.

Aggregate behavior and investigation reports into a reusable regression matrix:

.venv-lab/bin/python scripts/summarize_behavior_matrix.py \
  --input q4_live_investigation=runs/investigation-ornith-q4-live.json \
  --input q4_reasoning_off=runs/behavior-llamacpp-q4-reasoning-off.json \
  --input q4_script_default=runs/behavior-llamacpp-q4-script-default.json \
  --input q4_reasoning_on_bad=runs/behavior-llamacpp-q4.json \
  --input q3_k_m=runs/behavior-llamacpp-q3-k-m-parallel16.json \
  --input iq4_xs=runs/behavior-llamacpp-iq4-xs-parallel16.json \
  --input lora_checkpoint_8000_bad=runs/eval-lora-behavior-checkpoint-8000-fixed.json \
  --output runs/behavior-regression-matrix.json \
  --markdown-output runs/behavior-regression-matrix.md

The matrix treats high and critical issues as blocking by default and compares each artifact across the same case/check keys. The current Ornith run has 5 passing artifacts and 2 intentionally failing artifacts: the old reasoning-on Q4 profile and the stopped checkpoint-8000 LoRA adapter. Use this matrix after each backend, quant, MTP, or adapter change to catch regressions without manually diffing individual probe files.

Build a release-readiness report from the HF verifier, manifest, behavior matrix, quant/KLD docs, MTP profile, and SFT state:

.venv-lab/bin/python scripts/build_release_readiness_report.py \
  --allow-unreleased-sft \
  --required-doc hf_upload/README.md \
  --required-doc hf_upload/RELEASE_AUDIT.md \
  --required-doc hf_upload/PROJECT_STATUS_AUDIT.md \
  --required-doc hf_upload/MODEL_INVESTIGATION_HARNESS.md \
  --required-doc hf_upload/SFT_REMEDIATION_PLAN.md \
  --output runs/release-readiness-report.json \
  --markdown-output runs/release-readiness-report.md \
  --fail-on-serving-not-ready

For Ornith this reports serving_release_status=pass and full_project_status=incomplete. That distinction is intentional: the GGUF serving release is usable, while the fine-tuning/post-training deliverable remains unreleased after the stopped SFT run.

Validate the machine-readable serving profile catalog:

.venv-lab/bin/python scripts/validate_serving_profiles.py \
  --catalog configs/serving_profiles.yaml \
  --manifest hf_upload/ARTIFACT_MANIFEST.json \
  --hf-root hf_upload \
  --output runs/serving-profile-validation.json \
  --markdown-output runs/serving-profile-validation.md \
  --fail-on-error

The current catalog validates 5 single-GPU llama.cpp profiles with 43/43 checks passing: Q4_K_M short-context speed, Q4_K_M coding-context default, Q3_K_M lowest memory, IQ4_XS middle footprint, and adaptive IQ4_XS MTP graft.

Serving And Benchmark Flow

Start a single-GPU llama.cpp server:

QUANT=Q4_K_M PORT=8002 CTX_SIZE=8192 PARALLEL=16 \
  scripts/serve_llamacpp_gpu0.sh

Run a throughput sweep:

for c in 1 4 8 16; do
  .venv-lab/bin/python scripts/bench_openai.py \
    --base-url http://127.0.0.1:8002/v1 \
    --model Ornith-1.0-35B-GGUF-Q4_K_M \
    --concurrency "$c" \
    --max-tokens 256 \
    --stream \
    --output "runs/llamacpp-q4-k-m-c${c}-256.jsonl"
done

.venv-lab/bin/python scripts/summarize_benchmarks.py \
  runs/llamacpp-q4-k-m-c*-256.jsonl

After every backend/profile change, run the OpenAI-compatible contract probe:

.venv-lab/bin/python scripts/probe_openai_contract.py \
  --base-url http://127.0.0.1:8002/v1 \
  --model Ornith-1.0-35B-GGUF-Q4_K_M \
  --output runs/server-contract.json

Quantization Gates

For GGUF artifacts, verify materialization and serving behavior before treating a quant as usable:

.venv-lab/bin/python scripts/audit_gguf_vllm_materialization.py \
  --model deepreinforce-ai/Ornith-1.0-35B \
  --gguf deepreinforce-ai/Ornith-1.0-35B-GGUF:Q4_K_M \
  --output runs/gguf-materialization-q4.json

.venv-lab/bin/python scripts/probe_model_behavior.py \
  --base-url http://127.0.0.1:8002/v1 \
  --model Ornith-1.0-35B-GGUF-Q4_K_M \
  --output runs/behavior-llamacpp-q4.json

For quant drift, compare next-token top-K distributions against BF16:

.venv-lab/bin/python scripts/collect_next_token_logprobs.py \
  --base-url http://127.0.0.1:8002 \
  --served-model Ornith-1.0-35B-GGUF-Q4_K_M \
  --messages-jsonl data/eval/coding_sft_eval_messages.jsonl \
  --output runs/kld-native-q4-k-m-top64-32prompts.json

.venv-lab/bin/python scripts/compare_kld_logprobs.py \
  --baseline runs/kld-native-bf16-top64-32prompts.json \
  --candidate runs/kld-native-q4-k-m-top64-32prompts.json \
  --output runs/kld-native-q4-k-m-vs-bf16.json

Current Ornith KLD summary is in runs/kld-quant-vs-bf16-top64.md and the HF copy is hf_upload/benchmarks/kld-quant-vs-bf16-top64.md.

MTP Gates

First determine whether the source checkpoint truly contains MTP tensors:

HF_SNAPSHOT=artifacts/source/ornith-1.0-35b-index
.venv-lab/bin/huggingface-cli download deepreinforce-ai/Ornith-1.0-35B \
  config.json model.safetensors.index.json \
  --local-dir "$HF_SNAPSHOT"

.venv-lab/bin/python scripts/check_mtp_support.py \
  --hf-snapshot "$HF_SNAPSHOT" \
  --gguf artifacts/quant/ornith-1.0-35b-IQ4_XS.gguf \
  --output runs/mtp-support-model.json

For integrated graft MTP on Ornith, the current recommended serving profile is:

LLAMA_SPEC_MAX_DRAFTING_SLOTS=1 \
LLAMA_MTP_FAST_BACKEND_SAMPLE=1 \
LLAMA_MTP_DRAFT_TOP_K=1 LLAMA_MTP_DRAFT_TOP_P=1 LLAMA_MTP_DRAFT_TEMP=1 \
MODEL_PATH=artifacts/quant/ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf \
PORT=8002 CTX_SIZE=8192 PARALLEL=16 \
SPEC_TYPE=draft-mtp SPEC_DRAFT_N_MAX=2 SPEC_DRAFT_N_MIN=0 \
SPEC_DRAFT_BACKEND_SAMPLING=1 \
  scripts/serve_llamacpp_gpu0.sh

MTP must be judged on both acceptance and throughput. For Ornith, always-on MTP has high acceptance but loses at saturated c16; adaptive throttling is the usable profile. Current evidence is in:

hf_upload/benchmarks/llamacpp-iq4-xs-mtp-graft-adaptive-profile.md
hf_upload/patches/llamacpp-mtp-adaptive-serving-and-recurrent-rollback.patch
hf_upload/patches/llamacpp-qwen35moe-mtp-recurrent-row-index.patch

SFT Gates

SFT data must pass validation before loading the 35B model:

.venv-lab/bin/python scripts/audit_sft_format.py \
  --input data/train/coding_sft_prod_messages.jsonl \
  --input data/eval/coding_sft_prod_eval_messages.jsonl \
  --input data/train/coding_sft_prod.jsonl \
  --input data/eval/coding_sft_prod_eval.jsonl \
  --output runs/sft-format-audit-prod.json

For the stopped Ornith SFT run, the CPU-side repair path is:

.venv-lab/bin/python scripts/build_sft_repair_set.py \
  --train-size 4096 \
  --eval-size 512 \
  --seed 29 \
  --train-output data/train/coding_sft_repair_messages.jsonl \
  --eval-output data/eval/coding_sft_repair_eval_messages.jsonl \
  --report-output runs/sft-repair-set.json

.venv-lab/bin/python scripts/build_sft_code_only_from_fences.py \
  --train-size 20000 \
  --eval-size 1000 \
  --seed 41 \
  --train-output data/train/coding_sft_code_only_messages.jsonl \
  --eval-output data/eval/coding_sft_code_only_eval_messages.jsonl \
  --report-output runs/sft-code-only-extract.json

.venv-lab/bin/python scripts/build_sft_blend.py \
  --source data/train/coding_sft_prod_messages.jsonl=40000:base_coding \
  --source data/train/coding_sft_repair_messages.jsonl=4096:strict_repair \
  --output data/train/coding_sft_prod_repair_messages.jsonl \
  --report-output runs/sft-repair-blend-train.json \
  --seed 31

See configs/sft_repair_blend.yaml for the full train/eval render, validation, audit, and format-gate command sequence.

.venv-lab/bin/python scripts/check_sft_format_gates.py \
  --audit-report runs/sft-format-audit-prod-repair.json \
  --config configs/sft_format_gates.yaml \
  --profile strict_repair_messages \
  --output runs/sft-format-gate-strict-repair.json

.venv-lab/bin/python scripts/check_sft_format_gates.py \
  --audit-report runs/sft-format-audit-prod-repair.json \
  --config configs/sft_format_gates.yaml \
  --profile prod_repair_messages \
  --output runs/sft-format-gate-prod-repair.json

.venv-lab/bin/python scripts/check_sft_format_gates.py \
  --audit-report runs/sft-format-audit-prod-repair-strong.json \
  --config configs/sft_format_gates.yaml \
  --profile prod_repair_strong_messages \
  --output runs/sft-format-gate-prod-repair-strong.json

.venv-lab/bin/python scripts/preflight_sft_resume.py \
  --output runs/sft-resume-preflight-strong.json

.venv-train/bin/python scripts/validate_sft_jsonl.py \
  --train-jsonl data/train/coding_sft_prod.jsonl \
  --eval-jsonl data/eval/coding_sft_prod_eval.jsonl \
  --output runs/sft-data-validation-bigcode-prod-44k.json

Each LoRA checkpoint needs a behavior gate before release:

CUDA_VISIBLE_DEVICES=0 \
.venv-train/bin/python scripts/eval_lora_behavior.py \
  --base-model deepreinforce-ai/Ornith-1.0-35B \
  --adapter artifacts/train/ornith-35b-coding-lora-prod-20k/checkpoint-8000 \
  --suite configs/model_behavior_suite.yaml \
  --output runs/eval-lora-behavior-checkpoint-8000-fixed.json

Current stopped-run evidence:

runs/ornith-35b-coding-lora-prod-20k.stopped.json
artifacts/train/ornith-35b-coding-lora-prod-20k/checkpoint-8000
runs/sft-format-audit-prod.json
runs/sft-data-validation-prod-repair.json
runs/sft-format-audit-prod-repair.json
runs/sft-format-gate-strict-repair.json
runs/sft-format-gate-prod-repair.json
runs/sft-code-only-extract.json
runs/sft-data-validation-prod-repair-strong.json
runs/sft-format-audit-prod-repair-strong.json
runs/sft-format-gate-code-only-extract.json
runs/sft-format-gate-prod-repair-strong.json
runs/sft-resume-preflight-strong.json
runs/eval-lora-behavior-checkpoint-8000-fixed.json
runs/eval-lora-behavior-checkpoint-8000-fixed.gate.log

Publication Checklist

Before publishing a quant or serving profile:

Confirm no multi-GPU serving assumptions slipped in.
Record exact artifact path, quantization command, file size, and loaded VRAM.
Run OpenAI contract checks and model behavior suite.
Run throughput at c1/c4/c8/c16 on the intended server shape.
For quants, record BF16 KLD/top-1 agreement.
For MTP, record acceptance by position and matched AR throughput.
Update hf_upload/README.md plus the relevant hf_upload/benchmarks/*.md.
Upload changed docs and artifacts to the HF repo, then read back the remote files with huggingface_hub before calling the publication done.