Instructions to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1",
	filename="ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Use Docker

docker model run hf.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

LM Studio
Jan

vLLM

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Ollama
How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Ollama:
```
ollama run hf.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
```

Unsloth Studio

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 to start chatting

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Docker Model Runner:
```
docker model run hf.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M
```

Lemonade

How to use LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1:Q4_K_M

Run and chat with the model

lemonade run user.Ornith-1.0-35B-GGUF-llamacpp-tp1-Q4_K_M

List all available models

lemonade list

LordNeel commited on 22 days ago

Commit

c920ea4

verified ·

1 Parent(s): 532b5f8

Add SWE-bench quant and reasoning metrics

Browse files

Files changed (11) hide show

MODEL_RELEASE_REPORT.md +450 -0
README.md +407 -240
benchmarks/raw/swebench/ornith-q6-think-verified100-summary.json +67 -0
benchmarks/raw/swebench/ornith_iq4_xs_mtp_graft_verified_mini.json +170 -0
benchmarks/raw/swebench/ornith_iq4_xs_verified_mini.json +167 -0
benchmarks/raw/swebench/ornith_q3_k_m_verified_mini.json +169 -0
benchmarks/raw/swebench/ornith_q4_k_m_verified_mini.json +170 -0
benchmarks/raw/swebench/ornith_q5_k_m_verified_mini.json +171 -0
benchmarks/raw/swebench/ornith_q6_k_verified_mini.json +170 -0
benchmarks/raw/swebench/ornith_q8_0_verified_mini.json +169 -0
benchmarks/swebench-agent-evals.md +75 -0

MODEL_RELEASE_REPORT.md ADDED Viewed

	@@ -0,0 +1,450 @@

+# Ornith-1.0-35B GGUF Release Report
+Date: 2026-06-28
+Hardware for reported GPU numbers: one NVIDIA RTX PRO 6000 Blackwell Max-Q
+96GB, `tp=1`, llama.cpp CUDA server. The release policy is single-GPU serving
+only: one model copy per GPU, no tensor-parallel serving profile.
+## Executive Summary
+This release turns Ornith-1.0-35B into a practical llama.cpp GGUF stack with
+validated serving profiles, locally produced smaller quants, quantitative KLD
+quality checks, and a working native MTP path.
+Recommended use:
+| Use case | Artifact | Why |
+|---|---|---|
+| Default serving speed | `ornith-1.0-35b-Q4_K_M.gguf` | Fastest validated c16 profile, 655.6 tok/s, 21.31 GiB loaded VRAM |
+| Lowest memory | `ornith-1.0-35b-Q3_K_M.gguf` | 15.61 GiB on disk, 17.27 GiB loaded VRAM, behavior gate passed |
+| Middle footprint | `ornith-1.0-35b-IQ4_XS.gguf` | 17.64 GiB on disk, lower KLD drift than Q3_K_M |
+| Quality/footprint | `ornith-1.0-35b-Q6_K.gguf` | 0.0165 mean top-64 KLD nats and 32/32 top-1 match |
+| Native low-concurrency MTP | `ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf` | c1 MTP speedup with high acceptance; adaptive policy avoids saturated c16 loss |
+Key result: Q4_K_M is the production default. The integrated IQ4_XS-MTP graft is
+the MTP release candidate for low-concurrency/single-user experiments. Always-on
+MTP is not recommended for saturated c16 server traffic because target-only
+decoding already saturates the GPU more efficiently.
+## Quantization Work
+### What Was Built
+Two main-body GGUF quants were produced locally from the upstream BF16 GGUF:
+| Artifact | Quant | Quantization detail |
+|---|---|---|
+| `ornith-1.0-35b-Q3_K_M.gguf` | Q3_K_M | `llama-quantize ornith-1.0-35b-bf16.gguf ornith-1.0-35b-Q3_K_M.gguf Q3_K_M`; source 66,152.24 MiB / 16.01 BPW, output 15,977.65 MiB / 3.87 BPW |
+| `ornith-1.0-35b-IQ4_XS.gguf` | IQ4_XS | `llama-quantize ornith-1.0-35b-bf16.gguf ornith-1.0-35b-IQ4_XS.gguf IQ4_XS 32`; no importance matrix; source 66,152.24 MiB / 16.01 BPW, output 18,051.46 MiB / 4.37 BPW |
+The existing upstream GGUFs were mirrored and validated:
+| Artifact | Quant | Release status |
+|---|---|---|
+| `ornith-1.0-35b-Q4_K_M.gguf` | Q4_K_M | validated, production default |
+| `ornith-1.0-35b-Q5_K_M.gguf` | Q5_K_M | validated, higher-quality mid option |
+| `ornith-1.0-35b-Q6_K.gguf` | Q6_K | validated, quality/footprint option |
+| `ornith-1.0-35b-Q8_0.gguf` | Q8_0 | validated, closest to BF16 by mean KLD |
+MTP-specific artifacts were also produced or published:
+| Artifact | Type | Purpose |
+|---|---|---|
+| `ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf` | integrated target + MTP block | native `draft-mtp` low-concurrency profile |
+| `ornith-1.0-35b-mtp-bootstrap-layer39-IQ4_XS.gguf` | bootstrap external draft | proved MTP plumbing, not recommended |
+| `ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-Q6_K.gguf` | trained external draft | MTP development and regression testing |
+| `ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-Q5_K_M.gguf` | trained external draft | smaller draft variant |
+| `ornith-1.0-35b-mtp-chaincorr-h2r9fw125-h1r2-h3r6ce375-lr5e7-IQ4_XS.gguf` | trained external draft | smallest trained draft, lower acceptance |
+### Footprint Chart
+Loaded VRAM is from the short-context throughput profile
+`CTX_SIZE=8192 PARALLEL=16`.
+| Quant | GGUF GiB | Loaded VRAM GiB | Disk vs Q4_K_M | VRAM vs Q4_K_M |
+|---|---:|---:|---:|---:|
+| Q3_K_M | 15.61 | 17.27 | -20.8% | -19.0% |
+| IQ4_XS | 17.64 | 19.34 | -10.5% | -9.2% |
+| Q4_K_M | 19.71 | 21.31 | baseline | baseline |
+| Q5_K_M | 23.03 | 24.65 | +16.8% | +15.7% |
+| Q6_K | 26.56 | 28.03 | +34.8% | +31.5% |
+| Q8_0 | 34.37 | 35.72 | +74.4% | +67.6% |
+Footprint bar, loaded VRAM:
+| Quant | Loaded VRAM chart |
+|---|---|
+| Q3_K_M | `################# 17.27 GiB` |
+| IQ4_XS | `################### 19.34 GiB` |
+| Q4_K_M | `##################### 21.31 GiB` |
+| Q5_K_M | `######################### 24.65 GiB` |
+| Q6_K | `############################ 28.03 GiB` |
+| Q8_0 | `#################################### 35.72 GiB` |
+### Throughput Chart
+Aggregate output tokens/second from `scripts/bench_openai.py --stream
+--max-tokens 256`; all rows had 0 failed requests.
+| Quant | c1 tok/s | c4 tok/s | c8 tok/s | c16 tok/s | c16 p95 TTFT ms |
+|---|---:|---:|---:|---:|---:|
+| Q3_K_M | 240.5 | 422.0 | 464.4 | 493.0 | 493.7 |
+| IQ4_XS | 234.1 | 297.7 | 411.5 | 476.0 | 541.5 |
+| Q4_K_M | 243.3 | 458.3 | 615.0 | 655.6 | 650.0 |
+| Q5_K_M | 236.7 | 311.0 | 439.0 | 638.6 | 620.4 |
+| Q6_K | 225.9 | 295.8 | 409.6 | 603.3 | 657.4 |
+| Q8_0 | 208.5 | 281.5 | 405.8 | 601.4 | 725.8 |
+c16 throughput bar:
+| Quant | c16 tok/s chart |
+|---|---|
+| Q4_K_M | `################################################################ 655.6` |
+| Q5_K_M | `############################################################## 638.6` |
+| Q6_K | `########################################################### 603.3` |
+| Q8_0 | `########################################################### 601.4` |
+| Q3_K_M | `################################################ 493.0` |
+| IQ4_XS | `############################################### 476.0` |
+### KLD Quality Chart
+KLD method: next-token top-64 approximate `KL(P_bf16 || P_quant)` over 32
+coding prompts, native llama.cpp `/completion`, `n_predict=1`, `temperature=-1`,
+`n_probs=64`, token-ID matching. Lower is better.
+| Quant | Mean KLD nats | Mean KLD bits | P95 nats | Max nats | Top-1 match |
+|---|---:|---:|---:|---:|---:|
+| Q3_K_M | 0.3620 | 0.5223 | 1.1077 | 1.3730 | 27/32 (84.4%) |
+| IQ4_XS | 0.1426 | 0.2057 | 0.3195 | 0.6586 | 27/32 (84.4%) |
+| Q4_K_M | 0.0864 | 0.1247 | 0.2877 | 0.4503 | 29/32 (90.6%) |
+| Q5_K_M | 0.0354 | 0.0510 | 0.0943 | 0.2497 | 30/32 (93.8%) |
+| Q6_K | 0.0165 | 0.0238 | 0.0513 | 0.0586 | 32/32 (100.0%) |
+| Q8_0 | 0.0108 | 0.0156 | 0.0440 | 0.0590 | 31/32 (96.9%) |
+Mean KLD drift bar, lower is better:
+| Quant | Mean KLD chart |
+|---|---|
+| Q8_0 | `# 0.0108` |
+| Q6_K | `## 0.0165` |
+| Q5_K_M | `#### 0.0354` |
+| Q4_K_M | `######### 0.0864` |
+| IQ4_XS | `############## 0.1426` |
+| Q3_K_M | `#################################### 0.3620` |
+Interpretation:
+- Q4_K_M is the speed release target.
+- Q3_K_M is valid for minimum memory, but has the largest KLD drift.
+- IQ4_XS is useful as a middle-footprint body and as the MTP graft base.
+- Q5_K_M, Q6_K, and Q8_0 improve distribution fidelity but cost more VRAM.
+### SWE-bench Agent Scores
+Full report: `benchmarks/swebench-agent-evals.md`.
+The full 100-task SWE-bench Verified run used Q6_K with reasoning enabled,
+`temperature=1.0`, `top_p=1.0`, and no explicit thinking budget:
+| Profile | Dataset slice | Reasoning | Resolved | Score |
+|---|---|---|---:|---:|
+| Q6_K GGUF | `SWE-bench/SWE-bench_Verified` test `0:100` | on | 69/100 | 69.0% |
+The multi-quant sweep used `MariusHobbhahn/swe-bench-verified-mini`, split
+`test`, 50 tasks:
+| Quant/profile | Resolved | Score | Completed | Empty patch | Errors |
+|---|---:|---:|---:|---:|---:|
+| Q6_K | 33/50 | 66.0% | 48/50 | 2 | 0 |
+| Q3_K_M | 32/50 | 64.0% | 47/50 | 3 | 0 |
+| Q5_K_M | 32/50 | 64.0% | 49/50 | 1 | 0 |
+| Q4_K_M | 31/50 | 62.0% | 48/50 | 2 | 0 |
+| IQ4_XS | 30/50 | 60.0% | 45/50 | 5 | 0 |
+| IQ4_XS-MTP-graft-headQ6 | 30/50 | 60.0% | 48/50 | 2 | 0 |
+| Q8_0 | 29/50 | 58.0% | 47/50 | 3 | 0 |
+## MTP Work
+### Starting Problem
+The upstream Ornith config advertises one MTP/NextN layer, but the released HF
+weights and plain GGUF files do not contain a loadable MTP head. In practice,
+the plain model cannot simply be started with llama.cpp `--spec-type draft-mtp`.
+The work therefore split into two lines:
+1. Make a loadable MTP draft path for experimentation.
+2. Find a release artifact and serving policy that gives real single-user speed.
+### Bootstrap Draft
+The first bootstrap created a minimal external MTP draft by cloning trunk layer
+39 into the expected MTP layout, adding the missing MTP projection/norm tensors,
+converting with `convert_hf_to_gguf.py --mtp`, and quantizing the draft to
+IQ4_XS.
+Result:
+| Profile | Loaded VRAM GiB | c16 tok/s | Acceptance | Verdict |
+|---|---:|---:|---:|---|
+| IQ4_XS target only | 19.34 | 476.0 | n/a | baseline |
+| Bootstrap IQ4_XS MTP draft, `n_max=1` | 21.32 | 162.7 | 1.21% | not recommended |
+This proved the server plumbing and artifact layout but did not provide useful
+acceptance.
+### Trained External Drafts
+The second line trained chain-corrected Qwen3.5 MoE MTP drafts from target
+hidden-state captures. The goal was to train a draft that predicts future tokens
+from the target hidden state and can be exported as a separate MTP GGUF.
+Best external draft family:
+| Draft artifact | Quant | Loaded VRAM GiB | c16/128 tok/s | Acceptance | Acceptance by position |
+|---|---|---:|---:|---:|---|
+| `...h2r9fw125...Q6_K.gguf` | Q6_K | 22.44 | 286.77 | 67.22% | `(0.719, 0.621)` |
+| `...h2r9fw125...Q5_K_M.gguf` | Q5_K_M | 22.16 | 285.32 | 66.61% | `(0.715, 0.613)` |
+| `...h2r9fw125...IQ4_XS.gguf` | IQ4_XS | 21.53 | 265.06 | 59.20% | `(0.640, 0.541)` |
+The Q6_K draft reached the requested development acceptance band, but the
+serving path remained slower than the target-only matched baseline:
+| Profile | c1 tok/s | c4 tok/s | c8 tok/s | c16 tok/s |
+|---|---:|---:|---:|---:|
+| IQ4_XS target only, no prompt cache | 239.91 | 431.95 | 470.81 | 595.89 |
+| IQ4_XS + Q6_K trained MTP draft, `n_max=2`, per-head contexts | 235.99 | 245.28 | 247.82 | 288.67 |
+| IQ4_XS + Q6_K trained MTP draft, sequential verify | 192.48 | 165.63 | 167.47 | 169.24 |
+Conclusion: the external draft quality became good enough for development, but
+the separate-draft runtime path was not a production speedup.
+### Integrated IQ4_XS-MTP Graft
+The release MTP path is the integrated graft:
+`ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf`
+This file was made by starting from the MIT-licensed
+`wang-yang/Ornith-1.0-35B-MTP-GGUF` Q6_K graft and requantizing the body to
+IQ4_XS while preserving the appended `blk.40.*` MTP block at Q6_K/F16/F32:
+```bash
+llama-quantize --allow-requantize \
+  --tensor-type-file runs/tensor-types-mtp-blk40-preserve.txt \
+  Ornith-1.0-35B-Q6_K-MTP.gguf \
+  ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf \
+  IQ4_XS 24
+```
+Structural verification:
+| Key | Value |
+|---|---|
+| `general.architecture` | `qwen35moe` |
+| `qwen35moe.block_count` | `41` |
+| `qwen35moe.nextn_predict_layers` | `1` |
+| MTP tensors | appended `blk.40.*` block with `blk.40.nextn.*` tensors |
+| Body quant | IQ4_XS |
+| MTP block quant | preserved Q6_K/F16/F32 |
+| File size | 19.6 GB decimal, about 0.69 GB over plain IQ4_XS |
+The integrated graft reproduced the high-acceptance behavior reported by the
+wang-yang reference and avoided the heavy external draft context.
+Low-concurrency results:
+| Profile | Shape | tok/s | Speedup | Draft acceptance |
+|---|---|---:|---:|---:|
+| Integrated graft, target-only AR | c1/128 | 221.80 | 1.00x | n/a |
+| Integrated graft, `draft-mtp n_max=2` | c1/128 | 279.36 | 1.26x | high |
+| Integrated graft, adaptive MTP throttle | c1/128 | 319.53 | 1.44x | `(0.953, 0.865)` |
+Adaptive c1 server counters:
+| Counter | Value |
+|---|---:|
+| Generated speculative tokens | 339 |
+| Accepted speculative tokens | 309 |
+| Mean accepted length | 2.82 |
+| Acceptance by position | `(0.953, 0.865)` |
+Saturated c16 results:
+| Profile | Shape | tok/s | Ratio vs AR | Notes |
+|---|---|---:|---:|---|
+| Target-only AR | c16/64 | 568.57 | 1.00x | matched control |
+| Always-on active MTP, `n_max=2` | c16/64 | 323.12 | 0.57x | high acceptance but slower |
+| Adaptive MTP throttle, `LLAMA_SPEC_MAX_DRAFTING_SLOTS=1` | c16/64 | 564.48 | 0.99x | MTP disabled while saturated |
+The final serving policy is therefore adaptive MTP: enable MTP for a single
+active request and disable drafting when the server is saturated.
+Threshold sweep:
+| `LLAMA_SPEC_MAX_DRAFTING_SLOTS` | c1 tok/s | c2 tok/s | c4 tok/s | c8 tok/s | c16 tok/s |
+|---:|---:|---:|---:|---:|---:|
+| 1 | 271.12 | 316.46 | 424.14 | 564.24 | 592.45 |
+| 2 | 267.04 | 313.25 | 422.88 | 558.17 | 592.39 |
+| 4 | 269.23 | 313.06 | 328.45 | 547.56 | 591.50 |
+Recommended default: `LLAMA_SPEC_MAX_DRAFTING_SLOTS=1`.
+### Dedicated MTP KLD and Sequence Evaluation
+See [benchmarks/mtp-dedicated-kld-evaluation.md](benchmarks/mtp-dedicated-kld-evaluation.md)
+for raw paths and full details.
+Next-token top-64 KLD over 32 coding prompts:
+| Candidate | Mean KLD nats | Mean KLD bits | P95 nats | Max nats | Top-1 match |
+|---|---:|---:|---:|---:|---:|
+| Integrated IQ4_XS-MTP graft, target-only | 0.0731382442 | 0.1055161823 | 0.1593743642 | 0.3348258511 | 29/32 |
+| Integrated IQ4_XS-MTP graft, active `draft-mtp` | 0.0731382442 | 0.1055161823 | 0.1593743642 | 0.3348258511 | 29/32 |
+| Active `draft-mtp` vs target-only graft | 0.0 | 0.0 | 0.0 | 0.0 | 32/32 |
+The one-token API-visible distribution is clean: active `draft-mtp` returns the
+same next-token distribution as the graft running target-only. The integrated
+graft also measured lower drift than the prior plain IQ4_XS capture
+(0.0731382442 vs 0.1425748206 mean KLD nats).
+The longer 8 prompt x 64 token single-user sequence probe exercised real native
+MTP drafting:
+| Runtime | Client aggregate tok/s | Draft acceptance | Exact 64-token sequences | Token-position match |
+|---|---:|---:|---:|---:|
+| Target-only graft | 172.57 | n/a | 8/8 baseline | n/a |
+| Fast active `draft-mtp`, `n_max=2` | 233.81 | 310/378 = 82.01%; `(0.884, 0.747)` by position | 6/8 | 478/512 = 93.36% |
+| `LLAMA_SPEC_VERIFY_SEQUENTIAL=1` | 164.34 | 313/375 = 83.47%; `(0.884, 0.772)` by position | 5/8 | 455/512 = 88.87% |
+This means the native graft remains an experimental low-concurrency speed
+profile, not a strict target-equivalent deterministic serving mode. Also,
+llama.cpp omitted candidate top-logprobs for most accepted speculative tokens
+in sequence responses: 500/512 positions were missing candidate top-logprobs in
+the fast active run. Comparable same-emitted-token positions had near-zero KLD
+(0.0011215739 mean nats, 0.0040239103 max), but full sequence KLD is not valid
+without runtime support for returning verifier logprobs on accepted draft tokens.
+### Runtime Engineering Done
+The MTP work included artifact surgery and runtime work:
+| Area | Work completed | Impact |
+|---|---|---|
+| Missing MTP tensors | Built bootstrap MTP draft from trunk layer 39 and added required projection/norm tensors | Made llama.cpp `draft-mtp` loadable for local experiments |
+| Hidden-state capture | Added/used capture paths for target hidden rows and token positions | Enabled trained external MTP draft experiments |
+| Chain-corrected training | Trained h2/h1/h3 chain-corrected drafts and exported Q6_K/Q5_K_M/IQ4_XS GGUFs | Raised external-draft acceptance from about 1% to about 67% |
+| Integrated graft requantization | Requantized a Q6_K integrated MTP graft to IQ4_XS while preserving `blk.40.*` MTP tensors | Produced a 19.6 GB native MTP artifact |
+| Fast backend sampling | Added a fast path for unambiguous backend top-k=1 draft sampling | Reduced sampling overhead in the native graft path |
+| Qwen35MoE recurrent row index | Patched recurrent-state writes and rollback snapshots to use runtime row-index tensors rather than graph-baked head offsets | Fixed graph/recurrent mismatch across MTP head changes |
+| Per-head MTP contexts | Added optional `LLAMA_MTP_PER_HEAD_CONTEXT=1` path for trained external drafts | Improved graph reuse but did not make external drafts faster than target-only |
+| Sequential verification | Added `LLAMA_SPEC_VERIFY_SEQUENTIAL=1` correctness guard | Made temperature-0 MTP output match target-only in deterministic probes |
+| Adaptive throttling | Added/validated `LLAMA_SPEC_MAX_DRAFTING_SLOTS=1` policy | Keeps c1 MTP speedup while avoiding saturated c16 regression |
+Important correctness note: the external trained-draft path can diverge from
+target-only output without sequential verification. The sequential verifier
+matches target-only output but is slower, so it is a debug/correctness guard, not
+the production profile. The integrated graft's release value is low-concurrency
+speed with adaptive throttling.
+### MTP Recommendation
+Use this single-GPU command for the integrated MTP profile:
+```bash
+export CUDA_VISIBLE_DEVICES=0
+export LLAMA_SPEC_MAX_DRAFTING_SLOTS=1
+export LLAMA_MTP_FAST_BACKEND_SAMPLE=1
+export LLAMA_MTP_DRAFT_TOP_K=1
+export LLAMA_MTP_DRAFT_TOP_P=1
+export LLAMA_MTP_DRAFT_TEMP=1
+llama-server \
+  -m ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf \
+  --alias Ornith-1.0-35B-GGUF-IQ4_XS-MTP \
+  --host 127.0.0.1 --port 8002 \
+  -c 131072 -np 1 -b 4096 -ub 512 -ngl 99 \
+  --flash-attn on \
+  --reasoning off --reasoning-format deepseek \
+  --spec-type draft-mtp \
+  --spec-draft-ngl 99 \
+  --spec-draft-n-max 2 \
+  --spec-draft-n-min 0 \
+  --spec-draft-backend-sampling \
+  --cont-batching \
+  --metrics
+```
+For multi-slot server traffic, keep `LLAMA_SPEC_MAX_DRAFTING_SLOTS=1` so MTP is
+used only when the server has a small active batch. For a single-user agentic
+workload, this is the preferred MTP profile.
+## Pi-Agent Coding Smoke
+A Pi-agent smoke was run against the single-user integrated MTP profile with
+`CTX_SIZE=131072`, `PARALLEL=1`, and model alias `ornith-pi-agent-mtp`.
+| Test | Result |
+|---|---|
+| Calculator tool smoke | pass |
+| Ledger parser/summary repo-edit task | pass, 3/3 tests |
+| TTL cache repo-edit task | pass, 3/3 tests |
+| Router dispatch repo-edit task | pass, 4/4 tests |
+Serving counters from that Pi-agent run:
+| Metric | Value |
+|---|---:|
+| Prompt throughput | 3460.27 tok/s |
+| Decode throughput | 339.23 tok/s |
+| MTP generated draft tokens | 1234 |
+| MTP accepted draft tokens | 1114 |
+| Aggregate draft acceptance | 90.3% |
+| Acceptance by position | `(0.929, 0.877)` |
+| Mean accepted length | 2.81 |
+| Peak observed loaded VRAM | about 22.65 GiB |
+## Pi-Agent Quant Flappy Bird Test
+Each published serving profile was loaded as a single-user llama.cpp server and
+attached to Pi-agent with the same vanilla HTML/CSS/JS Flappy Bird prompt. The
+generated games are not hand-edited; Q3_K_M required three Pi-agent repair
+passes, and the other profiles passed on the first agent run.
+Full report: `benchmarks/pi-agent-flappy-quant-test.md`.
+Raw evidence: `probes/pi-agent-flappy/`.
+| Quant/Profile | Result | Decode TPS on Pi-agent task |
+|---|---:|---:|
+| Q3_K_M | pass after 3 Pi-agent repair passes | 250.88 |
+| IQ4_XS | pass first run | 242.44 |
+| Q4_K_M | pass first run | 249.59 |
+| Q5_K_M | pass first run | 239.67 |
+| Q6_K | pass first run | 226.51 |
+| Q8_0 | pass first run | 212.98 |
+| IQ4_XS-MTP-graft-headQ6 | pass first run | 320.91 |
+Interaction checks loaded every generated page, verified one canvas, pressed
+Space, captured an after-input screenshot, and observed zero page errors for all
+7 profiles.
+The MTP profile's cumulative server log for this agent workload reports 3,452
+accepted draft tokens out of 3,882 generated draft tokens, 88.92% draft-token
+acceptance, mean acceptance length 2.78, and per-position acceptance
+`(0.936, 0.843)`.
+## Evidence Index
+- Quant benchmarks: `benchmarks/llamacpp-quant-benchmarks.md`
+- Quant KLD probe: `benchmarks/kld-quant-vs-bf16-top64.md`
+- Integrated MTP profile: `benchmarks/llamacpp-iq4-xs-mtp-graft-adaptive-profile.md`
+- SWE-bench agent evaluations: `benchmarks/swebench-agent-evals.md`
+- Pi-agent quant Flappy Bird test: `benchmarks/pi-agent-flappy-quant-test.md`
+  with raw logs and metrics in `probes/pi-agent-flappy/`
+- Trained MTP profile: `benchmarks/llamacpp-iq4-xs-mtp-chaincorr-profile.md`
+- MTP profile catalog: `configs/mtp_profiles.yaml`
+- Quant artifact catalog: `configs/quant_artifacts.yaml`
+- Serving profile catalog: `configs/serving_profiles.yaml`
+- Runtime patches: `patches/llamacpp-qwen35moe-mtp-recurrent-row-index.patch`,
+  `patches/llamacpp-mtp-adaptive-serving-and-recurrent-rollback.patch`, and
+  `patches/llamacpp-server-mtp-seqverify-and-perhead.patch`

README.md CHANGED Viewed

@@ -16,270 +16,437 @@ tags:
   - tp1
 ---
-# Ornith-1.0-35B — GGUF (llama.cpp, single-GPU `tp=1`)
-Single-GPU llama.cpp GGUF package for [`deepreinforce-ai/Ornith-1.0-35B`](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B).
-The supported serving policy is `tp=1` — one model copy per GPU. Multi-GPU
-tensor-parallel serving is intentionally out of scope for this version.
-This release ships six body quants (Q3_K_M → Q8_0) plus an **integrated
-IQ4_XS-MTP graft** that adds a native multi-token-prediction (MTP) draft head for
-low-concurrency speculative decode. Quant quality is measured against the upstream
-BF16 GGUF with a native llama.cpp next-token top-64 KL-divergence probe over 32
-coding prompts.
-## TL;DR — pick an artifact
 | Use case | Recommended artifact | Key numbers |
 |---|---|---|
 | Default serving speed | `ornith-1.0-35b-Q4_K_M.gguf` | 19.71 GiB on disk, 21.31 GiB loaded VRAM, 243.3 tok/s c1, 655.6 tok/s c16 |
 | Lowest memory | `ornith-1.0-35b-Q3_K_M.gguf` | 15.61 GiB on disk, 17.27 GiB loaded VRAM, 240.5 tok/s c1, 493.0 tok/s c16 |
 | Middle footprint | `ornith-1.0-35b-IQ4_XS.gguf` | 17.64 GiB on disk, 19.34 GiB loaded VRAM, 0.1426 mean top-64 KLD nats |
-| Highest fidelity / footprint | `ornith-1.0-35b-Q6_K.gguf` | 26.56 GiB on disk, 28.03 GiB loaded VRAM, 0.0165 mean top-64 KLD nats, 32/32 top-1 |
-| Native low-concurrency MTP | `ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf` | 19.6 GB decimal, **0.0731 mean top-64 KLD (29/32)**, c1/128 adaptive MTP 319.53 tok/s, acceptance `(0.953, 0.865)` |
-> **MTP graft note:** active MTP is byte-for-byte identical to target-only at the
-> next-token (API-visible) level, and gives ~1.3× single-stream decode. It is
-> **not** bit-exact to target-only over long deterministic sequences — see
-> [MTP](#multi-token-prediction-mtp). Use target-only for strict reproduction,
-> MTP for throughput on open-ended work.
-## Quantization & fidelity
-Fidelity is `KL(P_bf16 || P_candidate)` over next-token top-64 distributions,
-32 coding prompts, `n_predict=1, temperature=-1, n_probs=64`, keyed by token ID.
-Lower KLD and higher greedy top-1 agreement are better.
-### Primary fidelity table
-| Artifact | Mean KLD (nats) | Greedy top-1 † |
-|---|---:|---:|
-| `Q8_0` | 0.0108 | 31/32 (96.9%) |
-| `Q6_K` | 0.0165 | 32/32 (100.0%) |
-| `Q5_K_M` | 0.0354 | 30/32 (93.8%) |
-| **`IQ4_XS-MTP` (graft)** | **0.0731382442** | **29/32 (90.6%)** |
-| `Q4_K_M` | 0.0864 | 29/32 (90.6%) |
-| `IQ4_XS` | 0.1426 | 27/32 (84.4%) |
-| `Q3_K_M` | 0.3620 | 27/32 (84.4%) |
-![Next-token fidelity ladder: mean top-64 KLD vs BF16 for the six body quants plus the IQ4_XS-MTP graft, lower is better.](assets/02_fidelity_ladder.png)
-*Fidelity ladder — the integrated IQ4_XS-MTP graft (purple) lands between Q5_K_M
-and Q4_K_M, i.e. better next-token fidelity than its Q4_K_M neighbor.*
-> **† Top-1 vs mean KLD are different axes.** *Top-1* counts how often the single
-> most-likely (argmax) token matches BF16's, over the 32 prompts; *mean KLD*
-> measures divergence across the full top-64 distribution. Two quants can share a
-> top-1 count yet differ a lot in KLD — e.g. `Q3_K_M` and `IQ4_XS` are both
-> 27/32 on top-1, but `Q3_K_M`'s distribution diverges ~2.5× more (0.3620 vs
-> 0.1426 nats). Neither value is a typo.
-- The **integrated IQ4_XS-MTP graft is 48.70% lower mean KLD than the plain
-  IQ4_XS body** (0.0731382442 vs 0.1425748206), and improves greedy top-1 from
-  27/32 to 29/32.
-- **Active `draft-mtp` ≡ target-only at the next-token level:** their top-64
-  next-token distributions are identical (KLD 0.0, 32/32) for the one-token
-  API-visible probe.
-### Body-quant distribution detail (secondary)
-Full per-quant distribution from the corrected top-64 probe (same 32 prompts).
-`Mean / P50 / P95 / Max` are explicit nats statistics of the per-prompt top-64
-KLD; in this run the BF16 top-64 distribution captured mean probability mass
-0.999965, so the top-64 approximation captured essentially all next-token mass.
-| Quant | GGUF GiB | KLD-probe VRAM GiB | Mean nats | P50 nats | P95 nats | Max nats | Top-1 † |
-|---|---:|---:|---:|---:|---:|---:|---:|
-| Q3_K_M | 15.61 | 16.32 | 0.3620 | 0.2548 | 1.1077 | 1.3730 | 27/32 (84.4%) |
-| IQ4_XS | 17.64 | 19.34 | 0.1426 | 0.0868 | 0.3195 | 0.6586 | 27/32 (84.4%) |
-| Q4_K_M | 19.71 | 20.35 | 0.0864 | 0.0379 | 0.2877 | 0.4503 | 29/32 (90.6%) |
-| Q5_K_M | 23.03 | 23.61 | 0.0354 | 0.0235 | 0.0943 | 0.2497 | 30/32 (93.8%) |
-| Q6_K | 26.56 | 27.07 | 0.0165 | 0.0092 | 0.0513 | 0.0586 | 32/32 (100.0%) |
-| Q8_0 | 34.37 | 34.77 | 0.0108 | 0.0052 | 0.0440 | 0.0590 | 31/32 (96.9%) |
-The IQ4_XS-MTP graft uses the same probe on a different breakpoint context; its
-distribution detail (`P50 0.0492811974 / P95 0.1593743642 / Max 0.3348258511`) is
-in the [MTP](#multi-token-prediction-mtp) section. Source:
-[benchmarks/kld-quant-vs-bf16-top64.md](benchmarks/kld-quant-vs-bf16-top64.md).
-## Serving performance
-Aggregate decode throughput and p95 time-to-first-token across concurrency,
-llama.cpp `tp=1`, from `scripts/bench_openai.py --stream --max-tokens 256` (0
-failed requests per row). Q4_K_M is the speed pick; Q5_K_M nearly ties it at c16.
-| Quant | c1 tok/s | c1 p95 TTFT ms | c4 tok/s | c4 p95 TTFT ms | c8 tok/s | c8 p95 TTFT ms | c16 tok/s | c16 p95 TTFT ms |
-|---|---:|---:|---:|---:|---:|---:|---:|---:|
-| Q3_K_M | 240.5 | 77.9 | 422.0 | 170.7 | 464.4 | 344.9 | 493.0 | 493.7 |
-| IQ4_XS | 234.1 | 75.1 | 297.7 | 159.7 | 411.5 | 330.2 | 476.0 | 541.5 |
-| Q4_K_M | 243.3 | 76.3 | 458.3 | 192.3 | 615.0 | 361.8 | 655.6 | 650.0 |
-| Q5_K_M | 236.7 | 75.1 | 311.0 | 198.8 | 439.0 | 383.6 | 638.6 | 620.4 |
-| Q6_K | 225.9 | 76.8 | 295.8 | 194.2 | 409.6 | 394.8 | 603.3 | 657.4 |
-| Q8_0 | 208.5 | 76.9 | 281.5 | 190.8 | 405.8 | 389.1 | 601.4 | 725.8 |
-![Aggregate decode throughput vs concurrency (c1–c16) for the six quants; Q4_K_M leads at c16.](assets/03_throughput_tps.png)
-![p95 time-to-first-token vs concurrency (c1–c16) for the six quants; lower is better.](assets/04_ttft_p95.png)
-> **Profile note:** the table above is a **short-context** profile — `CTX_SIZE=8192
-> PARALLEL=16` exposes `n_ctx = 512`/slot, with ~23-token prompts and 256-token
-> generations. Source: [benchmarks/llamacpp-quant-benchmarks.md](benchmarks/llamacpp-quant-benchmarks.md).
-> For how prefill latency scales with prompt length, see long-context TTFT below.
-### Long-context TTFT (single stream)
-p95 time-to-first-token vs prompt/context length, single GPU `tp=1`, single stream
-(`CTX_SIZE=131072 PARALLEL=1`, `n_ctx/slot=131072`, exact prompt token counts, 140
-rows/quant, 0 failures).
-| Context tokens | Q4_K_M p50/p95 ms | Q4_K_M tok/s | IQ4_XS p50/p95 ms | IQ4_XS tok/s | MTP-graft p50/p95 ms | MTP-graft tok/s |
-|---:|---:|---:|---:|---:|---:|---:|
-| 512 | 91.8 / 94.1 | 188.3 | 87.1 / 88.7 | 177.4 | 86.0 / 87.8 | 184.0 |
-| 1024 | 169.6 / 172.2 | 145.8 | 159.9 / 161.6 | 139.9 | 157.6 / 159.0 | 155.9 |
-| 2048 | 341.8 / 346.1 | 97.9 | 316.1 / 318.7 | 112.0 | 307.6 / 308.9 | 113.9 |
-| 4096 | 699.2 / 702.2 | 67.2 | 647.0 / 653.4 | 70.7 | 625.7 / 631.3 | 69.0 |
-| 8192 | 1447.5 / 1458.0 | 33.7 | 1342.9 / 1348.5 | 39.8 | 1297.4 / 1304.0 | 40.9 |
-| 16384 | 3009.7 / 3030.4 | 19.5 | 2793.2 / 2804.0 | 20.9 | 2709.8 / 2737.3 | 21.5 |
-| 32768 | 6302.4 / 6313.0 | 9.7 | 5829.9 / 5853.1 | 10.5 | 5673.5 / 5696.6 | 10.7 |
-![Long-context p95 TTFT vs prompt length (512–32768 tokens) for Q4_K_M, IQ4_XS, and the MTP graft; single stream, tp=1.](assets/07_longctx_ttft.png)
-p95 TTFT rises with prompt length (prefill cost) to ~6.3 s at 32k tokens for
-Q4_K_M; the IQ4_XS body and IQ4_XS-MTP graft prefill slightly faster at every
-length. Decode throughput falls from ~180–190 tok/s at 512 tokens to ~10 tok/s at
-32k as the KV cache grows. Source: [benchmarks/llamacpp-longctx-ttft.md](benchmarks/llamacpp-longctx-ttft.md).
-## Multi-token prediction (MTP)
-The integrated `IQ4_XS-MTP-graft-headQ6` artifact carries a native MTP draft head
-on the IQ4_XS body. The release recommendation is **adaptive**: use MTP for
-low-concurrency / single-user requests and keep `LLAMA_SPEC_MAX_DRAFTING_SLOTS=1`
-so saturated batches fall back to target-only throughput. The MTP catalog
-validates the published MTP profiles.
-### Next-token quality
-Active MTP **does not change** the one-token, API-visible next-token
-distribution. Mean BF16 KLD is **0.0731382442 nats** over the 32-prompt top-64
-probe (P50 0.0492811974 / P95 0.1593743642 / Max 0.3348258511), top-1 29/32. The
-active `draft-mtp` vs target-only graft next-token KLD is **0.0 (32/32 identical)**.
-### Sequence-level behavior
-A deterministic **8 prompt × 64 token** probe (`n_predict=64, temperature=-1,
-n_probs=64`) compares the same GGUF running active native `draft-mtp` against
-target-only.
-| Runtime | Client agg tok/s | Server decode tok/s | Draft acceptance | Exact 64-tok seqs | Token-position match |
-|---|---:|---|---|---:|---|
-| Target-only graft | 172.57 | ~210 (per-request timing) | n/a | 8/8 baseline | n/a |
-| Fast active `draft-mtp`, `n_max=2` | 233.81 | 325.70 (/metrics) | 310/378 = 82.01%; per-position `(0.884, 0.747)` | 6/8 | 478/512 = 93.36% |
-| `LLAMA_SPEC_VERIFY_SEQUENTIAL=1` | 164.34 | 204.88 (/metrics) | 313/375 = 83.47%; per-position `(0.884, 0.772)` | 5/8 | 455/512 = 88.87% |
-![MTP single-stream decode speedup on the deterministic 8x64 probe: target-only vs active MTP graft, client and server decode.](assets/06_mtp_tps.png)
-- Fast active MTP gives **~1.35× client throughput** (233.81 vs 172.57) and
-  server decode 325.70 vs ~210, with **82.01%** draft acceptance.
-- It matches **6/8** sequences exactly and **478/512 = 93.36%** of token
-  positions; the two non-exact sequences first diverge at positions **60** and
-  **34**.
-- The `LLAMA_SPEC_VERIFY_SEQUENTIAL=1` variant is both **slower** (164.34 /
-  204.88) **and** matches less (5/8, 88.87%; divergences at 25, 43, 60) — i.e. the
-  fast verifier is here both faster and more target-matching.
-> **Note:** MTP self-speculation is **not bit-exact to target-only over long
-> deterministic generations**. For workloads that require exact target
-> reproduction, run target-only; for throughput on open-ended work, MTP wins.
-> MTP is a single-user / low-concurrency win — it is not faster on saturated
-> batches, which is why the adaptive throttle falls back to target-only.
-### Sequence logprob comparability (API limitation)
-llama.cpp omits candidate `top_logprobs` for most accepted speculative tokens, so
-a naive "sequence KLD" looks **enormous** — that is an **API/logprob-availability
-artifact, not model quality**. On the comparable subset where the emitted token
-also matches, KLD is **≈ 0.001 nats** (near zero):
-| Runtime | Paired positions | Comparable top-logprobs | Missing | Comparable same-token mean KLD (nats) | Max (nats) |
-|---|---:|---:|---:|---:|---:|
-| Fast active `draft-mtp` vs target-only | 512 | 12 | 500 | 0.0011215739 | 0.0040239103 |
-| Sequential verifier vs target-only | 512 | 10 | 502 | 0.0012293172 | 0.0040239103 |
-Full sequence KLD cannot be computed without changing the runtime to return
-target verifier logprobs for accepted speculative tokens.
-### Draft-head distillation provenance
-The meaningful KL-style metric for a standalone MTP draft head is **teacher KL
-against cached target hidden states**, not standard base-model next-token KLD.
-The draft head was trained and **measurably improved** across checkpoints —
-teacher KL fell 1.92 → 1.50 → 0.76 and teacher argmax top-1 rose 0.54 → 0.69 →
-0.89.
-| Draft checkpoint | Tokens | Windows | Teacher KL | Teacher argmax top-1 | Teacher argmax top-5 | Gold top-1 |
-|---|---:|---:|---:|---:|---:|---:|
-| `mtp-distill-kl-step500` | 22,637 | 128 | 1.9222025748 | 0.5360250917 | 0.7881786456 | 0.4425939833 |
-| `snapshots-iq4-live-accepted-a/step-1000` | 58,754 | 413 | 1.5038724942 | 0.6876808388 | 0.8368451510 | 0.6764816013 |
-| `snapshots-iq4-live-allrows-b/step-1000` | 32,264 | 2,048 | 0.7591610373 | 0.8908070915 | 0.9389722291 | 0.8076183982 |
-![MTP draft-head distillation progress: teacher KL falls (left) while top-1 agreement rises (right) across three checkpoints.](assets/05_mtp_draft_head_distill.png)
-*Teacher KL is measured against cached target hidden states (not base next-token
-KLD). These training artifacts live under `artifacts/mtp/` locally and are
-referenced by name as provenance — they are not committed in this HF repo.*
-## Run with llama.cpp
-Serve a body quant on GPU0 (`tp=1`):
 ```bash
-QUANT=Q4_K_M PORT=8000 CTX_SIZE=8192 PARALLEL=16 REASONING=off \
-  scripts/serve_llamacpp_gpu0.sh
 ```
-Serve the integrated MTP graft for low-concurrency speculative decode (keep
-adaptive throttling so saturated batches fall back to target-only):
 ```bash
-QUANT=IQ4_XS-MTP-graft-headQ6 PORT=8000 CTX_SIZE=8192 PARALLEL=1 \
-  CACHE_RAM=0 REASONING=off LLAMA_SPEC_MAX_DRAFTING_SLOTS=1 \
   scripts/serve_llamacpp_gpu0.sh
 ```
-`REASONING=off` is the default because the model otherwise spends simple coding
-prompts in `reasoning_content` before producing final `content`. See
-[benchmarks/llamacpp-q4-reasoning-off-fix.md](benchmarks/llamacpp-q4-reasoning-off-fix.md).
-Serving / quant / MTP profile catalogs:
-[configs/serving_profiles.yaml](configs/serving_profiles.yaml),
-[configs/quant_artifacts.yaml](configs/quant_artifacts.yaml),
-[configs/mtp_profiles.yaml](configs/mtp_profiles.yaml).
-## Benchmark environment
-**Hardware:** NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 97,887 MiB
-VRAM, single GPU (`tp=1`). Driver 580.159.03, CUDA runtime reported by driver 13.0.
-**Dedicated MTP KLD eval config (2026-06-28):** local llama.cpp CUDA server,
-single **RTX PRO 6000 Blackwell 96GB**, `tp=1`, `CTX_SIZE=8192`, `PARALLEL=1`,
-`CACHE_RAM=0`, `REASONING=off`.
-- **llama.cpp build/commit:** `050ee92d04c2e1f639025786dea701c70e7d4204` (pinned for
-  the long-context TTFT run; earlier KLD/throughput sweeps were not separately pinned).
-- **Reasoning mode:** all benchmarks in this card ran with `REASONING=off`.
-## Provenance & reproducibility
-- **Figures:** regenerate with [`scripts/make_charts.py`](scripts/make_charts.py)
-  → `assets/02_fidelity_ladder.png`, `assets/03_throughput_tps.png`,
-  `assets/04_ttft_p95.png`, `assets/05_mtp_draft_head_distill.png`,
-  `assets/06_mtp_tps.png`.
-- **Serving throughput / TTFT:** [benchmarks/llamacpp-quant-benchmarks.md](benchmarks/llamacpp-quant-benchmarks.md).
-- **Body-quant fidelity:** [benchmarks/kld-quant-vs-bf16-top64.md](benchmarks/kld-quant-vs-bf16-top64.md).
-- **Dedicated MTP KLD / sequence eval (2026-06-28):**
-  [benchmarks/mtp-kld-eval-2026-06-28.md](benchmarks/mtp-kld-eval-2026-06-28.md)
-  and [benchmarks/mtp-dedicated-kld-evaluation.md](benchmarks/mtp-dedicated-kld-evaluation.md),
-  with raw evidence under [benchmarks/raw/](benchmarks/raw/). Includes the
-  sequence-level probe and its logprob-availability limitation (full sequence KLD
-  is not computable from the current llama.cpp API response).
-- **Adaptive MTP serving profile:** [benchmarks/llamacpp-iq4-xs-mtp-graft-adaptive-profile.md](benchmarks/llamacpp-iq4-xs-mtp-graft-adaptive-profile.md).
-- **Long-context TTFT:** [benchmarks/llamacpp-longctx-ttft.md](benchmarks/llamacpp-longctx-ttft.md).

   - tp1
 ---
+# Ornith 35B Lab
+This workspace tracks optimization work for `deepreinforce-ai/Ornith-1.0-35B`.
+## Release Summary
+Full release writeup: [MODEL_RELEASE_REPORT.md](MODEL_RELEASE_REPORT.md).
+This release is a single-GPU llama.cpp GGUF package for Ornith-1.0-35B. The
+supported serving policy is `tp=1`, one model copy per GPU. Multi-GPU tensor
+parallel serving is intentionally out of scope for this version.
 | Use case | Recommended artifact | Key numbers |
 |---|---|---|
 | Default serving speed | `ornith-1.0-35b-Q4_K_M.gguf` | 19.71 GiB on disk, 21.31 GiB loaded VRAM, 243.3 tok/s c1, 655.6 tok/s c16 |
 | Lowest memory | `ornith-1.0-35b-Q3_K_M.gguf` | 15.61 GiB on disk, 17.27 GiB loaded VRAM, 240.5 tok/s c1, 493.0 tok/s c16 |
 | Middle footprint | `ornith-1.0-35b-IQ4_XS.gguf` | 17.64 GiB on disk, 19.34 GiB loaded VRAM, 0.1426 mean top-64 KLD nats |
+| Quality/footprint | `ornith-1.0-35b-Q6_K.gguf` | 26.56 GiB on disk, 28.03 GiB loaded VRAM, 0.0165 mean top-64 KLD nats, 32/32 top-1 match |
+| Native low-concurrency MTP | `ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf` | 19.6 GB decimal, c1/128 adaptive MTP 319.53 tok/s, acceptance `(0.953, 0.865)` |
+Quant quality was measured against the upstream BF16 GGUF with a native
+llama.cpp next-token top-64 KLD probe over 32 coding prompts. MTP work includes
+a bootstrapped external draft, trained chain-corrected draft artifacts, a
+requantized integrated IQ4_XS-MTP graft, and patched llama.cpp runtime support
+for recurrent row-index handling, fast backend sampling, sequential verification,
+per-head contexts, and adaptive MTP throttling. The release MTP recommendation
+is adaptive: use MTP for low-concurrency/single-user requests and keep
+`LLAMA_SPEC_MAX_DRAFTING_SLOTS=1` so saturated batches fall back to target-only
+throughput.
+Dedicated MTP KLD/sequence evaluation is available at
+[benchmarks/mtp-dedicated-kld-evaluation.md](benchmarks/mtp-dedicated-kld-evaluation.md).
+The integrated IQ4_XS-MTP graft measured 0.073138 mean top-64
+`KL(P_bf16 || P_candidate)` nats over 32 prompts. In an 8 prompt x 64 token
+single-user sequence probe, fast native `draft-mtp` reached 82.01% draft-token
+acceptance and 233.81 client tok/s vs 172.57 target-only, but matched only
+478/512 generated token positions, so the active MTP path remains an
+experimental low-concurrency speed profile rather than a strict target-equivalent
+serving mode.
+## SWE-bench Agent Scores
+Full report and raw score JSON:
+[benchmarks/swebench-agent-evals.md](benchmarks/swebench-agent-evals.md).
+Full SWE-bench Verified 100-task slice, reasoning mode enabled:
+| Profile | Dataset slice | Reasoning | Resolved | Score |
+|---|---|---|---:|---:|
+| Q6_K GGUF | `SWE-bench/SWE-bench_Verified` test `0:100` | on | 69/100 | 69.0% |
+Verified-mini quant sweep, 50 tasks:
+| Quant/profile | Resolved chart | Score |
+|---|---|---:|
+| Q6_K | `##################################################################` | 66.0% |
+| Q3_K_M | `################################################################` | 64.0% |
+| Q5_K_M | `################################################################` | 64.0% |
+| Q4_K_M | `##############################################################` | 62.0% |
+| IQ4_XS | `############################################################` | 60.0% |
+| IQ4_XS-MTP-graft-headQ6 | `############################################################` | 60.0% |
+| Q8_0 | `##########################################################` | 58.0% |
+## Scope
+- Serving is single GPU only: `tp=1`, one model copy per GPU.
+- GPU0 is the serving speed lane.
+- GPU1 is an independent experiment lane for quantization, fine-tuning, or audit jobs.
+- Multi-GPU serving profiles are intentionally out of scope for this model version.
+## Initial Tracks
+1. Baseline and tune single-GPU serving with SGLang, vLLM, and llama.cpp GGUF.
+2. Build a reusable model audit harness for tokenizer, chat template, special-token, reasoning, and tool-call issues.
+3. Evaluate existing GGUF quantizations, then test smaller or higher-throughput formats.
+4. Prepare a coding-data fine-tuning and post-training pipeline.
+Current reusable workflow: [MODEL_INVESTIGATION_HARNESS.md](MODEL_INVESTIGATION_HARNESS.md).
+Detailed quant/MTP/model-release report: [MODEL_RELEASE_REPORT.md](MODEL_RELEASE_REPORT.md).
+Pi-agent quant coding task report:
+[benchmarks/pi-agent-flappy-quant-test.md](benchmarks/pi-agent-flappy-quant-test.md)
+with raw evidence under [probes/pi-agent-flappy/](probes/pi-agent-flappy/).
+Dedicated MTP KLD/sequence evaluation:
+[benchmarks/mtp-dedicated-kld-evaluation.md](benchmarks/mtp-dedicated-kld-evaluation.md)
+with raw evidence under [benchmarks/raw/](benchmarks/raw/).
+SWE-bench agent evaluation:
+[benchmarks/swebench-agent-evals.md](benchmarks/swebench-agent-evals.md)
+with raw score reports under [benchmarks/raw/swebench/](benchmarks/raw/swebench/).
+Current requirement audit: [PROJECT_STATUS_AUDIT.md](PROJECT_STATUS_AUDIT.md).
+SFT remediation plan: [SFT_REMEDIATION_PLAN.md](SFT_REMEDIATION_PLAN.md).
+Serving profile catalog: [configs/serving_profiles.yaml](configs/serving_profiles.yaml)
+with validation report [probes/serving-profile-validation.md](probes/serving-profile-validation.md).
+Quant artifact catalog: [configs/quant_artifacts.yaml](configs/quant_artifacts.yaml)
+with validation report [probes/quant-artifact-validation.md](probes/quant-artifact-validation.md).
+MTP profile catalog: [configs/mtp_profiles.yaml](configs/mtp_profiles.yaml)
+with validation report [probes/mtp-profile-validation.md](probes/mtp-profile-validation.md).
+Project requirement catalog: [configs/project_requirements.yaml](configs/project_requirements.yaml)
+with validation report [probes/project-requirement-validation.md](probes/project-requirement-validation.md).
+Idle-state policy: [configs/idle_state_policy.yaml](configs/idle_state_policy.yaml)
+with validation report [probes/idle-state-validation.md](probes/idle-state-validation.md).
+CPU validation suite: [scripts/run_cpu_validation_suite.py](scripts/run_cpu_validation_suite.py)
+with report [probes/cpu-validation-suite.md](probes/cpu-validation-suite.md).
+Goal completion audit: [scripts/build_goal_completion_audit.py](scripts/build_goal_completion_audit.py)
+with report [probes/goal-completion-audit.md](probes/goal-completion-audit.md).
+Release-reference consistency: [scripts/validate_release_references.py](scripts/validate_release_references.py)
+with report [probes/release-reference-validation.md](probes/release-reference-validation.md).
+Behavior regression matrix: [probes/behavior-regression-matrix.md](probes/behavior-regression-matrix.md)
+via [scripts/summarize_behavior_matrix.py](scripts/summarize_behavior_matrix.py).
+Release readiness report: [probes/release-readiness-report.md](probes/release-readiness-report.md)
+via [scripts/build_release_readiness_report.py](scripts/build_release_readiness_report.py).
+HF release verification: [probes/hf-release-verification.json](probes/hf-release-verification.json)
+via [scripts/verify_hf_release.py](scripts/verify_hf_release.py). Latest local
+run passed 48 checks with 0 failures and 3 expected incomplete-project warnings;
+the exact verified commit is recorded in `remote.sha` in the JSON report.
+Publishing a verifier report can leave the staged published report one metadata
+commit behind the newest remote head; `scripts/validate_release_references.py`
+and `scripts/build_goal_completion_audit.py` record both published and latest
+local verifier evidence to prevent stale completion claims.
+When the same script runs from a fresh HF repo clone, `hf_upload/...` defaults
+fall back to repo-root paths; a missing local latest-verifier run is a warning,
+not a failure.
+## Current Serving Baseline
+- Best current GGUF serving profile: llama.cpp `Q4_K_M` `tp=1` with `REASONING=off`.
+- Integrated MTP serving profile: `IQ4_XS-MTP-graft-headQ6` with adaptive throttling, `LLAMA_SPEC_MAX_DRAFTING_SLOTS=1`; this keeps low-concurrency MTP speedups and avoids the saturated c16 loss. The MTP catalog validates all 8 published MTP GGUF artifacts.
+- Pi-agent Flappy Bird quant test: all 7 tested profiles generated a runnable vanilla HTML/CSS/JS canvas game. `IQ4_XS-MTP-graft-headQ6` was fastest on this agent workload at 320.91 decode tok/s, with 88.92% MTP draft-token acceptance (`3452/3882`) and per-position acceptance `(0.936, 0.843)`. Q3_K_M passed only after 3 Pi-agent repair passes; all other profiles passed on the first run.
+- Earlier vLLM BF16 `tp=1` and SGLang BF16 `tp=1` profiles are preserved as baseline notes, but the publishable model package is the llama.cpp GGUF stack.
+- The default llama.cpp script uses `REASONING=off` because the model otherwise spends simple coding prompts in `reasoning_content` before producing final `content`.
+- Custom Q3_K_M `tp=1` is now validated as the lowest-memory GGUF option: 15.61 GiB on disk, 17.27 GiB loaded VRAM in the short-context benchmark profile, and 0 behavior-suite issues. Q4_K_M remains faster.
+- Custom IQ4_XS `tp=1` is validated as the middle-footprint GGUF option and is the base body for the integrated MTP graft artifact.
+- vLLM GGUF Q4_K_M currently loads but has broken output and is not benchmark-valid.
+- Profile notes: `runs/vllm-initial-profile.md` and `runs/sglang-initial-profile.md`.
+- Quantized control note: `runs/llamacpp-q4-control.md`.
+- GGUF/vLLM materialization audit: `runs/gguf-vllm-materialization-q4.md`.
+- Reusable benchmark summary:
+```bash
+.venv-lab/bin/python scripts/summarize_benchmarks.py runs/*-seq-c*-256.jsonl
+```
+## GPU1 Experiment Lane
+- Lane config: `configs/experiment_lanes.yaml`.
+- Quantization matrix: `configs/quantization_matrix.yaml`.
+- Custom Q3_K_M profile: `runs/llamacpp-q3-k-m-profile.md`.
+- Training plan: `configs/training_plan.yaml`.
+- Source-backed run note: `runs/gpu1-quant-train-plan.md`.
+## Hardware Baseline
+- GPU0: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 97,887 MiB VRAM.
+- GPU1: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 97,887 MiB VRAM.
+- Driver: 580.159.03, CUDA runtime reported by driver: 13.0.
+- Host had no `nvcc` and no `cmake` at project start. A local `.venv-build` now provides CMake, Ninja, and CUDA 13.2 wheels for the llama.cpp CUDA build.
+## Quick Start
+Create the lightweight lab environment:
+```bash
+cd /home/ripper/ornith-35b-lab
+uv venv .venv-lab --python 3.12
+source .venv-lab/bin/activate
+uv pip install -r requirements-lab.txt
+```
+Run the metadata audit without downloading full model weights:
 ```bash
+python scripts/audit_model.py --output runs/audit-ornith-35b.json
+.venv-lab/bin/python scripts/check_audit_gates.py runs/audit-ornith-35b.json
+```
+Run the combined reusable investigation report:
+```bash
+.venv-lab/bin/python scripts/run_model_investigation.py \
+  --output runs/investigation-ornith-metadata.json
+.venv-lab/bin/python scripts/summarize_investigation.py \
+  runs/investigation-ornith-metadata.json \
+  --output runs/investigation-ornith-metadata.md
+```
+Add `--base-url` and `--served-model` when a live OpenAI-compatible backend is
+running to include the behavior suite in the same report.
+Verify the published HF release and local staged support-file hashes:
+```bash
+.venv-lab/bin/python scripts/verify_hf_release.py \
+  --output runs/hf-release-verification.json
+```
+When running from a clone of the HF repo rather than this lab workspace, pass
+`--local-dir . --manifest ARTIFACT_MANIFEST.json`.
+Run the local CPU-only validation suite:
+```bash
+.venv-lab/bin/python scripts/run_cpu_validation_suite.py \
+  --root . \
+  --output runs/cpu-validation-suite.json \
+  --markdown-output runs/cpu-validation-suite.md \
+  --fail-on-error
+```
+This runs idle-state, serving profile, quant artifact, MTP profile,
+release-readiness, release-reference, project-requirement, and goal-completion validation in
+dependency order. It does not start model serving, SFT, quantization, or eval
+workloads. Its summary includes both failed steps and report-level warnings so
+expected incomplete-project warnings stay visible without failing the suite.
+Build the CPU-only release-readiness report:
+```bash
+.venv-lab/bin/python scripts/build_release_readiness_report.py \
+  --allow-unreleased-sft \
+  --serving-profile-validation hf_upload/probes/serving-profile-validation.json \
+  --quant-artifact-validation hf_upload/probes/quant-artifact-validation.json \
+  --mtp-profile-validation hf_upload/probes/mtp-profile-validation.json \
+  --required-doc hf_upload/README.md \
+  --required-doc hf_upload/RELEASE_AUDIT.md \
+  --required-doc hf_upload/PROJECT_STATUS_AUDIT.md \
+  --required-doc hf_upload/MODEL_INVESTIGATION_HARNESS.md \
+  --required-doc hf_upload/SFT_REMEDIATION_PLAN.md \
+  --output runs/release-readiness-report.json \
+  --markdown-output runs/release-readiness-report.md \
+  --fail-on-serving-not-ready
+```
+The current readiness report passes the GGUF serving release and marks the full
+project incomplete because no SFT/post-training adapter is released.
+Validate the machine-readable serving profiles:
+```bash
+.venv-lab/bin/python scripts/validate_serving_profiles.py \
+  --catalog configs/serving_profiles.yaml \
+  --manifest hf_upload/ARTIFACT_MANIFEST.json \
+  --hf-root hf_upload \
+  --output runs/serving-profile-validation.json \
+  --markdown-output runs/serving-profile-validation.md \
+  --fail-on-error
+```
+Validate the machine-readable quant artifacts:
+```bash
+.venv-lab/bin/python scripts/validate_quant_artifacts.py \
+  --catalog configs/quant_artifacts.yaml \
+  --manifest hf_upload/ARTIFACT_MANIFEST.json \
+  --hf-root hf_upload \
+  --output runs/quant-artifact-validation.json \
+  --markdown-output runs/quant-artifact-validation.md \
+  --fail-on-error
+```
+Validate the machine-readable MTP profiles:
+```bash
+.venv-lab/bin/python scripts/validate_mtp_profiles.py \
+  --catalog configs/mtp_profiles.yaml \
+  --manifest hf_upload/ARTIFACT_MANIFEST.json \
+  --hf-root hf_upload \
+  --output runs/mtp-profile-validation.json \
+  --markdown-output runs/mtp-profile-validation.md \
+  --fail-on-error
+```
+Validate the project-level requirement status:
+```bash
+.venv-lab/bin/python scripts/validate_project_requirements.py \
+  --catalog configs/project_requirements.yaml \
+  --root . \
+  --output runs/project-requirement-validation.json \
+  --markdown-output runs/project-requirement-validation.md \
+  --fail-on-error
+```
+This report intentionally separates `serving_release_status=pass` from
+`full_project_status=incomplete` until a release-candidate SFT/post-training
+adapter exists.
+Validate the stopped/idle runtime state:
+```bash
+.venv-lab/bin/python scripts/validate_idle_state.py \
+  --config configs/idle_state_policy.yaml \
+  --root . \
+  --output runs/idle-state-validation.json \
+  --markdown-output runs/idle-state-validation.md \
+  --fail-on-error
+```
+This is a live local guard: it checks the saved SFT stop record, `nvidia-smi`
+GPU utilization and compute apps, and known training/server/quant process names.
+Run reusable behavior probes against any OpenAI-compatible backend:
+```bash
+.venv-lab/bin/python scripts/probe_model_behavior.py \
+  --base-url http://localhost:8000/v1 \
+  --model Ornith-1.0-35B \
+  --output runs/behavior-vllm-bf16.json
+```
+The suite lives in `configs/model_behavior_suite.yaml` and is designed to be
+copied to other models. It turns model bugs into named, repeatable checks for
+coding correctness, JSON following, chat-template leakage, decode corruption,
+and optional tool-call handling.
+Run the GGUF/vLLM materialization audit for a quantized artifact:
+```bash
+.venv-lab/bin/python scripts/audit_gguf_vllm_materialization.py \
+  --model deepreinforce-ai/Ornith-1.0-35B \
+  --gguf deepreinforce-ai/Ornith-1.0-35B-GGUF:Q4_K_M \
+  --output runs/gguf-vllm-materialization-q4.json
 ```
+Start one serving backend on GPU0 after installing that backend in its own environment:
 ```bash
+CUDA_VISIBLE_DEVICES=0 scripts/serve_vllm_gpu0.sh
+CUDA_VISIBLE_DEVICES=0 scripts/serve_sglang_gpu0.sh
+CUDA_VISIBLE_DEVICES=0 scripts/serve_llamacpp_gpu0.sh
+```
+Serve the custom local Q3_K_M GGUF:
+```bash
+QUANT=Q3_K_M PORT=8002 CTX_SIZE=8192 PARALLEL=16 \
   scripts/serve_llamacpp_gpu0.sh
 ```
+Run a smoke benchmark against an OpenAI-compatible server:
+```bash
+python scripts/bench_openai.py \
+  --base-url http://localhost:8000/v1 \
+  --model Ornith-1.0-35B \
+  --concurrency 1 \
+  --max-tokens 256 \
+  --output runs/smoke-bench.jsonl
+```
+Probe the OpenAI-compatible server contract after each backend/profile change:
+```bash
+.venv-lab/bin/python scripts/probe_openai_contract.py \
+  --base-url http://localhost:8000/v1 \
+  --model Ornith-1.0-35B \
+  --output runs/server-contract.json
+```
+Prepare and gate SFT data before launching GPU training:
+```bash
+.venv-train/bin/python scripts/render_sft_jsonl.py \
+  --input data/train/coding_sft_messages.jsonl \
+  --output data/train/coding_sft.jsonl
+.venv-train/bin/python scripts/validate_sft_jsonl.py \
+  --train-jsonl data/train/coding_sft.jsonl \
+  --eval-jsonl data/eval/coding_sft_eval.jsonl \
+  --output runs/sft-data-validation.json
+```
+`scripts/train_sft_lora.py` runs the same validation before importing Torch and
+loading the 35B model unless `--skip-data-validation` is passed.
+The production SFT run was stopped at user request. Latest durable checkpoint:
+`artifacts/train/ornith-35b-coding-lora-prod-20k/checkpoint-8000`; stop record:
+`runs/ornith-35b-coding-lora-prod-20k.stopped.json`.
+CPU-side SFT remediation is prepared but not running. The repair path is defined
+in `configs/sft_repair_blend.yaml` and uses:
+- `scripts/build_sft_repair_set.py`
+- `scripts/build_sft_code_only_from_fences.py`
+- `scripts/build_sft_blend.py`
+- `data/train/coding_sft_prod_repair_strong.jsonl`
+- `data/eval/coding_sft_prod_repair_strong_eval.jsonl`
+- `runs/sft-data-validation-prod-repair-strong.json`
+- `runs/sft-format-audit-prod-repair-strong.json`
+- `runs/sft-format-gate-code-only-extract.json`
+- `runs/sft-format-gate-prod-repair-strong.json`
+- `runs/sft-resume-preflight-strong.json`
+- `runs/sft-format-gate-strict-repair.json`
+- `runs/sft-format-gate-prod-repair.json`
+The recommended strong repair split has 44,096 train rows, 2,312 eval rows, and
+no blocking validation issues. It reduces train message code fences to 54.43%
+and `Here's`/`Here is` preambles to 30.49%. The code-only extraction gate passes
+8/8 checks, and the strong blended repair gate passes 10/10 checks. Do not
+resume SFT unless explicitly requested.
+Before any future resume, run the report-only guard:
+```bash
+.venv-lab/bin/python scripts/preflight_sft_resume.py \
+  --output runs/sft-resume-preflight-strong.json
+```
+The current preflight passes 16/16 checks and includes the exact resume command
+preview without launching training.
+Run the production SFT monitor only for a resumed run:
+```bash
+scripts/monitor_sft_prod.sh
+```
+Start the automatic checkpoint behavior gate watcher only for a resumed run:
+```bash
+scripts/watch_sft_checkpoint_gate.sh
+```
+Evaluate a LoRA checkpoint against the reusable behavior suite:
+```bash
+CUDA_VISIBLE_DEVICES=0 \
+.venv-train/bin/python scripts/eval_lora_behavior.py \
+  --base-model deepreinforce-ai/Ornith-1.0-35B \
+  --adapter artifacts/train/ornith-35b-coding-lora-prod-20k/checkpoint-<step> \
+  --suite configs/model_behavior_suite.yaml \
+  --output runs/eval-lora-behavior-checkpoint-<step>.json
+```
+The LoRA evaluator defaults to final-answer mode, matching llama.cpp
+`REASONING=off`. Add `--enable-thinking` only for explicit reasoning-mode tests.

benchmarks/raw/swebench/ornith-q6-think-verified100-summary.json ADDED Viewed

	@@ -0,0 +1,67 @@

+{
+  "run": "ornith_q6_think_verified100",
+  "date": "2026-06-29",
+  "model": "Ornith-1.0-35B Q6_K GGUF",
+  "reasoning": {
+    "enabled": true,
+    "enable_thinking": true,
+    "preserve_thinking": true,
+    "auto_disable_thinking_with_tools": false,
+    "explicit_thinking_budget": null
+  },
+  "sampling": {
+    "temperature": 1.0,
+    "top_p": 1.0,
+    "max_output_tokens": 8192
+  },
+  "dataset": {
+    "name": "SWE-bench/SWE-bench_Verified",
+    "split": "test",
+    "slice": "0:100"
+  },
+  "score": {
+    "resolved": 69,
+    "total": 100,
+    "resolved_pct": 69.0,
+    "unresolved": 25,
+    "empty_patch_or_timeout": 6,
+    "infra_errors_after_rerun": 0
+  },
+  "original_harness_report": {
+    "path": "/home/ripper/dsv4-local/eval/swebench/reports/ornith_q6_think_verified100.json",
+    "resolved": 69,
+    "unresolved": 23,
+    "empty_patch": 6,
+    "infra_errors": 2,
+    "note": "The wrapper's total_instances=500 / metric_value=13.80 is not the intended 100-task slice score because the first harness invocation did not pass instance filters."
+  },
+  "rerun_report_for_original_errors": {
+    "path": "/home/ripper/dsv4-local/eval/swebench/reports/ornith_q6_think_verified100_rerun_errors.json",
+    "instances": [
+      "astropy__astropy-8707",
+      "astropy__astropy-8872"
+    ],
+    "resolved": 0,
+    "unresolved": 2,
+    "infra_errors": 0
+  },
+  "prediction_artifacts": {
+    "merged_predictions": "/home/ripper/ornith-35b-lab/runs/reasoning-verified100-20260629/predictions/q6_think_verified100_merged/preds.json",
+    "merge_manifest": "/home/ripper/ornith-35b-lab/runs/reasoning-verified100-20260629/predictions/q6_think_verified100_merged/merge_manifest.json"
+  },
+  "trace_reliability": {
+    "path": "/home/ripper/ornith-35b-lab/runs/reasoning-verified100-20260629/trace_reliability_q6_think_verified100.json",
+    "trajectory_count": 100,
+    "assistant_turns": 6821,
+    "assistant_tool_call_turns": 6821,
+    "assistant_no_tool_call_turns": 0,
+    "finish_reason_length_turns": 0,
+    "format_error_no_tool_calls": 23,
+    "duplicate_tool_call_turns": 2,
+    "hidden_reasoning_repeat_turns": 0,
+    "visible_think_tag_turns": 1,
+    "visible_tool_tag_turns": 1,
+    "submitted": 94,
+    "time_exceeded": 6
+  }
+}

benchmarks/raw/swebench/ornith_iq4_xs_mtp_graft_verified_mini.json ADDED Viewed

	@@ -0,0 +1,170 @@

+{
+    "total_instances": 50,
+    "submitted_instances": 50,
+    "completed_instances": 48,
+    "resolved_instances": 30,
+    "unresolved_instances": 18,
+    "empty_patch_instances": 2,
+    "error_instances": 0,
+    "completed_ids": [
+        "django__django-11790",
+        "django__django-11815",
+        "django__django-11848",
+        "django__django-11880",
+        "django__django-11885",
+        "django__django-11951",
+        "django__django-11964",
+        "django__django-11999",
+        "django__django-12039",
+        "django__django-12050",
+        "django__django-12143",
+        "django__django-12155",
+        "django__django-12193",
+        "django__django-12209",
+        "django__django-12262",
+        "django__django-12273",
+        "django__django-12276",
+        "django__django-12304",
+        "django__django-12308",
+        "django__django-12325",
+        "django__django-12406",
+        "django__django-12708",
+        "django__django-12713",
+        "django__django-12774",
+        "django__django-9296",
+        "sphinx-doc__sphinx-10323",
+        "sphinx-doc__sphinx-10435",
+        "sphinx-doc__sphinx-10466",
+        "sphinx-doc__sphinx-10673",
+        "sphinx-doc__sphinx-11510",
+        "sphinx-doc__sphinx-7590",
+        "sphinx-doc__sphinx-7748",
+        "sphinx-doc__sphinx-7757",
+        "sphinx-doc__sphinx-7985",
+        "sphinx-doc__sphinx-8035",
+        "sphinx-doc__sphinx-8056",
+        "sphinx-doc__sphinx-8265",
+        "sphinx-doc__sphinx-8269",
+        "sphinx-doc__sphinx-8475",
+        "sphinx-doc__sphinx-8548",
+        "sphinx-doc__sphinx-8551",
+        "sphinx-doc__sphinx-8638",
+        "sphinx-doc__sphinx-8721",
+        "sphinx-doc__sphinx-9230",
+        "sphinx-doc__sphinx-9281",
+        "sphinx-doc__sphinx-9320",
+        "sphinx-doc__sphinx-9367",
+        "sphinx-doc__sphinx-9698"
+    ],
+    "incomplete_ids": [],
+    "empty_patch_ids": [
+        "sphinx-doc__sphinx-9229",
+        "sphinx-doc__sphinx-9461"
+    ],
+    "submitted_ids": [
+        "django__django-11790",
+        "django__django-11815",
+        "django__django-11848",
+        "django__django-11880",
+        "django__django-11885",
+        "django__django-11951",
+        "django__django-11964",
+        "django__django-11999",
+        "django__django-12039",
+        "django__django-12050",
+        "django__django-12143",
+        "django__django-12155",
+        "django__django-12193",
+        "django__django-12209",
+        "django__django-12262",
+        "django__django-12273",
+        "django__django-12276",
+        "django__django-12304",
+        "django__django-12308",
+        "django__django-12325",
+        "django__django-12406",
+        "django__django-12708",
+        "django__django-12713",
+        "django__django-12774",
+        "django__django-9296",
+        "sphinx-doc__sphinx-10323",
+        "sphinx-doc__sphinx-10435",
+        "sphinx-doc__sphinx-10466",
+        "sphinx-doc__sphinx-10673",
+        "sphinx-doc__sphinx-11510",
+        "sphinx-doc__sphinx-7590",
+        "sphinx-doc__sphinx-7748",
+        "sphinx-doc__sphinx-7757",
+        "sphinx-doc__sphinx-7985",
+        "sphinx-doc__sphinx-8035",
+        "sphinx-doc__sphinx-8056",
+        "sphinx-doc__sphinx-8265",
+        "sphinx-doc__sphinx-8269",
+        "sphinx-doc__sphinx-8475",
+        "sphinx-doc__sphinx-8548",
+        "sphinx-doc__sphinx-8551",
+        "sphinx-doc__sphinx-8638",
+        "sphinx-doc__sphinx-8721",
+        "sphinx-doc__sphinx-9229",
+        "sphinx-doc__sphinx-9230",
+        "sphinx-doc__sphinx-9281",
+        "sphinx-doc__sphinx-9320",
+        "sphinx-doc__sphinx-9367",
+        "sphinx-doc__sphinx-9461",
+        "sphinx-doc__sphinx-9698"
+    ],
+    "resolved_ids": [
+        "django__django-11815",
+        "django__django-11880",
+        "django__django-11951",
+        "django__django-11964",
+        "django__django-11999",
+        "django__django-12039",
+        "django__django-12050",
+        "django__django-12143",
+        "django__django-12155",
+        "django__django-12193",
+        "django__django-12209",
+        "django__django-12262",
+        "django__django-12276",
+        "django__django-12304",
+        "django__django-12708",
+        "django__django-12713",
+        "django__django-12774",
+        "django__django-9296",
+        "sphinx-doc__sphinx-10466",
+        "sphinx-doc__sphinx-7757",
+        "sphinx-doc__sphinx-8035",
+        "sphinx-doc__sphinx-8269",
+        "sphinx-doc__sphinx-8475",
+        "sphinx-doc__sphinx-8551",
+        "sphinx-doc__sphinx-8721",
+        "sphinx-doc__sphinx-9230",
+        "sphinx-doc__sphinx-9281",
+        "sphinx-doc__sphinx-9320",
+        "sphinx-doc__sphinx-9367",
+        "sphinx-doc__sphinx-9698"
+    ],
+    "unresolved_ids": [
+        "django__django-11790",
+        "django__django-11848",
+        "django__django-11885",
+        "django__django-12273",
+        "django__django-12308",
+        "django__django-12325",
+        "django__django-12406",
+        "sphinx-doc__sphinx-10323",
+        "sphinx-doc__sphinx-10435",
+        "sphinx-doc__sphinx-10673",
+        "sphinx-doc__sphinx-11510",
+        "sphinx-doc__sphinx-7590",
+        "sphinx-doc__sphinx-7748",
+        "sphinx-doc__sphinx-7985",
+        "sphinx-doc__sphinx-8056",
+        "sphinx-doc__sphinx-8265",
+        "sphinx-doc__sphinx-8548",
+        "sphinx-doc__sphinx-8638"
+    ],
+    "error_ids": [],
+    "schema_version": 2
+}

benchmarks/raw/swebench/ornith_iq4_xs_verified_mini.json ADDED Viewed

	@@ -0,0 +1,167 @@

+{
+    "total_instances": 50,
+    "submitted_instances": 50,
+    "completed_instances": 45,
+    "resolved_instances": 30,
+    "unresolved_instances": 15,
+    "empty_patch_instances": 5,
+    "error_instances": 0,
+    "completed_ids": [
+        "django__django-11790",
+        "django__django-11815",
+        "django__django-11848",
+        "django__django-11880",
+        "django__django-11885",
+        "django__django-11951",
+        "django__django-11999",
+        "django__django-12039",
+        "django__django-12050",
+        "django__django-12143",
+        "django__django-12155",
+        "django__django-12193",
+        "django__django-12262",
+        "django__django-12273",
+        "django__django-12276",
+        "django__django-12304",
+        "django__django-12308",
+        "django__django-12325",
+        "django__django-12406",
+        "django__django-12708",
+        "django__django-12713",
+        "django__django-12774",
+        "django__django-9296",
+        "sphinx-doc__sphinx-10323",
+        "sphinx-doc__sphinx-10435",
+        "sphinx-doc__sphinx-10466",
+        "sphinx-doc__sphinx-11510",
+        "sphinx-doc__sphinx-7590",
+        "sphinx-doc__sphinx-7748",
+        "sphinx-doc__sphinx-7757",
+        "sphinx-doc__sphinx-7985",
+        "sphinx-doc__sphinx-8035",
+        "sphinx-doc__sphinx-8056",
+        "sphinx-doc__sphinx-8265",
+        "sphinx-doc__sphinx-8269",
+        "sphinx-doc__sphinx-8475",
+        "sphinx-doc__sphinx-8548",
+        "sphinx-doc__sphinx-8551",
+        "sphinx-doc__sphinx-8638",
+        "sphinx-doc__sphinx-8721",
+        "sphinx-doc__sphinx-9230",
+        "sphinx-doc__sphinx-9281",
+        "sphinx-doc__sphinx-9320",
+        "sphinx-doc__sphinx-9367",
+        "sphinx-doc__sphinx-9698"
+    ],
+    "incomplete_ids": [],
+    "empty_patch_ids": [
+        "django__django-11964",
+        "django__django-12209",
+        "sphinx-doc__sphinx-10673",
+        "sphinx-doc__sphinx-9229",
+        "sphinx-doc__sphinx-9461"
+    ],
+    "submitted_ids": [
+        "django__django-11790",
+        "django__django-11815",
+        "django__django-11848",
+        "django__django-11880",
+        "django__django-11885",
+        "django__django-11951",
+        "django__django-11964",
+        "django__django-11999",
+        "django__django-12039",
+        "django__django-12050",
+        "django__django-12143",
+        "django__django-12155",
+        "django__django-12193",
+        "django__django-12209",
+        "django__django-12262",
+        "django__django-12273",
+        "django__django-12276",
+        "django__django-12304",
+        "django__django-12308",
+        "django__django-12325",
+        "django__django-12406",
+        "django__django-12708",
+        "django__django-12713",
+        "django__django-12774",
+        "django__django-9296",
+        "sphinx-doc__sphinx-10323",
+        "sphinx-doc__sphinx-10435",
+        "sphinx-doc__sphinx-10466",
+        "sphinx-doc__sphinx-10673",
+        "sphinx-doc__sphinx-11510",
+        "sphinx-doc__sphinx-7590",
+        "sphinx-doc__sphinx-7748",
+        "sphinx-doc__sphinx-7757",
+        "sphinx-doc__sphinx-7985",
+        "sphinx-doc__sphinx-8035",
+        "sphinx-doc__sphinx-8056",
+        "sphinx-doc__sphinx-8265",
+        "sphinx-doc__sphinx-8269",
+        "sphinx-doc__sphinx-8475",
+        "sphinx-doc__sphinx-8548",
+        "sphinx-doc__sphinx-8551",
+        "sphinx-doc__sphinx-8638",
+        "sphinx-doc__sphinx-8721",
+        "sphinx-doc__sphinx-9229",
+        "sphinx-doc__sphinx-9230",
+        "sphinx-doc__sphinx-9281",
+        "sphinx-doc__sphinx-9320",
+        "sphinx-doc__sphinx-9367",
+        "sphinx-doc__sphinx-9461",
+        "sphinx-doc__sphinx-9698"
+    ],
+    "resolved_ids": [
+        "django__django-11790",
+        "django__django-11848",
+        "django__django-11880",
+        "django__django-11951",
+        "django__django-11999",
+        "django__django-12039",
+        "django__django-12050",
+        "django__django-12143",
+        "django__django-12155",
+        "django__django-12262",
+        "django__django-12276",
+        "django__django-12304",
+        "django__django-12708",
+        "django__django-12713",
+        "django__django-12774",
+        "django__django-9296",
+        "sphinx-doc__sphinx-10466",
+        "sphinx-doc__sphinx-7757",
+        "sphinx-doc__sphinx-8035",
+        "sphinx-doc__sphinx-8265",
+        "sphinx-doc__sphinx-8269",
+        "sphinx-doc__sphinx-8475",
+        "sphinx-doc__sphinx-8548",
+        "sphinx-doc__sphinx-8551",
+        "sphinx-doc__sphinx-8721",
+        "sphinx-doc__sphinx-9230",
+        "sphinx-doc__sphinx-9281",
+        "sphinx-doc__sphinx-9320",
+        "sphinx-doc__sphinx-9367",
+        "sphinx-doc__sphinx-9698"
+    ],
+    "unresolved_ids": [
+        "django__django-11815",
+        "django__django-11885",
+        "django__django-12193",
+        "django__django-12273",
+        "django__django-12308",
+        "django__django-12325",
+        "django__django-12406",
+        "sphinx-doc__sphinx-10323",
+        "sphinx-doc__sphinx-10435",
+        "sphinx-doc__sphinx-11510",
+        "sphinx-doc__sphinx-7590",
+        "sphinx-doc__sphinx-7748",
+        "sphinx-doc__sphinx-7985",
+        "sphinx-doc__sphinx-8056",
+        "sphinx-doc__sphinx-8638"
+    ],
+    "error_ids": [],
+    "schema_version": 2
+}

benchmarks/raw/swebench/ornith_q3_k_m_verified_mini.json ADDED Viewed

	@@ -0,0 +1,169 @@

+{
+    "total_instances": 50,
+    "submitted_instances": 50,
+    "completed_instances": 47,
+    "resolved_instances": 32,
+    "unresolved_instances": 15,
+    "empty_patch_instances": 3,
+    "error_instances": 0,
+    "completed_ids": [
+        "django__django-11790",
+        "django__django-11815",
+        "django__django-11848",
+        "django__django-11880",
+        "django__django-11951",
+        "django__django-11964",
+        "django__django-11999",
+        "django__django-12039",
+        "django__django-12050",
+        "django__django-12143",
+        "django__django-12155",
+        "django__django-12193",
+        "django__django-12209",
+        "django__django-12262",
+        "django__django-12276",
+        "django__django-12304",
+        "django__django-12308",
+        "django__django-12325",
+        "django__django-12406",
+        "django__django-12708",
+        "django__django-12713",
+        "django__django-12774",
+        "django__django-9296",
+        "sphinx-doc__sphinx-10323",
+        "sphinx-doc__sphinx-10435",
+        "sphinx-doc__sphinx-10466",
+        "sphinx-doc__sphinx-10673",
+        "sphinx-doc__sphinx-11510",
+        "sphinx-doc__sphinx-7590",
+        "sphinx-doc__sphinx-7748",
+        "sphinx-doc__sphinx-7757",
+        "sphinx-doc__sphinx-7985",
+        "sphinx-doc__sphinx-8035",
+        "sphinx-doc__sphinx-8056",
+        "sphinx-doc__sphinx-8265",
+        "sphinx-doc__sphinx-8269",
+        "sphinx-doc__sphinx-8475",
+        "sphinx-doc__sphinx-8548",
+        "sphinx-doc__sphinx-8551",
+        "sphinx-doc__sphinx-8638",
+        "sphinx-doc__sphinx-8721",
+        "sphinx-doc__sphinx-9230",
+        "sphinx-doc__sphinx-9281",
+        "sphinx-doc__sphinx-9320",
+        "sphinx-doc__sphinx-9367",
+        "sphinx-doc__sphinx-9461",
+        "sphinx-doc__sphinx-9698"
+    ],
+    "incomplete_ids": [],
+    "empty_patch_ids": [
+        "django__django-11885",
+        "django__django-12273",
+        "sphinx-doc__sphinx-9229"
+    ],
+    "submitted_ids": [
+        "django__django-11790",
+        "django__django-11815",
+        "django__django-11848",
+        "django__django-11880",
+        "django__django-11885",
+        "django__django-11951",
+        "django__django-11964",
+        "django__django-11999",
+        "django__django-12039",
+        "django__django-12050",
+        "django__django-12143",
+        "django__django-12155",
+        "django__django-12193",
+        "django__django-12209",
+        "django__django-12262",
+        "django__django-12273",
+        "django__django-12276",
+        "django__django-12304",
+        "django__django-12308",
+        "django__django-12325",
+        "django__django-12406",
+        "django__django-12708",
+        "django__django-12713",
+        "django__django-12774",
+        "django__django-9296",
+        "sphinx-doc__sphinx-10323",
+        "sphinx-doc__sphinx-10435",
+        "sphinx-doc__sphinx-10466",
+        "sphinx-doc__sphinx-10673",
+        "sphinx-doc__sphinx-11510",
+        "sphinx-doc__sphinx-7590",
+        "sphinx-doc__sphinx-7748",
+        "sphinx-doc__sphinx-7757",
+        "sphinx-doc__sphinx-7985",
+        "sphinx-doc__sphinx-8035",
+        "sphinx-doc__sphinx-8056",
+        "sphinx-doc__sphinx-8265",
+        "sphinx-doc__sphinx-8269",
+        "sphinx-doc__sphinx-8475",
+        "sphinx-doc__sphinx-8548",
+        "sphinx-doc__sphinx-8551",
+        "sphinx-doc__sphinx-8638",
+        "sphinx-doc__sphinx-8721",
+        "sphinx-doc__sphinx-9229",
+        "sphinx-doc__sphinx-9230",
+        "sphinx-doc__sphinx-9281",
+        "sphinx-doc__sphinx-9320",
+        "sphinx-doc__sphinx-9367",
+        "sphinx-doc__sphinx-9461",
+        "sphinx-doc__sphinx-9698"
+    ],
+    "resolved_ids": [
+        "django__django-11815",
+        "django__django-11848",
+        "django__django-11880",
+        "django__django-11951",
+        "django__django-11964",
+        "django__django-11999",
+        "django__django-12039",
+        "django__django-12050",
+        "django__django-12143",
+        "django__django-12155",
+        "django__django-12193",
+        "django__django-12209",
+        "django__django-12262",
+        "django__django-12276",
+        "django__django-12304",
+        "django__django-12708",
+        "django__django-12713",
+        "django__django-12774",
+        "django__django-9296",
+        "sphinx-doc__sphinx-10466",
+        "sphinx-doc__sphinx-10673",
+        "sphinx-doc__sphinx-7757",
+        "sphinx-doc__sphinx-8035",
+        "sphinx-doc__sphinx-8265",
+        "sphinx-doc__sphinx-8269",
+        "sphinx-doc__sphinx-8475",
+        "sphinx-doc__sphinx-8551",
+        "sphinx-doc__sphinx-8721",
+        "sphinx-doc__sphinx-9281",
+        "sphinx-doc__sphinx-9320",
+        "sphinx-doc__sphinx-9367",
+        "sphinx-doc__sphinx-9698"
+    ],
+    "unresolved_ids": [
+        "django__django-11790",
+        "django__django-12308",
+        "django__django-12325",
+        "django__django-12406",
+        "sphinx-doc__sphinx-10323",
+        "sphinx-doc__sphinx-10435",
+        "sphinx-doc__sphinx-11510",
+        "sphinx-doc__sphinx-7590",
+        "sphinx-doc__sphinx-7748",
+        "sphinx-doc__sphinx-7985",
+        "sphinx-doc__sphinx-8056",
+        "sphinx-doc__sphinx-8548",
+        "sphinx-doc__sphinx-8638",
+        "sphinx-doc__sphinx-9230",
+        "sphinx-doc__sphinx-9461"
+    ],
+    "error_ids": [],
+    "schema_version": 2
+}

benchmarks/raw/swebench/ornith_q4_k_m_verified_mini.json ADDED Viewed

	@@ -0,0 +1,170 @@

+{
+    "total_instances": 50,
+    "submitted_instances": 50,
+    "completed_instances": 48,
+    "resolved_instances": 31,
+    "unresolved_instances": 17,
+    "empty_patch_instances": 2,
+    "error_instances": 0,
+    "completed_ids": [
+        "django__django-11790",
+        "django__django-11815",
+        "django__django-11848",
+        "django__django-11880",
+        "django__django-11885",
+        "django__django-11951",
+        "django__django-11964",
+        "django__django-11999",
+        "django__django-12039",
+        "django__django-12050",
+        "django__django-12143",
+        "django__django-12155",
+        "django__django-12193",
+        "django__django-12262",
+        "django__django-12273",
+        "django__django-12276",
+        "django__django-12304",
+        "django__django-12308",
+        "django__django-12325",
+        "django__django-12406",
+        "django__django-12708",
+        "django__django-12713",
+        "django__django-12774",
+        "django__django-9296",
+        "sphinx-doc__sphinx-10323",
+        "sphinx-doc__sphinx-10435",
+        "sphinx-doc__sphinx-10466",
+        "sphinx-doc__sphinx-10673",
+        "sphinx-doc__sphinx-11510",
+        "sphinx-doc__sphinx-7590",
+        "sphinx-doc__sphinx-7748",
+        "sphinx-doc__sphinx-7757",
+        "sphinx-doc__sphinx-7985",
+        "sphinx-doc__sphinx-8035",
+        "sphinx-doc__sphinx-8056",
+        "sphinx-doc__sphinx-8265",
+        "sphinx-doc__sphinx-8269",
+        "sphinx-doc__sphinx-8475",
+        "sphinx-doc__sphinx-8548",
+        "sphinx-doc__sphinx-8551",
+        "sphinx-doc__sphinx-8638",
+        "sphinx-doc__sphinx-8721",
+        "sphinx-doc__sphinx-9230",
+        "sphinx-doc__sphinx-9281",
+        "sphinx-doc__sphinx-9320",
+        "sphinx-doc__sphinx-9367",
+        "sphinx-doc__sphinx-9461",
+        "sphinx-doc__sphinx-9698"
+    ],
+    "incomplete_ids": [],
+    "empty_patch_ids": [
+        "django__django-12209",
+        "sphinx-doc__sphinx-9229"
+    ],
+    "submitted_ids": [
+        "django__django-11790",
+        "django__django-11815",
+        "django__django-11848",
+        "django__django-11880",
+        "django__django-11885",
+        "django__django-11951",
+        "django__django-11964",
+        "django__django-11999",
+        "django__django-12039",
+        "django__django-12050",
+        "django__django-12143",
+        "django__django-12155",
+        "django__django-12193",
+        "django__django-12209",
+        "django__django-12262",
+        "django__django-12273",
+        "django__django-12276",
+        "django__django-12304",
+        "django__django-12308",
+        "django__django-12325",
+        "django__django-12406",
+        "django__django-12708",
+        "django__django-12713",
+        "django__django-12774",
+        "django__django-9296",
+        "sphinx-doc__sphinx-10323",
+        "sphinx-doc__sphinx-10435",
+        "sphinx-doc__sphinx-10466",
+        "sphinx-doc__sphinx-10673",
+        "sphinx-doc__sphinx-11510",
+        "sphinx-doc__sphinx-7590",
+        "sphinx-doc__sphinx-7748",
+        "sphinx-doc__sphinx-7757",
+        "sphinx-doc__sphinx-7985",
+        "sphinx-doc__sphinx-8035",
+        "sphinx-doc__sphinx-8056",
+        "sphinx-doc__sphinx-8265",
+        "sphinx-doc__sphinx-8269",
+        "sphinx-doc__sphinx-8475",
+        "sphinx-doc__sphinx-8548",
+        "sphinx-doc__sphinx-8551",
+        "sphinx-doc__sphinx-8638",
+        "sphinx-doc__sphinx-8721",
+        "sphinx-doc__sphinx-9229",
+        "sphinx-doc__sphinx-9230",
+        "sphinx-doc__sphinx-9281",
+        "sphinx-doc__sphinx-9320",
+        "sphinx-doc__sphinx-9367",
+        "sphinx-doc__sphinx-9461",
+        "sphinx-doc__sphinx-9698"
+    ],
+    "resolved_ids": [
+        "django__django-11815",
+        "django__django-11848",
+        "django__django-11880",
+        "django__django-11951",
+        "django__django-11964",
+        "django__django-11999",
+        "django__django-12039",
+        "django__django-12050",
+        "django__django-12143",
+        "django__django-12155",
+        "django__django-12262",
+        "django__django-12273",
+        "django__django-12276",
+        "django__django-12304",
+        "django__django-12708",
+        "django__django-12713",
+        "django__django-12774",
+        "django__django-9296",
+        "sphinx-doc__sphinx-10466",
+        "sphinx-doc__sphinx-7757",
+        "sphinx-doc__sphinx-8035",
+        "sphinx-doc__sphinx-8269",
+        "sphinx-doc__sphinx-8475",
+        "sphinx-doc__sphinx-8548",
+        "sphinx-doc__sphinx-8551",
+        "sphinx-doc__sphinx-8721",
+        "sphinx-doc__sphinx-9230",
+        "sphinx-doc__sphinx-9281",
+        "sphinx-doc__sphinx-9320",
+        "sphinx-doc__sphinx-9367",
+        "sphinx-doc__sphinx-9698"
+    ],
+    "unresolved_ids": [
+        "django__django-11790",
+        "django__django-11885",
+        "django__django-12193",
+        "django__django-12308",
+        "django__django-12325",
+        "django__django-12406",
+        "sphinx-doc__sphinx-10323",
+        "sphinx-doc__sphinx-10435",
+        "sphinx-doc__sphinx-10673",
+        "sphinx-doc__sphinx-11510",
+        "sphinx-doc__sphinx-7590",
+        "sphinx-doc__sphinx-7748",
+        "sphinx-doc__sphinx-7985",
+        "sphinx-doc__sphinx-8056",
+        "sphinx-doc__sphinx-8265",
+        "sphinx-doc__sphinx-8638",
+        "sphinx-doc__sphinx-9461"
+    ],
+    "error_ids": [],
+    "schema_version": 2
+}

benchmarks/raw/swebench/ornith_q5_k_m_verified_mini.json ADDED Viewed

	@@ -0,0 +1,171 @@

+{
+    "total_instances": 50,
+    "submitted_instances": 50,
+    "completed_instances": 49,
+    "resolved_instances": 32,
+    "unresolved_instances": 17,
+    "empty_patch_instances": 1,
+    "error_instances": 0,
+    "completed_ids": [
+        "django__django-11790",
+        "django__django-11815",
+        "django__django-11848",
+        "django__django-11880",
+        "django__django-11885",
+        "django__django-11951",
+        "django__django-11964",
+        "django__django-11999",
+        "django__django-12039",
+        "django__django-12050",
+        "django__django-12143",
+        "django__django-12155",
+        "django__django-12193",
+        "django__django-12209",
+        "django__django-12262",
+        "django__django-12273",
+        "django__django-12276",
+        "django__django-12304",
+        "django__django-12308",
+        "django__django-12325",
+        "django__django-12406",
+        "django__django-12708",
+        "django__django-12713",
+        "django__django-12774",
+        "django__django-9296",
+        "sphinx-doc__sphinx-10323",
+        "sphinx-doc__sphinx-10435",
+        "sphinx-doc__sphinx-10466",
+        "sphinx-doc__sphinx-10673",
+        "sphinx-doc__sphinx-11510",
+        "sphinx-doc__sphinx-7590",
+        "sphinx-doc__sphinx-7748",
+        "sphinx-doc__sphinx-7757",
+        "sphinx-doc__sphinx-7985",
+        "sphinx-doc__sphinx-8035",
+        "sphinx-doc__sphinx-8056",
+        "sphinx-doc__sphinx-8265",
+        "sphinx-doc__sphinx-8269",
+        "sphinx-doc__sphinx-8475",
+        "sphinx-doc__sphinx-8548",
+        "sphinx-doc__sphinx-8551",
+        "sphinx-doc__sphinx-8638",
+        "sphinx-doc__sphinx-8721",
+        "sphinx-doc__sphinx-9230",
+        "sphinx-doc__sphinx-9281",
+        "sphinx-doc__sphinx-9320",
+        "sphinx-doc__sphinx-9367",
+        "sphinx-doc__sphinx-9461",
+        "sphinx-doc__sphinx-9698"
+    ],
+    "incomplete_ids": [],
+    "empty_patch_ids": [
+        "sphinx-doc__sphinx-9229"
+    ],
+    "submitted_ids": [
+        "django__django-11790",
+        "django__django-11815",
+        "django__django-11848",
+        "django__django-11880",
+        "django__django-11885",
+        "django__django-11951",
+        "django__django-11964",
+        "django__django-11999",
+        "django__django-12039",
+        "django__django-12050",
+        "django__django-12143",
+        "django__django-12155",
+        "django__django-12193",
+        "django__django-12209",
+        "django__django-12262",
+        "django__django-12273",
+        "django__django-12276",
+        "django__django-12304",
+        "django__django-12308",
+        "django__django-12325",
+        "django__django-12406",
+        "django__django-12708",
+        "django__django-12713",
+        "django__django-12774",
+        "django__django-9296",
+        "sphinx-doc__sphinx-10323",
+        "sphinx-doc__sphinx-10435",
+        "sphinx-doc__sphinx-10466",
+        "sphinx-doc__sphinx-10673",
+        "sphinx-doc__sphinx-11510",
+        "sphinx-doc__sphinx-7590",
+        "sphinx-doc__sphinx-7748",
+        "sphinx-doc__sphinx-7757",
+        "sphinx-doc__sphinx-7985",
+        "sphinx-doc__sphinx-8035",
+        "sphinx-doc__sphinx-8056",
+        "sphinx-doc__sphinx-8265",
+        "sphinx-doc__sphinx-8269",
+        "sphinx-doc__sphinx-8475",
+        "sphinx-doc__sphinx-8548",
+        "sphinx-doc__sphinx-8551",
+        "sphinx-doc__sphinx-8638",
+        "sphinx-doc__sphinx-8721",
+        "sphinx-doc__sphinx-9229",
+        "sphinx-doc__sphinx-9230",
+        "sphinx-doc__sphinx-9281",
+        "sphinx-doc__sphinx-9320",
+        "sphinx-doc__sphinx-9367",
+        "sphinx-doc__sphinx-9461",
+        "sphinx-doc__sphinx-9698"
+    ],
+    "resolved_ids": [
+        "django__django-11815",
+        "django__django-11848",
+        "django__django-11880",
+        "django__django-11951",
+        "django__django-11964",
+        "django__django-11999",
+        "django__django-12039",
+        "django__django-12050",
+        "django__django-12143",
+        "django__django-12155",
+        "django__django-12209",
+        "django__django-12262",
+        "django__django-12276",
+        "django__django-12304",
+        "django__django-12708",
+        "django__django-12713",
+        "django__django-12774",
+        "django__django-9296",
+        "sphinx-doc__sphinx-10466",
+        "sphinx-doc__sphinx-10673",
+        "sphinx-doc__sphinx-7757",
+        "sphinx-doc__sphinx-8035",
+        "sphinx-doc__sphinx-8265",
+        "sphinx-doc__sphinx-8269",
+        "sphinx-doc__sphinx-8475",
+        "sphinx-doc__sphinx-8551",
+        "sphinx-doc__sphinx-8721",
+        "sphinx-doc__sphinx-9230",
+        "sphinx-doc__sphinx-9281",
+        "sphinx-doc__sphinx-9320",
+        "sphinx-doc__sphinx-9367",
+        "sphinx-doc__sphinx-9698"
+    ],
+    "unresolved_ids": [
+        "django__django-11790",
+        "django__django-11885",
+        "django__django-12193",
+        "django__django-12273",
+        "django__django-12308",
+        "django__django-12325",
+        "django__django-12406",
+        "sphinx-doc__sphinx-10323",
+        "sphinx-doc__sphinx-10435",
+        "sphinx-doc__sphinx-11510",
+        "sphinx-doc__sphinx-7590",
+        "sphinx-doc__sphinx-7748",
+        "sphinx-doc__sphinx-7985",
+        "sphinx-doc__sphinx-8056",
+        "sphinx-doc__sphinx-8548",
+        "sphinx-doc__sphinx-8638",
+        "sphinx-doc__sphinx-9461"
+    ],
+    "error_ids": [],
+    "schema_version": 2
+}

benchmarks/raw/swebench/ornith_q6_k_verified_mini.json ADDED Viewed

	@@ -0,0 +1,170 @@

+{
+    "total_instances": 50,
+    "submitted_instances": 50,
+    "completed_instances": 48,
+    "resolved_instances": 33,
+    "unresolved_instances": 15,
+    "empty_patch_instances": 2,
+    "error_instances": 0,
+    "completed_ids": [
+        "django__django-11790",
+        "django__django-11815",
+        "django__django-11848",
+        "django__django-11880",
+        "django__django-11951",
+        "django__django-11964",
+        "django__django-11999",
+        "django__django-12039",
+        "django__django-12050",
+        "django__django-12143",
+        "django__django-12155",
+        "django__django-12193",
+        "django__django-12209",
+        "django__django-12262",
+        "django__django-12273",
+        "django__django-12276",
+        "django__django-12304",
+        "django__django-12308",
+        "django__django-12325",
+        "django__django-12406",
+        "django__django-12708",
+        "django__django-12713",
+        "django__django-12774",
+        "django__django-9296",
+        "sphinx-doc__sphinx-10323",
+        "sphinx-doc__sphinx-10435",
+        "sphinx-doc__sphinx-10466",
+        "sphinx-doc__sphinx-10673",
+        "sphinx-doc__sphinx-11510",
+        "sphinx-doc__sphinx-7590",
+        "sphinx-doc__sphinx-7748",
+        "sphinx-doc__sphinx-7757",
+        "sphinx-doc__sphinx-7985",
+        "sphinx-doc__sphinx-8035",
+        "sphinx-doc__sphinx-8056",
+        "sphinx-doc__sphinx-8265",
+        "sphinx-doc__sphinx-8269",
+        "sphinx-doc__sphinx-8475",
+        "sphinx-doc__sphinx-8548",
+        "sphinx-doc__sphinx-8551",
+        "sphinx-doc__sphinx-8638",
+        "sphinx-doc__sphinx-8721",
+        "sphinx-doc__sphinx-9230",
+        "sphinx-doc__sphinx-9281",
+        "sphinx-doc__sphinx-9320",
+        "sphinx-doc__sphinx-9367",
+        "sphinx-doc__sphinx-9461",
+        "sphinx-doc__sphinx-9698"
+    ],
+    "incomplete_ids": [],
+    "empty_patch_ids": [
+        "django__django-11885",
+        "sphinx-doc__sphinx-9229"
+    ],
+    "submitted_ids": [
+        "django__django-11790",
+        "django__django-11815",
+        "django__django-11848",
+        "django__django-11880",
+        "django__django-11885",
+        "django__django-11951",
+        "django__django-11964",
+        "django__django-11999",
+        "django__django-12039",
+        "django__django-12050",
+        "django__django-12143",
+        "django__django-12155",
+        "django__django-12193",
+        "django__django-12209",
+        "django__django-12262",
+        "django__django-12273",
+        "django__django-12276",
+        "django__django-12304",
+        "django__django-12308",
+        "django__django-12325",
+        "django__django-12406",
+        "django__django-12708",
+        "django__django-12713",
+        "django__django-12774",
+        "django__django-9296",
+        "sphinx-doc__sphinx-10323",
+        "sphinx-doc__sphinx-10435",
+        "sphinx-doc__sphinx-10466",
+        "sphinx-doc__sphinx-10673",
+        "sphinx-doc__sphinx-11510",
+        "sphinx-doc__sphinx-7590",
+        "sphinx-doc__sphinx-7748",
+        "sphinx-doc__sphinx-7757",
+        "sphinx-doc__sphinx-7985",
+        "sphinx-doc__sphinx-8035",
+        "sphinx-doc__sphinx-8056",
+        "sphinx-doc__sphinx-8265",
+        "sphinx-doc__sphinx-8269",
+        "sphinx-doc__sphinx-8475",
+        "sphinx-doc__sphinx-8548",
+        "sphinx-doc__sphinx-8551",
+        "sphinx-doc__sphinx-8638",
+        "sphinx-doc__sphinx-8721",
+        "sphinx-doc__sphinx-9229",
+        "sphinx-doc__sphinx-9230",
+        "sphinx-doc__sphinx-9281",
+        "sphinx-doc__sphinx-9320",
+        "sphinx-doc__sphinx-9367",
+        "sphinx-doc__sphinx-9461",
+        "sphinx-doc__sphinx-9698"
+    ],
+    "resolved_ids": [
+        "django__django-11815",
+        "django__django-11848",
+        "django__django-11880",
+        "django__django-11951",
+        "django__django-11999",
+        "django__django-12039",
+        "django__django-12050",
+        "django__django-12143",
+        "django__django-12155",
+        "django__django-12193",
+        "django__django-12209",
+        "django__django-12262",
+        "django__django-12276",
+        "django__django-12304",
+        "django__django-12708",
+        "django__django-12713",
+        "django__django-12774",
+        "django__django-9296",
+        "sphinx-doc__sphinx-10466",
+        "sphinx-doc__sphinx-10673",
+        "sphinx-doc__sphinx-7757",
+        "sphinx-doc__sphinx-8035",
+        "sphinx-doc__sphinx-8056",
+        "sphinx-doc__sphinx-8265",
+        "sphinx-doc__sphinx-8269",
+        "sphinx-doc__sphinx-8475",
+        "sphinx-doc__sphinx-8551",
+        "sphinx-doc__sphinx-8721",
+        "sphinx-doc__sphinx-9230",
+        "sphinx-doc__sphinx-9281",
+        "sphinx-doc__sphinx-9320",
+        "sphinx-doc__sphinx-9367",
+        "sphinx-doc__sphinx-9698"
+    ],
+    "unresolved_ids": [
+        "django__django-11790",
+        "django__django-11964",
+        "django__django-12273",
+        "django__django-12308",
+        "django__django-12325",
+        "django__django-12406",
+        "sphinx-doc__sphinx-10323",
+        "sphinx-doc__sphinx-10435",
+        "sphinx-doc__sphinx-11510",
+        "sphinx-doc__sphinx-7590",
+        "sphinx-doc__sphinx-7748",
+        "sphinx-doc__sphinx-7985",
+        "sphinx-doc__sphinx-8548",
+        "sphinx-doc__sphinx-8638",
+        "sphinx-doc__sphinx-9461"
+    ],
+    "error_ids": [],
+    "schema_version": 2
+}

benchmarks/raw/swebench/ornith_q8_0_verified_mini.json ADDED Viewed

	@@ -0,0 +1,169 @@

+{
+    "total_instances": 50,
+    "submitted_instances": 50,
+    "completed_instances": 47,
+    "resolved_instances": 29,
+    "unresolved_instances": 18,
+    "empty_patch_instances": 3,
+    "error_instances": 0,
+    "completed_ids": [
+        "django__django-11790",
+        "django__django-11815",
+        "django__django-11848",
+        "django__django-11880",
+        "django__django-11885",
+        "django__django-11951",
+        "django__django-11964",
+        "django__django-11999",
+        "django__django-12039",
+        "django__django-12050",
+        "django__django-12143",
+        "django__django-12155",
+        "django__django-12193",
+        "django__django-12262",
+        "django__django-12273",
+        "django__django-12276",
+        "django__django-12304",
+        "django__django-12308",
+        "django__django-12325",
+        "django__django-12406",
+        "django__django-12708",
+        "django__django-12713",
+        "django__django-12774",
+        "django__django-9296",
+        "sphinx-doc__sphinx-10435",
+        "sphinx-doc__sphinx-10466",
+        "sphinx-doc__sphinx-10673",
+        "sphinx-doc__sphinx-11510",
+        "sphinx-doc__sphinx-7590",
+        "sphinx-doc__sphinx-7748",
+        "sphinx-doc__sphinx-7757",
+        "sphinx-doc__sphinx-7985",
+        "sphinx-doc__sphinx-8035",
+        "sphinx-doc__sphinx-8056",
+        "sphinx-doc__sphinx-8265",
+        "sphinx-doc__sphinx-8269",
+        "sphinx-doc__sphinx-8475",
+        "sphinx-doc__sphinx-8548",
+        "sphinx-doc__sphinx-8551",
+        "sphinx-doc__sphinx-8638",
+        "sphinx-doc__sphinx-8721",
+        "sphinx-doc__sphinx-9230",
+        "sphinx-doc__sphinx-9281",
+        "sphinx-doc__sphinx-9320",
+        "sphinx-doc__sphinx-9367",
+        "sphinx-doc__sphinx-9461",
+        "sphinx-doc__sphinx-9698"
+    ],
+    "incomplete_ids": [],
+    "empty_patch_ids": [
+        "django__django-12209",
+        "sphinx-doc__sphinx-10323",
+        "sphinx-doc__sphinx-9229"
+    ],
+    "submitted_ids": [
+        "django__django-11790",
+        "django__django-11815",
+        "django__django-11848",
+        "django__django-11880",
+        "django__django-11885",
+        "django__django-11951",
+        "django__django-11964",
+        "django__django-11999",
+        "django__django-12039",
+        "django__django-12050",
+        "django__django-12143",
+        "django__django-12155",
+        "django__django-12193",
+        "django__django-12209",
+        "django__django-12262",
+        "django__django-12273",
+        "django__django-12276",
+        "django__django-12304",
+        "django__django-12308",
+        "django__django-12325",
+        "django__django-12406",
+        "django__django-12708",
+        "django__django-12713",
+        "django__django-12774",
+        "django__django-9296",
+        "sphinx-doc__sphinx-10323",
+        "sphinx-doc__sphinx-10435",
+        "sphinx-doc__sphinx-10466",
+        "sphinx-doc__sphinx-10673",
+        "sphinx-doc__sphinx-11510",
+        "sphinx-doc__sphinx-7590",
+        "sphinx-doc__sphinx-7748",
+        "sphinx-doc__sphinx-7757",
+        "sphinx-doc__sphinx-7985",
+        "sphinx-doc__sphinx-8035",
+        "sphinx-doc__sphinx-8056",
+        "sphinx-doc__sphinx-8265",
+        "sphinx-doc__sphinx-8269",
+        "sphinx-doc__sphinx-8475",
+        "sphinx-doc__sphinx-8548",
+        "sphinx-doc__sphinx-8551",
+        "sphinx-doc__sphinx-8638",
+        "sphinx-doc__sphinx-8721",
+        "sphinx-doc__sphinx-9229",
+        "sphinx-doc__sphinx-9230",
+        "sphinx-doc__sphinx-9281",
+        "sphinx-doc__sphinx-9320",
+        "sphinx-doc__sphinx-9367",
+        "sphinx-doc__sphinx-9461",
+        "sphinx-doc__sphinx-9698"
+    ],
+    "resolved_ids": [
+        "django__django-11815",
+        "django__django-11848",
+        "django__django-11880",
+        "django__django-11951",
+        "django__django-11999",
+        "django__django-12039",
+        "django__django-12050",
+        "django__django-12143",
+        "django__django-12155",
+        "django__django-12262",
+        "django__django-12276",
+        "django__django-12304",
+        "django__django-12708",
+        "django__django-12713",
+        "django__django-12774",
+        "django__django-9296",
+        "sphinx-doc__sphinx-10466",
+        "sphinx-doc__sphinx-7757",
+        "sphinx-doc__sphinx-7985",
+        "sphinx-doc__sphinx-8035",
+        "sphinx-doc__sphinx-8265",
+        "sphinx-doc__sphinx-8269",
+        "sphinx-doc__sphinx-8475",
+        "sphinx-doc__sphinx-8551",
+        "sphinx-doc__sphinx-8721",
+        "sphinx-doc__sphinx-9281",
+        "sphinx-doc__sphinx-9320",
+        "sphinx-doc__sphinx-9367",
+        "sphinx-doc__sphinx-9698"
+    ],
+    "unresolved_ids": [
+        "django__django-11790",
+        "django__django-11885",
+        "django__django-11964",
+        "django__django-12193",
+        "django__django-12273",
+        "django__django-12308",
+        "django__django-12325",
+        "django__django-12406",
+        "sphinx-doc__sphinx-10435",
+        "sphinx-doc__sphinx-10673",
+        "sphinx-doc__sphinx-11510",
+        "sphinx-doc__sphinx-7590",
+        "sphinx-doc__sphinx-7748",
+        "sphinx-doc__sphinx-8056",
+        "sphinx-doc__sphinx-8548",
+        "sphinx-doc__sphinx-8638",
+        "sphinx-doc__sphinx-9230",
+        "sphinx-doc__sphinx-9461"
+    ],
+    "error_ids": [],
+    "schema_version": 2
+}

benchmarks/swebench-agent-evals.md ADDED Viewed

	@@ -0,0 +1,75 @@

+# SWE-bench Agent Evaluations
+Date: 2026-06-29
+Harness: mini-SWE-agent plus the SWE-bench evaluation harness from
+`/home/ripper/dsv4-local/eval/swebench`.
+## Full SWE-bench Verified 100-task Slice
+This is the faithful 100-task run requested for reasoning mode. The run used
+`SWE-bench/SWE-bench_Verified`, split `test`, slice `0:100`.
+| Profile | Reasoning | Temperature | Resolved | Score |
+|---|---|---:|---:|---:|
+| Q6_K GGUF | on | 1.0 | 69/100 | 69.0% |
+Score chart:
+| Profile | Resolved chart |
+|---|---|
+| Q6_K reasoning-on | `##################################################################### 69.0%` |
+Notes:
+- `enable_thinking=true`, `preserve_thinking=true`.
+- No explicit thinking budget was enforced.
+- `top_p=1.0`, `max_output_tokens=8192`.
+- 94/100 tasks submitted patches.
+- 6/100 ended as empty patch or timeout.
+- Rerunning the two original infrastructure-error cases produced 0 remaining
+  infra errors.
+- The original harness wrapper row can look like `69/500 = 13.80%` because the
+  first invocation did not pass the intended instance filters. The corrected
+  intended score for the slice is `69/100 = 69.0%`.
+Raw summary:
+[`benchmarks/raw/swebench/ornith-q6-think-verified100-summary.json`](raw/swebench/ornith-q6-think-verified100-summary.json).
+## SWE-bench Verified Mini Quant Sweep
+This sweep used `MariusHobbhahn/swe-bench-verified-mini`, split `test`, with the
+same mini-SWE-agent setup across quants. These are 50-task verified-mini scores,
+not the full 100-task slice above.
+| Quant/profile | Resolved | Score | Completed | Empty patch | Errors |
+|---|---:|---:|---:|---:|---:|
+| Q6_K | 33/50 | 66.0% | 48/50 | 2 | 0 |
+| Q3_K_M | 32/50 | 64.0% | 47/50 | 3 | 0 |
+| Q5_K_M | 32/50 | 64.0% | 49/50 | 1 | 0 |
+| Q4_K_M | 31/50 | 62.0% | 48/50 | 2 | 0 |
+| IQ4_XS | 30/50 | 60.0% | 45/50 | 5 | 0 |
+| IQ4_XS-MTP-graft-headQ6 | 30/50 | 60.0% | 48/50 | 2 | 0 |
+| Q8_0 | 29/50 | 58.0% | 47/50 | 3 | 0 |
+Resolved chart:
+| Quant/profile | Resolved chart |
+|---|---|
+| Q6_K | `################################################################## 66.0%` |
+| Q3_K_M | `################################################################ 64.0%` |
+| Q5_K_M | `################################################################ 64.0%` |
+| Q4_K_M | `############################################################## 62.0%` |
+| IQ4_XS | `############################################################ 60.0%` |
+| IQ4_XS-MTP-graft-headQ6 | `############################################################ 60.0%` |
+| Q8_0 | `########################################################## 58.0%` |
+Raw reports:
+- [`ornith_q6_k_verified_mini.json`](raw/swebench/ornith_q6_k_verified_mini.json)
+- [`ornith_q3_k_m_verified_mini.json`](raw/swebench/ornith_q3_k_m_verified_mini.json)
+- [`ornith_q5_k_m_verified_mini.json`](raw/swebench/ornith_q5_k_m_verified_mini.json)
+- [`ornith_q4_k_m_verified_mini.json`](raw/swebench/ornith_q4_k_m_verified_mini.json)
+- [`ornith_iq4_xs_verified_mini.json`](raw/swebench/ornith_iq4_xs_verified_mini.json)
+- [`ornith_iq4_xs_mtp_graft_verified_mini.json`](raw/swebench/ornith_iq4_xs_mtp_graft_verified_mini.json)
+- [`ornith_q8_0_verified_mini.json`](raw/swebench/ornith_q8_0_verified_mini.json)