# Ornith-1.0-35B IQ4_XS GGUF Profile

Date: 2026-06-27

Hardware: single RTX PRO 6000 Blackwell Max-Q 96GB, `tp=1`.

Backend: local `llama.cpp` CUDA server from `vendor/llama.cpp/build-cuda/bin/llama-server`, launched through `scripts/serve_llamacpp_gpu0.sh`.

Artifact:

- Path: `artifacts/quant/ornith-1.0-35b-IQ4_XS.gguf`
- Size: 18,939,311,680 bytes / 17.64 GiB
- Created from the upstream BF16 GGUF with `llama-quantize IQ4_XS`
- Command: `llama-quantize ornith-1.0-35b-bf16.gguf ornith-1.0-35b-IQ4_XS.gguf IQ4_XS 32`
- Quantizer output: source 66,152.24 MiB / 16.01 BPW; quantized 18,051.46 MiB / 4.37 BPW
- Importance matrix: none

## MTP Status

MTP is not loadable for this released Ornith checkpoint. The HF config advertises
`mtp_num_hidden_layers=1`, but the current source repo and GGUF repo contain no
MTP/NextN/draft-like files, no `mtp.*` or appended MTP tensors in the safetensors
index, and no `nextn` metadata or MTP tensors in the BF16/Q4/IQ4_XS GGUF files.

Evidence:

- Source repo commit: `5df2ed3f675c7beaa490328cc70bb573b65fb660`
- GGUF repo commit: `c2e1703039380de4ce6820e97afd185682d3c16c`
- `runs/mtp-support-ornith-iq4-xs.json`: `config_only_not_loadable`
- `convert_hf_to_gguf.py --mtp --dry-run` produced only embeddings, output head,
  and output norm, not the required Qwen3.5 MoE MTP block tensors

So this artifact is a validated IQ4_XS trunk quant, not an MTP-enabled draft pair.

## Server Profile

```bash
HOST=127.0.0.1 PORT=18192 \
MODEL_PATH=/home/ripper/ornith-35b-lab/artifacts/quant/ornith-1.0-35b-IQ4_XS.gguf \
ALIAS=Ornith-1.0-35B-GGUF-IQ4_XS \
CTX_SIZE=8192 PARALLEL=16 BATCH_SIZE=4096 UBATCH_SIZE=512 \
CONT_BATCHING=1 METRICS=1 REASONING=off \
scripts/serve_llamacpp_gpu0.sh
```

Loaded VRAM after model load and benchmark warmup: 19,806 MiB / 19.34 GiB.

Important: with llama.cpp server, `-c` is split across `-np`, so this benchmark
profile exposes `n_ctx = 512` per slot (`8192 / 16`). For about 8k tokens per
slot at 16-way concurrency, use `CTX_SIZE=131072 PARALLEL=16`.

## Validation

- Correctness probe: `runs/probe-llamacpp-iq4-xs-correctness.json`
  - 8 checks
  - 0 failed checks
- Behavior suite on the exact 16-slot server profile:
  `runs/behavior-llamacpp-iq4-xs-parallel16.json`
  - 4 cases
  - 14 checks
  - 0 failed checks
  - 0 issues

## Throughput

Values are aggregate output tokens/second from
`scripts/bench_openai.py --stream --max-tokens 256`; each row had 0 failed
requests.

| Concurrency | Requests | Output tok/s | p95 TTFT ms | p95 latency s |
|---:|---:|---:|---:|---:|
| 1 | 16 | 234.1 | 75.1 | 1.05 |
| 4 | 16 | 297.7 | 159.7 | 3.31 |
| 8 | 16 | 411.5 | 330.2 | 4.78 |
| 16 | 32 | 476.0 | 541.5 | 6.97 |

## BF16 KLD Probe

Corrected native llama.cpp next-token top-64 `KL(P_bf16 || P_IQ4_XS)` over the
same 32 coding prompts used for the other quant rows:

- Mean KLD: 0.1426 nats / 0.2057 bits
- P50 KLD: 0.0868 nats
- P95 KLD: 0.3195 nats
- Max KLD: 0.6586 nats
- Top-1 match: 27/32 (84.4%)

This places IQ4_XS between Q3_K_M and Q4_K_M for this fast next-token fidelity
signal: materially closer than Q3_K_M, but not as close as Q4_K_M.

## Recommendation

IQ4_XS is a usable middle-footprint quant. It is about 10.5% smaller on disk than
Q4_K_M and about 9.2% lower loaded VRAM in the short-context benchmark profile,
with better KLD than Q3_K_M. Q4_K_M remains the speed recommendation; Q3_K_M
remains the lowest-footprint recommendation.

## Raw Artifacts

- `runs/llamacpp-iq4-xs-c1-256.jsonl`
- `runs/llamacpp-iq4-xs-c4-256.jsonl`
- `runs/llamacpp-iq4-xs-c8-256.jsonl`
- `runs/llamacpp-iq4-xs-c16-256.jsonl`
- `runs/probe-llamacpp-iq4-xs-correctness.json`
- `runs/behavior-llamacpp-iq4-xs-parallel16.json`
- `runs/kld-native-iq4-xs-top64-32prompts.json`
- `runs/kld-native-iq4-xs-vs-bf16.json`
- `runs/mtp-support-ornith-iq4-xs.json`