# Ornith-1.0-35B Q3_K_M GGUF Profile

Date: 2026-06-26

Hardware: single RTX PRO 6000 Blackwell Max-Q 96GB, `tp=1`.

Backend: local `llama.cpp` CUDA server from `vendor/llama.cpp/build-cuda/bin/llama-server`, launched through `scripts/serve_llamacpp_gpu1.sh`.

Artifact:

- Path: `artifacts/quant/ornith-1.0-35b-Q3_K_M.gguf`
- Size: 16,764,763,712 bytes / 15.61 GiB
- Created from the upstream BF16 GGUF with `llama-quantize Q3_K_M`
- Quantizer output: 3.87 BPW, 15,977.65 MiB quantized model size

## Server Profile

```bash
HOST=127.0.0.1 PORT=18188 \
MODEL_PATH=/home/ripper/ornith-35b-lab/artifacts/quant/ornith-1.0-35b-Q3_K_M.gguf \
ALIAS=Ornith-1.0-35B-GGUF-Q3_K_M \
CTX_SIZE=8192 PARALLEL=16 BATCH_SIZE=4096 UBATCH_SIZE=512 \
CONT_BATCHING=1 METRICS=1 \
scripts/serve_llamacpp_gpu1.sh
```

The patched default `REASONING=off` was used. This is the usable coding-serving
profile; the earlier reasoning-on/auto path can return empty final `content` for
short requests.

Loaded VRAM after model load: 17,684 MiB / 17.27 GiB.

Important: with llama.cpp server, `-c` is split across `-np`, so this benchmark
profile exposes `n_ctx = 512` per slot (`8192 / 16`). For about 8k tokens per
slot at 16-way concurrency, use `CTX_SIZE=131072 PARALLEL=16`.

## Validation

- GGUF materialization audit: `runs/gguf-materialization-q3-k-m.json`
  - 12 checks
  - 0 issues
  - 0 missing layer groups
  - 0 unexpected layer groups
- Behavior suite on the exact 16-slot server profile:
  `runs/behavior-llamacpp-q3-k-m-parallel16.json`
  - 4 cases
  - 14 checks
  - 0 failed checks
  - 0 issues

## Throughput

Values are aggregate output tokens/second from
`scripts/bench_openai.py --stream --max-tokens 256`; each row had 0 failed
requests.

| Concurrency | Requests | Output tok/s | p95 TTFT ms | p95 latency s |
|---:|---:|---:|---:|---:|
| 1 | 16 | 240.5 | 77.9 | 1.04 |
| 4 | 16 | 422.0 | 170.7 | 2.39 |
| 8 | 16 | 464.4 | 344.9 | 3.75 |
| 16 | 32 | 493.0 | 493.7 | 6.96 |

## Recommendation

Q3_K_M is usable as the smallest validated GGUF. It reduces disk footprint by
about 20.8% and loaded VRAM by about 19.0% versus Q4_K_M in the same short-context
benchmark profile.

Q4_K_M remains the speed recommendation: Q3_K_M is roughly tied at c1, then trails
Q4_K_M by about 7.9% at c4 and about 24.5% at c8/c16. Use Q3_K_M when memory
footprint matters more than peak throughput.

## Raw Artifacts

- `runs/llamacpp-q3-k-m-c1-256.jsonl`
- `runs/llamacpp-q3-k-m-c4-256.jsonl`
- `runs/llamacpp-q3-k-m-c8-256.jsonl`
- `runs/llamacpp-q3-k-m-c16-256.jsonl`
- `runs/behavior-llamacpp-q3-k-m-parallel16.json`
- `runs/gguf-materialization-q3-k-m.json`