# Model Investigation Harness

This document is the reusable entry point for applying the Ornith 35B audit,
serving, quantization, MTP, and SFT gates to another model.

## Current Ornith Snapshot

- Scope: single-GPU serving only, `tp=1`; use the two GPUs as independent lanes,
  not as one serving replica.
- Published HF package:
  `LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1`.
- Fast GGUF serving default: `Q4_K_M` through llama.cpp with `REASONING=off`.
- Lowest-memory validated GGUF: local `Q3_K_M`, 15.61 GiB on disk, 17.27 GiB
  loaded VRAM in the short-context benchmark profile.
- Middle-footprint validated GGUF: local `IQ4_XS`, 17.64 GiB on disk, 19.34 GiB
  loaded VRAM in the KLD profile.
- Integrated MTP artifact:
  `artifacts/quant/ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf`.
- Recommended MTP serving policy:
  `LLAMA_SPEC_MAX_DRAFTING_SLOTS=1`, `SPEC_DRAFT_N_MAX=2`, adaptive low
  concurrency only. This keeps the c1 MTP win and avoids the saturated c16 loss.
- SFT status: production LoRA run stopped by user request. Latest durable
  checkpoint is `checkpoint-8000`; no SFT adapter is released.

## Core Audit Flow

Run metadata/tokenizer/config checks first. This does not load full weights.

```bash
cd /home/ripper/ornith-35b-lab
.venv-lab/bin/python scripts/run_model_investigation.py \
  --model deepreinforce-ai/Ornith-1.0-35B \
  --gguf deepreinforce-ai/Ornith-1.0-35B-GGUF \
  --output runs/investigation-ornith-metadata.json

.venv-lab/bin/python scripts/summarize_investigation.py \
  runs/investigation-ornith-metadata.json \
  --output runs/investigation-ornith-metadata.md

.venv-lab/bin/python scripts/check_audit_gates.py \
  runs/investigation-ornith-metadata.json
```

When an OpenAI-compatible backend is running, add the behavior suite to the
same combined report:

```bash
.venv-lab/bin/python scripts/run_model_investigation.py \
  --model deepreinforce-ai/Ornith-1.0-35B \
  --gguf deepreinforce-ai/Ornith-1.0-35B-GGUF \
  --base-url http://127.0.0.1:8002/v1 \
  --served-model Ornith-1.0-35B-GGUF-Q4_K_M \
  --output runs/investigation-ornith-live.json
```

Behavior checks live in `configs/model_behavior_suite.yaml`. They cover coding
correctness, JSON formatting, chat-template leakage, decode corruption, and
optional tool-call behavior.

Aggregate behavior and investigation reports into a reusable regression matrix:

```bash
.venv-lab/bin/python scripts/summarize_behavior_matrix.py \
  --input q4_live_investigation=runs/investigation-ornith-q4-live.json \
  --input q4_reasoning_off=runs/behavior-llamacpp-q4-reasoning-off.json \
  --input q4_script_default=runs/behavior-llamacpp-q4-script-default.json \
  --input q4_reasoning_on_bad=runs/behavior-llamacpp-q4.json \
  --input q3_k_m=runs/behavior-llamacpp-q3-k-m-parallel16.json \
  --input iq4_xs=runs/behavior-llamacpp-iq4-xs-parallel16.json \
  --input lora_checkpoint_8000_bad=runs/eval-lora-behavior-checkpoint-8000-fixed.json \
  --output runs/behavior-regression-matrix.json \
  --markdown-output runs/behavior-regression-matrix.md
```

The matrix treats high and critical issues as blocking by default and compares
each artifact across the same case/check keys. The current Ornith run has 5
passing artifacts and 2 intentionally failing artifacts: the old reasoning-on
Q4 profile and the stopped checkpoint-8000 LoRA adapter. Use this matrix after
each backend, quant, MTP, or adapter change to catch regressions without
manually diffing individual probe files.

Build a release-readiness report from the HF verifier, manifest, behavior
matrix, quant/KLD docs, MTP profile, and SFT state:

```bash
.venv-lab/bin/python scripts/build_release_readiness_report.py \
  --allow-unreleased-sft \
  --required-doc hf_upload/README.md \
  --required-doc hf_upload/RELEASE_AUDIT.md \
  --required-doc hf_upload/PROJECT_STATUS_AUDIT.md \
  --required-doc hf_upload/MODEL_INVESTIGATION_HARNESS.md \
  --required-doc hf_upload/SFT_REMEDIATION_PLAN.md \
  --output runs/release-readiness-report.json \
  --markdown-output runs/release-readiness-report.md \
  --fail-on-serving-not-ready
```

For Ornith this reports `serving_release_status=pass` and
`full_project_status=incomplete`. That distinction is intentional: the GGUF
serving release is usable, while the fine-tuning/post-training deliverable
remains unreleased after the stopped SFT run.

Validate the machine-readable serving profile catalog:

```bash
.venv-lab/bin/python scripts/validate_serving_profiles.py \
  --catalog configs/serving_profiles.yaml \
  --manifest hf_upload/ARTIFACT_MANIFEST.json \
  --hf-root hf_upload \
  --output runs/serving-profile-validation.json \
  --markdown-output runs/serving-profile-validation.md \
  --fail-on-error
```

The current catalog validates 5 single-GPU llama.cpp profiles with 43/43 checks
passing: Q4_K_M short-context speed, Q4_K_M coding-context default, Q3_K_M
lowest memory, IQ4_XS middle footprint, and adaptive IQ4_XS MTP graft.

## Serving And Benchmark Flow

Start a single-GPU llama.cpp server:

```bash
QUANT=Q4_K_M PORT=8002 CTX_SIZE=8192 PARALLEL=16 \
  scripts/serve_llamacpp_gpu0.sh
```

Run a throughput sweep:

```bash
for c in 1 4 8 16; do
  .venv-lab/bin/python scripts/bench_openai.py \
    --base-url http://127.0.0.1:8002/v1 \
    --model Ornith-1.0-35B-GGUF-Q4_K_M \
    --concurrency "$c" \
    --max-tokens 256 \
    --stream \
    --output "runs/llamacpp-q4-k-m-c${c}-256.jsonl"
done

.venv-lab/bin/python scripts/summarize_benchmarks.py \
  runs/llamacpp-q4-k-m-c*-256.jsonl
```

After every backend/profile change, run the OpenAI-compatible contract probe:

```bash
.venv-lab/bin/python scripts/probe_openai_contract.py \
  --base-url http://127.0.0.1:8002/v1 \
  --model Ornith-1.0-35B-GGUF-Q4_K_M \
  --output runs/server-contract.json
```

## Quantization Gates

For GGUF artifacts, verify materialization and serving behavior before treating
a quant as usable:

```bash
.venv-lab/bin/python scripts/audit_gguf_vllm_materialization.py \
  --model deepreinforce-ai/Ornith-1.0-35B \
  --gguf deepreinforce-ai/Ornith-1.0-35B-GGUF:Q4_K_M \
  --output runs/gguf-materialization-q4.json

.venv-lab/bin/python scripts/probe_model_behavior.py \
  --base-url http://127.0.0.1:8002/v1 \
  --model Ornith-1.0-35B-GGUF-Q4_K_M \
  --output runs/behavior-llamacpp-q4.json
```

For quant drift, compare next-token top-K distributions against BF16:

```bash
.venv-lab/bin/python scripts/collect_next_token_logprobs.py \
  --base-url http://127.0.0.1:8002 \
  --served-model Ornith-1.0-35B-GGUF-Q4_K_M \
  --messages-jsonl data/eval/coding_sft_eval_messages.jsonl \
  --output runs/kld-native-q4-k-m-top64-32prompts.json

.venv-lab/bin/python scripts/compare_kld_logprobs.py \
  --baseline runs/kld-native-bf16-top64-32prompts.json \
  --candidate runs/kld-native-q4-k-m-top64-32prompts.json \
  --output runs/kld-native-q4-k-m-vs-bf16.json
```

Current Ornith KLD summary is in `runs/kld-quant-vs-bf16-top64.md` and the HF
copy is `hf_upload/benchmarks/kld-quant-vs-bf16-top64.md`.

## MTP Gates

First determine whether the source checkpoint truly contains MTP tensors:

```bash
HF_SNAPSHOT=artifacts/source/ornith-1.0-35b-index
.venv-lab/bin/huggingface-cli download deepreinforce-ai/Ornith-1.0-35B \
  config.json model.safetensors.index.json \
  --local-dir "$HF_SNAPSHOT"

.venv-lab/bin/python scripts/check_mtp_support.py \
  --hf-snapshot "$HF_SNAPSHOT" \
  --gguf artifacts/quant/ornith-1.0-35b-IQ4_XS.gguf \
  --output runs/mtp-support-model.json
```

For integrated graft MTP on Ornith, the current recommended serving profile is:

```bash
LLAMA_SPEC_MAX_DRAFTING_SLOTS=1 \
LLAMA_MTP_FAST_BACKEND_SAMPLE=1 \
LLAMA_MTP_DRAFT_TOP_K=1 LLAMA_MTP_DRAFT_TOP_P=1 LLAMA_MTP_DRAFT_TEMP=1 \
MODEL_PATH=artifacts/quant/ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf \
PORT=8002 CTX_SIZE=8192 PARALLEL=16 \
SPEC_TYPE=draft-mtp SPEC_DRAFT_N_MAX=2 SPEC_DRAFT_N_MIN=0 \
SPEC_DRAFT_BACKEND_SAMPLING=1 \
  scripts/serve_llamacpp_gpu0.sh
```

MTP must be judged on both acceptance and throughput. For Ornith, always-on MTP
has high acceptance but loses at saturated c16; adaptive throttling is the
usable profile. Current evidence is in:

- `hf_upload/benchmarks/llamacpp-iq4-xs-mtp-graft-adaptive-profile.md`
- `hf_upload/patches/llamacpp-mtp-adaptive-serving-and-recurrent-rollback.patch`
- `hf_upload/patches/llamacpp-qwen35moe-mtp-recurrent-row-index.patch`

## SFT Gates

SFT data must pass validation before loading the 35B model:

```bash
.venv-lab/bin/python scripts/audit_sft_format.py \
  --input data/train/coding_sft_prod_messages.jsonl \
  --input data/eval/coding_sft_prod_eval_messages.jsonl \
  --input data/train/coding_sft_prod.jsonl \
  --input data/eval/coding_sft_prod_eval.jsonl \
  --output runs/sft-format-audit-prod.json
```

For the stopped Ornith SFT run, the CPU-side repair path is:

```bash
.venv-lab/bin/python scripts/build_sft_repair_set.py \
  --train-size 4096 \
  --eval-size 512 \
  --seed 29 \
  --train-output data/train/coding_sft_repair_messages.jsonl \
  --eval-output data/eval/coding_sft_repair_eval_messages.jsonl \
  --report-output runs/sft-repair-set.json

.venv-lab/bin/python scripts/build_sft_code_only_from_fences.py \
  --train-size 20000 \
  --eval-size 1000 \
  --seed 41 \
  --train-output data/train/coding_sft_code_only_messages.jsonl \
  --eval-output data/eval/coding_sft_code_only_eval_messages.jsonl \
  --report-output runs/sft-code-only-extract.json

.venv-lab/bin/python scripts/build_sft_blend.py \
  --source data/train/coding_sft_prod_messages.jsonl=40000:base_coding \
  --source data/train/coding_sft_repair_messages.jsonl=4096:strict_repair \
  --output data/train/coding_sft_prod_repair_messages.jsonl \
  --report-output runs/sft-repair-blend-train.json \
  --seed 31
```

See `configs/sft_repair_blend.yaml` for the full train/eval render, validation,
audit, and format-gate command sequence.

```bash
.venv-lab/bin/python scripts/check_sft_format_gates.py \
  --audit-report runs/sft-format-audit-prod-repair.json \
  --config configs/sft_format_gates.yaml \
  --profile strict_repair_messages \
  --output runs/sft-format-gate-strict-repair.json

.venv-lab/bin/python scripts/check_sft_format_gates.py \
  --audit-report runs/sft-format-audit-prod-repair.json \
  --config configs/sft_format_gates.yaml \
  --profile prod_repair_messages \
  --output runs/sft-format-gate-prod-repair.json

.venv-lab/bin/python scripts/check_sft_format_gates.py \
  --audit-report runs/sft-format-audit-prod-repair-strong.json \
  --config configs/sft_format_gates.yaml \
  --profile prod_repair_strong_messages \
  --output runs/sft-format-gate-prod-repair-strong.json

.venv-lab/bin/python scripts/preflight_sft_resume.py \
  --output runs/sft-resume-preflight-strong.json
```

```bash
.venv-train/bin/python scripts/validate_sft_jsonl.py \
  --train-jsonl data/train/coding_sft_prod.jsonl \
  --eval-jsonl data/eval/coding_sft_prod_eval.jsonl \
  --output runs/sft-data-validation-bigcode-prod-44k.json
```

Each LoRA checkpoint needs a behavior gate before release:

```bash
CUDA_VISIBLE_DEVICES=0 \
.venv-train/bin/python scripts/eval_lora_behavior.py \
  --base-model deepreinforce-ai/Ornith-1.0-35B \
  --adapter artifacts/train/ornith-35b-coding-lora-prod-20k/checkpoint-8000 \
  --suite configs/model_behavior_suite.yaml \
  --output runs/eval-lora-behavior-checkpoint-8000-fixed.json
```

Current stopped-run evidence:

- `runs/ornith-35b-coding-lora-prod-20k.stopped.json`
- `artifacts/train/ornith-35b-coding-lora-prod-20k/checkpoint-8000`
- `runs/sft-format-audit-prod.json`
- `runs/sft-data-validation-prod-repair.json`
- `runs/sft-format-audit-prod-repair.json`
- `runs/sft-format-gate-strict-repair.json`
- `runs/sft-format-gate-prod-repair.json`
- `runs/sft-code-only-extract.json`
- `runs/sft-data-validation-prod-repair-strong.json`
- `runs/sft-format-audit-prod-repair-strong.json`
- `runs/sft-format-gate-code-only-extract.json`
- `runs/sft-format-gate-prod-repair-strong.json`
- `runs/sft-resume-preflight-strong.json`
- `runs/eval-lora-behavior-checkpoint-8000-fixed.json`
- `runs/eval-lora-behavior-checkpoint-8000-fixed.gate.log`

## Publication Checklist

Before publishing a quant or serving profile:

- Confirm no multi-GPU serving assumptions slipped in.
- Record exact artifact path, quantization command, file size, and loaded VRAM.
- Run OpenAI contract checks and model behavior suite.
- Run throughput at c1/c4/c8/c16 on the intended server shape.
- For quants, record BF16 KLD/top-1 agreement.
- For MTP, record acceptance by position and matched AR throughput.
- Update `hf_upload/README.md` plus the relevant `hf_upload/benchmarks/*.md`.
- Upload changed docs and artifacts to the HF repo, then read back the remote
  files with `huggingface_hub` before calling the publication done.