# Project Status Audit

Generated: 2026-06-28

This audit maps the original Ornith 35B project goals to current evidence. It
is intentionally conservative: a requirement is complete only when current files,
published artifacts, and verification outputs prove it.

## Verdict

The serving, quantization, MTP, publication, and reusable investigation-harness
tracks have usable verified outputs. The full project is not complete because
the fine-tuning/post-training track was stopped at user request and has no
release-candidate adapter.

## Requirement Status

| Requirement | Status | Evidence | Notes |
|---|---|---|---|
| Single-GPU serving only, `tp=1` | Complete | `hf_upload/ARTIFACT_MANIFEST.json`, `hf_upload/RELEASE_AUDIT.md`, `hf_upload/README.md` | Multi-GPU serving is explicitly out of scope. Both GPUs are used only as independent experiment lanes. |
| Fast serving profile | Complete for GGUF | `configs/serving_profiles.yaml`, `runs/serving-profile-validation.json`, `benchmarks/llamacpp-quant-benchmarks.md`, `scripts/serve_llamacpp_gpu0.sh`, `scripts/serve_llamacpp_gpu1.sh` | Recommended default is llama.cpp `Q4_K_M` coding context with `REASONING=off`. The serving catalog validates 5 single-GPU profiles with 43/43 checks passing. |
| Integrated MTP serving profile | Complete with adaptive policy | `benchmarks/llamacpp-iq4-xs-mtp-graft-adaptive-profile.md`, `patches/llamacpp-mtp-adaptive-serving-and-recurrent-rollback.patch` | Low concurrency benefits from MTP; saturated c16 should use adaptive throttling via `LLAMA_SPEC_MAX_DRAFTING_SLOTS=1`. Always-on MTP is not recommended for high-concurrency batches. |
| Find/fix model or serving issues | Complete for discovered issues | `probes/investigation-ornith-q4-live.json`, `benchmarks/llamacpp-q4-reasoning-off-fix.md`, llama.cpp patches under `patches/` | Found reasoning-content empty-final issue, vLLM GGUF corruption, upstream config-only MTP gap, and llama.cpp MTP recurrent/scheduler issues. Documented/fixed where locally controllable. |
| Reusable bug-discovery harness | Complete | `MODEL_INVESTIGATION_HARNESS.md`, `configs/model_behavior_suite.yaml`, `scripts/run_model_investigation.py`, `scripts/probe_model_behavior.py`, `scripts/summarize_behavior_matrix.py`, `runs/behavior-regression-matrix.json` | Published to HF with dependencies and gate scripts. Metadata-only smoke passed with 13 checks, 0 failures, 2 low-severity notes. The behavior matrix compares 7 artifacts across 14 reusable checks and separates 5 passing artifacts from 2 known-bad artifacts. |
| Quantize smaller while preserving capability | Complete for GGUF set | `ARTIFACT_MANIFEST.json`, `benchmarks/kld-quant-vs-bf16-top64.md`, `benchmarks/llamacpp-quant-benchmarks.md` | Published Q3_K_M, IQ4_XS, Q4_K_M, Q5_K_M, Q6_K, Q8_0, plus MTP draft variants. KLD, behavior, throughput, and footprint are documented. |
| Make quants usable on HF | Complete | HF repo `LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1`, `runs/hf-release-verification-final.json`, `RELEASE_AUDIT.md` | Current HF verifier readback passed against `04a18958f05ac002c2ff846d7b5ced6361f9af30`: 14 GGUF files present, 207 staged support files hash-matched, and 3 generated release files hash-matched. |
| Fine-tune on coding data and post-train | Incomplete by user stop | `runs/ornith-35b-coding-lora-prod-20k.stopped.json`, `hf_upload/probes/sft-prod-20k-stopped.json`, `benchmarks/sft-prod-20k-preflight.md` | Production LoRA run stopped at user request. Latest durable checkpoint is `checkpoint-8000`; training log reached step 8673 but no `checkpoint-9000` exists. |
| Diagnose SFT gate failure | Complete | `runs/sft-format-audit-prod.json`, `SFT_REMEDIATION_PLAN.md`, `scripts/audit_sft_format.py` | Format audit found 100% code-fenced answers, about 56% `Here's` answers, 0 short exact-answer rows, and empty rendered think blocks in every rendered row. |
| Prepare CPU-side SFT repair path | Complete | `configs/sft_repair_blend.yaml`, `configs/sft_format_gates.yaml`, `scripts/build_sft_repair_set.py`, `scripts/build_sft_code_only_from_fences.py`, `scripts/build_sft_blend.py`, `scripts/check_sft_format_gates.py`, `scripts/preflight_sft_resume.py`, `runs/sft-data-validation-prod-repair-strong.json`, `runs/sft-format-audit-prod-repair-strong.json`, `runs/sft-format-gate-code-only-extract.json`, `runs/sft-format-gate-prod-repair-strong.json`, `runs/sft-resume-preflight-strong.json` | Generated strict-output and code-only repair components. Recommended strong blend has 44,096 train / 2,312 eval rendered rows, no blocking validation issues, 54.43% train code-fence rate, and 30.49% `Here's` rate. Gates pass 8/8 and 10/10 checks. Resume preflight passes 16/16 checks. SFT remains stopped. |
| Release fine-tuned adapter | Not achieved | `runs/eval-lora-behavior-checkpoint-8000-fixed.json`, `runs/eval-lora-behavior-checkpoint-8000-fixed.gate.log` | Checkpoint-8000 loaded cleanly but failed 4 behavior checks, including high-severity strict-output failures. No adapter should be published from this run. |
| HF model cards/docs updated | Complete | Remote commit `04a18958f05ac002c2ff846d7b5ced6361f9af30` before this status audit refresh; `runs/hf-release-verification-final.json` | README, release audit, artifact manifest, benchmark docs, probes, scripts, configs, harness, and release verifier were uploaded and read back. This status audit creates a later commit when published. |
| Reusable HF release verifier | Complete | `scripts/verify_hf_release.py`, `runs/hf-release-verification-final.json` | Verifies model-card metadata, required support files, staged file hashes, generated release-file hashes, all published GGUF artifacts, and SFT gate summaries. Latest local run passed 21/21 checks with 0 warnings. |
| Reusable behavior regression matrix | Complete | `scripts/summarize_behavior_matrix.py`, `runs/behavior-regression-matrix.json`, `runs/behavior-regression-matrix.md` | CPU-only aggregation over existing behavior/investigation reports. Current run: 7 artifacts, 98 checks, 5 passing artifacts, 2 failing artifacts, 9 failed checks, and 6 blocking issues in known-bad profiles. |
| Reusable release-readiness gate | Complete | `scripts/build_release_readiness_report.py`, `runs/release-readiness-report.json`, `runs/release-readiness-report.md` | CPU-only gate over HF readback, manifest, behavior matrix, quant/KLD docs, MTP policy, required docs, and SFT state. Current run: serving release `pass`, full project `incomplete`, 22 checks, 0 failures, 5 warnings. |
| Machine-readable serving profile catalog | Complete | `configs/serving_profiles.yaml`, `scripts/validate_serving_profiles.py`, `runs/serving-profile-validation.json`, `runs/serving-profile-validation.md` | Catalog captures Q4_K_M speed/default, Q3_K_M lowest-memory, IQ4_XS middle-footprint, and adaptive IQ4_XS MTP graft profiles. Validation passes 43/43 checks against manifest, scripts, evidence docs, and tp=1 policy. |
| Keep GPUs idle after SFT stop | Complete at audit time | `nvidia-smi`: GPU0 10 MiB / 0%, GPU1 2 MiB / 0%; process scan empty | No `train_sft_lora`, watcher, eval, server, quantize, or convert processes were running. |

## Current Release State

- HF repo: `LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1`
- Last verified remote commit before this status-audit refresh: `04a18958f05ac002c2ff846d7b5ced6361f9af30`
- Latest local HF verifier report: `runs/hf-release-verification-final.json` (`21/21` checks passed, `0` warnings)
- Published HF verifier entrypoint: `scripts/verify_hf_release.py`
- Serving profile catalog: `configs/serving_profiles.yaml`
- Latest serving profile validation: `runs/serving-profile-validation.json` and `runs/serving-profile-validation.md`
- Latest behavior regression matrix: `runs/behavior-regression-matrix.json` and `runs/behavior-regression-matrix.md`
- Latest release-readiness report: `runs/release-readiness-report.json` and `runs/release-readiness-report.md`
- Release audit: `RELEASE_AUDIT.md`
- Machine-readable manifest: `ARTIFACT_MANIFEST.json`
- SFT remediation plan: `SFT_REMEDIATION_PLAN.md`
- Recommended speed serving profile: llama.cpp `Q4_K_M`, `REASONING=off`
- Recommended integrated MTP profile: `ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf` with `LLAMA_SPEC_MAX_DRAFTING_SLOTS=1`
- Lowest-memory validated quant: `ornith-1.0-35b-Q3_K_M.gguf`
- Middle-footprint validated quant: `ornith-1.0-35b-IQ4_XS.gguf`

## Remaining Work

1. Fine-tuning/post-training needs a new or resumed training plan if a release
   adapter is still desired. The stopped run should resume only from
   `checkpoint-8000`, and later checkpoints must pass `configs/model_behavior_suite.yaml`.
   Before spending GPU time, use the prepared repair blend in
   `configs/sft_repair_blend.yaml` and `SFT_REMEDIATION_PLAN.md`.
2. Always-on MTP should not be promoted for saturated batches unless a future
   runtime change beats matched AR throughput at c16.
3. vLLM GGUF should remain unsupported for this model family until its corrupted
   output behavior is fixed and revalidated.

## Completion Decision

Do not mark the full original goal complete. The serving, quantization, MTP,
publication, and reusable harness deliverables are complete enough to use, but
the fine-tuning/post-training deliverable is intentionally incomplete after the
requested SFT stop.