--- license: mit license_link: https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B/blob/main/LICENSE base_model: deepreinforce-ai/Ornith-1.0-35B pipeline_tag: text-generation library_name: gguf tags: - gguf - llama-cpp - quantized - text-generation - coding - mtp - qwen3_5_moe - single-gpu - tp1 --- # Ornith 35B Lab This workspace tracks optimization work for `deepreinforce-ai/Ornith-1.0-35B`. ## Release Summary Full release writeup: [MODEL_RELEASE_REPORT.md](MODEL_RELEASE_REPORT.md). This release is a single-GPU llama.cpp GGUF package for Ornith-1.0-35B. The supported serving policy is `tp=1`, one model copy per GPU. Multi-GPU tensor parallel serving is intentionally out of scope for this version. | Use case | Recommended artifact | Key numbers | |---|---|---| | Default serving speed | `ornith-1.0-35b-Q4_K_M.gguf` | 19.71 GiB on disk, 21.31 GiB loaded VRAM, 243.3 tok/s c1, 655.6 tok/s c16 | | Lowest memory | `ornith-1.0-35b-Q3_K_M.gguf` | 15.61 GiB on disk, 17.27 GiB loaded VRAM, 240.5 tok/s c1, 493.0 tok/s c16 | | Middle footprint | `ornith-1.0-35b-IQ4_XS.gguf` | 17.64 GiB on disk, 19.34 GiB loaded VRAM, 0.1426 mean top-64 KLD nats | | Quality/footprint | `ornith-1.0-35b-Q6_K.gguf` | 26.56 GiB on disk, 28.03 GiB loaded VRAM, 0.0165 mean top-64 KLD nats, 32/32 top-1 match | | Native low-concurrency MTP | `ornith-1.0-35b-IQ4_XS-MTP-graft-headQ6.gguf` | 19.6 GB decimal, c1/128 adaptive MTP 319.53 tok/s, acceptance `(0.953, 0.865)` | Quant quality was measured against the upstream BF16 GGUF with a native llama.cpp next-token top-64 KLD probe over 32 coding prompts. MTP work includes a bootstrapped external draft, trained chain-corrected draft artifacts, a requantized integrated IQ4_XS-MTP graft, and patched llama.cpp runtime support for recurrent row-index handling, fast backend sampling, sequential verification, per-head contexts, and adaptive MTP throttling. The release MTP recommendation is adaptive: use MTP for low-concurrency/single-user requests and keep `LLAMA_SPEC_MAX_DRAFTING_SLOTS=1` so saturated batches fall back to target-only throughput. Dedicated MTP KLD/sequence evaluation is available at [benchmarks/mtp-dedicated-kld-evaluation.md](benchmarks/mtp-dedicated-kld-evaluation.md). The integrated IQ4_XS-MTP graft measured 0.073138 mean top-64 `KL(P_bf16 || P_candidate)` nats over 32 prompts. In an 8 prompt x 64 token single-user sequence probe, fast native `draft-mtp` reached 82.01% draft-token acceptance and 233.81 client tok/s vs 172.57 target-only, but matched only 478/512 generated token positions, so the active MTP path remains an experimental low-concurrency speed profile rather than a strict target-equivalent serving mode. ## SWE-bench Agent Scores Full report and raw score JSON: [benchmarks/swebench-agent-evals.md](benchmarks/swebench-agent-evals.md). Full SWE-bench Verified 100-task slice, reasoning mode enabled: | Profile | Dataset slice | Reasoning | Resolved | Score | |---|---|---|---:|---:| | Q6_K GGUF | `SWE-bench/SWE-bench_Verified` test `0:100` | on | 69/100 | 69.0% | Verified-mini quant sweep, 50 tasks: | Quant/profile | Resolved chart | Score | |---|---|---:| | Q6_K | `##################################################################` | 66.0% | | Q3_K_M | `################################################################` | 64.0% | | Q5_K_M | `################################################################` | 64.0% | | Q4_K_M | `##############################################################` | 62.0% | | IQ4_XS | `############################################################` | 60.0% | | IQ4_XS-MTP-graft-headQ6 | `############################################################` | 60.0% | | Q8_0 | `##########################################################` | 58.0% | ## Scope - Serving is single GPU only: `tp=1`, one model copy per GPU. - GPU0 is the serving speed lane. - GPU1 is an independent experiment lane for quantization, fine-tuning, or audit jobs. - Multi-GPU serving profiles are intentionally out of scope for this model version. ## Initial Tracks 1. Baseline and tune single-GPU serving with SGLang, vLLM, and llama.cpp GGUF. 2. Build a reusable model audit harness for tokenizer, chat template, special-token, reasoning, and tool-call issues. 3. Evaluate existing GGUF quantizations, then test smaller or higher-throughput formats. 4. Prepare a coding-data fine-tuning and post-training pipeline. Current reusable workflow: [MODEL_INVESTIGATION_HARNESS.md](MODEL_INVESTIGATION_HARNESS.md). Detailed quant/MTP/model-release report: [MODEL_RELEASE_REPORT.md](MODEL_RELEASE_REPORT.md). Pi-agent quant coding task report: [benchmarks/pi-agent-flappy-quant-test.md](benchmarks/pi-agent-flappy-quant-test.md) with raw evidence under [probes/pi-agent-flappy/](probes/pi-agent-flappy/). Dedicated MTP KLD/sequence evaluation: [benchmarks/mtp-dedicated-kld-evaluation.md](benchmarks/mtp-dedicated-kld-evaluation.md) with raw evidence under [benchmarks/raw/](benchmarks/raw/). SWE-bench agent evaluation: [benchmarks/swebench-agent-evals.md](benchmarks/swebench-agent-evals.md) with raw score reports under [benchmarks/raw/swebench/](benchmarks/raw/swebench/). Current requirement audit: [PROJECT_STATUS_AUDIT.md](PROJECT_STATUS_AUDIT.md). SFT remediation plan: [SFT_REMEDIATION_PLAN.md](SFT_REMEDIATION_PLAN.md). Serving profile catalog: [configs/serving_profiles.yaml](configs/serving_profiles.yaml) with validation report [probes/serving-profile-validation.md](probes/serving-profile-validation.md). Quant artifact catalog: [configs/quant_artifacts.yaml](configs/quant_artifacts.yaml) with validation report [probes/quant-artifact-validation.md](probes/quant-artifact-validation.md). MTP profile catalog: [configs/mtp_profiles.yaml](configs/mtp_profiles.yaml) with validation report [probes/mtp-profile-validation.md](probes/mtp-profile-validation.md). Project requirement catalog: [configs/project_requirements.yaml](configs/project_requirements.yaml) with validation report [probes/project-requirement-validation.md](probes/project-requirement-validation.md). Idle-state policy: [configs/idle_state_policy.yaml](configs/idle_state_policy.yaml) with validation report [probes/idle-state-validation.md](probes/idle-state-validation.md). CPU validation suite: [scripts/run_cpu_validation_suite.py](scripts/run_cpu_validation_suite.py) with report [probes/cpu-validation-suite.md](probes/cpu-validation-suite.md). Goal completion audit: [scripts/build_goal_completion_audit.py](scripts/build_goal_completion_audit.py) with report [probes/goal-completion-audit.md](probes/goal-completion-audit.md). Release-reference consistency: [scripts/validate_release_references.py](scripts/validate_release_references.py) with report [probes/release-reference-validation.md](probes/release-reference-validation.md). Behavior regression matrix: [probes/behavior-regression-matrix.md](probes/behavior-regression-matrix.md) via [scripts/summarize_behavior_matrix.py](scripts/summarize_behavior_matrix.py). Release readiness report: [probes/release-readiness-report.md](probes/release-readiness-report.md) via [scripts/build_release_readiness_report.py](scripts/build_release_readiness_report.py). HF release verification: [probes/hf-release-verification.json](probes/hf-release-verification.json) via [scripts/verify_hf_release.py](scripts/verify_hf_release.py). Latest local run passed 48 checks with 0 failures and 3 expected incomplete-project warnings; the exact verified commit is recorded in `remote.sha` in the JSON report. Publishing a verifier report can leave the staged published report one metadata commit behind the newest remote head; `scripts/validate_release_references.py` and `scripts/build_goal_completion_audit.py` record both published and latest local verifier evidence to prevent stale completion claims. When the same script runs from a fresh HF repo clone, `hf_upload/...` defaults fall back to repo-root paths; a missing local latest-verifier run is a warning, not a failure. ## Current Serving Baseline - Best current GGUF serving profile: llama.cpp `Q4_K_M` `tp=1` with `REASONING=off`. - Integrated MTP serving profile: `IQ4_XS-MTP-graft-headQ6` with adaptive throttling, `LLAMA_SPEC_MAX_DRAFTING_SLOTS=1`; this keeps low-concurrency MTP speedups and avoids the saturated c16 loss. The MTP catalog validates all 8 published MTP GGUF artifacts. - Pi-agent Flappy Bird quant test: all 7 tested profiles generated a runnable vanilla HTML/CSS/JS canvas game. `IQ4_XS-MTP-graft-headQ6` was fastest on this agent workload at 320.91 decode tok/s, with 88.92% MTP draft-token acceptance (`3452/3882`) and per-position acceptance `(0.936, 0.843)`. Q3_K_M passed only after 3 Pi-agent repair passes; all other profiles passed on the first run. - Earlier vLLM BF16 `tp=1` and SGLang BF16 `tp=1` profiles are preserved as baseline notes, but the publishable model package is the llama.cpp GGUF stack. - The default llama.cpp script uses `REASONING=off` because the model otherwise spends simple coding prompts in `reasoning_content` before producing final `content`. - Custom Q3_K_M `tp=1` is now validated as the lowest-memory GGUF option: 15.61 GiB on disk, 17.27 GiB loaded VRAM in the short-context benchmark profile, and 0 behavior-suite issues. Q4_K_M remains faster. - Custom IQ4_XS `tp=1` is validated as the middle-footprint GGUF option and is the base body for the integrated MTP graft artifact. - vLLM GGUF Q4_K_M currently loads but has broken output and is not benchmark-valid. - Profile notes: `runs/vllm-initial-profile.md` and `runs/sglang-initial-profile.md`. - Quantized control note: `runs/llamacpp-q4-control.md`. - GGUF/vLLM materialization audit: `runs/gguf-vllm-materialization-q4.md`. - Reusable benchmark summary: ```bash .venv-lab/bin/python scripts/summarize_benchmarks.py runs/*-seq-c*-256.jsonl ``` ## GPU1 Experiment Lane - Lane config: `configs/experiment_lanes.yaml`. - Quantization matrix: `configs/quantization_matrix.yaml`. - Custom Q3_K_M profile: `runs/llamacpp-q3-k-m-profile.md`. - Training plan: `configs/training_plan.yaml`. - Source-backed run note: `runs/gpu1-quant-train-plan.md`. ## Hardware Baseline - GPU0: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 97,887 MiB VRAM. - GPU1: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 97,887 MiB VRAM. - Driver: 580.159.03, CUDA runtime reported by driver: 13.0. - Host had no `nvcc` and no `cmake` at project start. A local `.venv-build` now provides CMake, Ninja, and CUDA 13.2 wheels for the llama.cpp CUDA build. ## Quick Start Create the lightweight lab environment: ```bash cd /home/ripper/ornith-35b-lab uv venv .venv-lab --python 3.12 source .venv-lab/bin/activate uv pip install -r requirements-lab.txt ``` Run the metadata audit without downloading full model weights: ```bash python scripts/audit_model.py --output runs/audit-ornith-35b.json .venv-lab/bin/python scripts/check_audit_gates.py runs/audit-ornith-35b.json ``` Run the combined reusable investigation report: ```bash .venv-lab/bin/python scripts/run_model_investigation.py \ --output runs/investigation-ornith-metadata.json .venv-lab/bin/python scripts/summarize_investigation.py \ runs/investigation-ornith-metadata.json \ --output runs/investigation-ornith-metadata.md ``` Add `--base-url` and `--served-model` when a live OpenAI-compatible backend is running to include the behavior suite in the same report. Verify the published HF release and local staged support-file hashes: ```bash .venv-lab/bin/python scripts/verify_hf_release.py \ --output runs/hf-release-verification.json ``` When running from a clone of the HF repo rather than this lab workspace, pass `--local-dir . --manifest ARTIFACT_MANIFEST.json`. Run the local CPU-only validation suite: ```bash .venv-lab/bin/python scripts/run_cpu_validation_suite.py \ --root . \ --output runs/cpu-validation-suite.json \ --markdown-output runs/cpu-validation-suite.md \ --fail-on-error ``` This runs idle-state, serving profile, quant artifact, MTP profile, release-readiness, release-reference, project-requirement, and goal-completion validation in dependency order. It does not start model serving, SFT, quantization, or eval workloads. Its summary includes both failed steps and report-level warnings so expected incomplete-project warnings stay visible without failing the suite. Build the CPU-only release-readiness report: ```bash .venv-lab/bin/python scripts/build_release_readiness_report.py \ --allow-unreleased-sft \ --serving-profile-validation hf_upload/probes/serving-profile-validation.json \ --quant-artifact-validation hf_upload/probes/quant-artifact-validation.json \ --mtp-profile-validation hf_upload/probes/mtp-profile-validation.json \ --required-doc hf_upload/README.md \ --required-doc hf_upload/RELEASE_AUDIT.md \ --required-doc hf_upload/PROJECT_STATUS_AUDIT.md \ --required-doc hf_upload/MODEL_INVESTIGATION_HARNESS.md \ --required-doc hf_upload/SFT_REMEDIATION_PLAN.md \ --output runs/release-readiness-report.json \ --markdown-output runs/release-readiness-report.md \ --fail-on-serving-not-ready ``` The current readiness report passes the GGUF serving release and marks the full project incomplete because no SFT/post-training adapter is released. Validate the machine-readable serving profiles: ```bash .venv-lab/bin/python scripts/validate_serving_profiles.py \ --catalog configs/serving_profiles.yaml \ --manifest hf_upload/ARTIFACT_MANIFEST.json \ --hf-root hf_upload \ --output runs/serving-profile-validation.json \ --markdown-output runs/serving-profile-validation.md \ --fail-on-error ``` Validate the machine-readable quant artifacts: ```bash .venv-lab/bin/python scripts/validate_quant_artifacts.py \ --catalog configs/quant_artifacts.yaml \ --manifest hf_upload/ARTIFACT_MANIFEST.json \ --hf-root hf_upload \ --output runs/quant-artifact-validation.json \ --markdown-output runs/quant-artifact-validation.md \ --fail-on-error ``` Validate the machine-readable MTP profiles: ```bash .venv-lab/bin/python scripts/validate_mtp_profiles.py \ --catalog configs/mtp_profiles.yaml \ --manifest hf_upload/ARTIFACT_MANIFEST.json \ --hf-root hf_upload \ --output runs/mtp-profile-validation.json \ --markdown-output runs/mtp-profile-validation.md \ --fail-on-error ``` Validate the project-level requirement status: ```bash .venv-lab/bin/python scripts/validate_project_requirements.py \ --catalog configs/project_requirements.yaml \ --root . \ --output runs/project-requirement-validation.json \ --markdown-output runs/project-requirement-validation.md \ --fail-on-error ``` This report intentionally separates `serving_release_status=pass` from `full_project_status=incomplete` until a release-candidate SFT/post-training adapter exists. Validate the stopped/idle runtime state: ```bash .venv-lab/bin/python scripts/validate_idle_state.py \ --config configs/idle_state_policy.yaml \ --root . \ --output runs/idle-state-validation.json \ --markdown-output runs/idle-state-validation.md \ --fail-on-error ``` This is a live local guard: it checks the saved SFT stop record, `nvidia-smi` GPU utilization and compute apps, and known training/server/quant process names. Run reusable behavior probes against any OpenAI-compatible backend: ```bash .venv-lab/bin/python scripts/probe_model_behavior.py \ --base-url http://localhost:8000/v1 \ --model Ornith-1.0-35B \ --output runs/behavior-vllm-bf16.json ``` The suite lives in `configs/model_behavior_suite.yaml` and is designed to be copied to other models. It turns model bugs into named, repeatable checks for coding correctness, JSON following, chat-template leakage, decode corruption, and optional tool-call handling. Run the GGUF/vLLM materialization audit for a quantized artifact: ```bash .venv-lab/bin/python scripts/audit_gguf_vllm_materialization.py \ --model deepreinforce-ai/Ornith-1.0-35B \ --gguf deepreinforce-ai/Ornith-1.0-35B-GGUF:Q4_K_M \ --output runs/gguf-vllm-materialization-q4.json ``` Start one serving backend on GPU0 after installing that backend in its own environment: ```bash CUDA_VISIBLE_DEVICES=0 scripts/serve_vllm_gpu0.sh CUDA_VISIBLE_DEVICES=0 scripts/serve_sglang_gpu0.sh CUDA_VISIBLE_DEVICES=0 scripts/serve_llamacpp_gpu0.sh ``` Serve the custom local Q3_K_M GGUF: ```bash QUANT=Q3_K_M PORT=8002 CTX_SIZE=8192 PARALLEL=16 \ scripts/serve_llamacpp_gpu0.sh ``` Run a smoke benchmark against an OpenAI-compatible server: ```bash python scripts/bench_openai.py \ --base-url http://localhost:8000/v1 \ --model Ornith-1.0-35B \ --concurrency 1 \ --max-tokens 256 \ --output runs/smoke-bench.jsonl ``` Probe the OpenAI-compatible server contract after each backend/profile change: ```bash .venv-lab/bin/python scripts/probe_openai_contract.py \ --base-url http://localhost:8000/v1 \ --model Ornith-1.0-35B \ --output runs/server-contract.json ``` Prepare and gate SFT data before launching GPU training: ```bash .venv-train/bin/python scripts/render_sft_jsonl.py \ --input data/train/coding_sft_messages.jsonl \ --output data/train/coding_sft.jsonl .venv-train/bin/python scripts/validate_sft_jsonl.py \ --train-jsonl data/train/coding_sft.jsonl \ --eval-jsonl data/eval/coding_sft_eval.jsonl \ --output runs/sft-data-validation.json ``` `scripts/train_sft_lora.py` runs the same validation before importing Torch and loading the 35B model unless `--skip-data-validation` is passed. The production SFT run was stopped at user request. Latest durable checkpoint: `artifacts/train/ornith-35b-coding-lora-prod-20k/checkpoint-8000`; stop record: `runs/ornith-35b-coding-lora-prod-20k.stopped.json`. CPU-side SFT remediation is prepared but not running. The repair path is defined in `configs/sft_repair_blend.yaml` and uses: - `scripts/build_sft_repair_set.py` - `scripts/build_sft_code_only_from_fences.py` - `scripts/build_sft_blend.py` - `data/train/coding_sft_prod_repair_strong.jsonl` - `data/eval/coding_sft_prod_repair_strong_eval.jsonl` - `runs/sft-data-validation-prod-repair-strong.json` - `runs/sft-format-audit-prod-repair-strong.json` - `runs/sft-format-gate-code-only-extract.json` - `runs/sft-format-gate-prod-repair-strong.json` - `runs/sft-resume-preflight-strong.json` - `runs/sft-format-gate-strict-repair.json` - `runs/sft-format-gate-prod-repair.json` The recommended strong repair split has 44,096 train rows, 2,312 eval rows, and no blocking validation issues. It reduces train message code fences to 54.43% and `Here's`/`Here is` preambles to 30.49%. The code-only extraction gate passes 8/8 checks, and the strong blended repair gate passes 10/10 checks. Do not resume SFT unless explicitly requested. Before any future resume, run the report-only guard: ```bash .venv-lab/bin/python scripts/preflight_sft_resume.py \ --output runs/sft-resume-preflight-strong.json ``` The current preflight passes 16/16 checks and includes the exact resume command preview without launching training. Run the production SFT monitor only for a resumed run: ```bash scripts/monitor_sft_prod.sh ``` Start the automatic checkpoint behavior gate watcher only for a resumed run: ```bash scripts/watch_sft_checkpoint_gate.sh ``` Evaluate a LoRA checkpoint against the reusable behavior suite: ```bash CUDA_VISIBLE_DEVICES=0 \ .venv-train/bin/python scripts/eval_lora_behavior.py \ --base-model deepreinforce-ai/Ornith-1.0-35B \ --adapter artifacts/train/ornith-35b-coding-lora-prod-20k/checkpoint- \ --suite configs/model_behavior_suite.yaml \ --output runs/eval-lora-behavior-checkpoint-.json ``` The LoRA evaluator defaults to final-answer mode, matching llama.cpp `REASONING=off`. Add `--enable-thinking` only for explicit reasoning-mode tests.