--- license: mit license_link: https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B/blob/main/LICENSE base_model: deepreinforce-ai/Ornith-1.0-35B pipeline_tag: text-generation library_name: gguf tags: - gguf - llama-cpp - quantized - text-generation - coding - mtp - qwen3_5_moe - single-gpu - tp1 --- # Ornith 35B Lab This workspace tracks optimization work for `deepreinforce-ai/Ornith-1.0-35B`. ## Scope - Serving is single GPU only: `tp=1`, one model copy per GPU. - GPU0 is the serving speed lane. - GPU1 is an independent experiment lane for quantization, fine-tuning, or audit jobs. - Multi-GPU serving profiles are intentionally out of scope for this model version. ## Initial Tracks 1. Baseline and tune single-GPU serving with SGLang, vLLM, and llama.cpp GGUF. 2. Build a reusable model audit harness for tokenizer, chat template, special-token, reasoning, and tool-call issues. 3. Evaluate existing GGUF quantizations, then test smaller or higher-throughput formats. 4. Prepare a coding-data fine-tuning and post-training pipeline. Current reusable workflow: [MODEL_INVESTIGATION_HARNESS.md](MODEL_INVESTIGATION_HARNESS.md). Current requirement audit: [PROJECT_STATUS_AUDIT.md](PROJECT_STATUS_AUDIT.md). SFT remediation plan: [SFT_REMEDIATION_PLAN.md](SFT_REMEDIATION_PLAN.md). Serving profile catalog: [configs/serving_profiles.yaml](configs/serving_profiles.yaml) with validation report [probes/serving-profile-validation.md](probes/serving-profile-validation.md). Behavior regression matrix: [probes/behavior-regression-matrix.md](probes/behavior-regression-matrix.md) via [scripts/summarize_behavior_matrix.py](scripts/summarize_behavior_matrix.py). Release readiness report: [probes/release-readiness-report.md](probes/release-readiness-report.md) via [scripts/build_release_readiness_report.py](scripts/build_release_readiness_report.py). HF release verification: [probes/hf-release-verification.json](probes/hf-release-verification.json) via [scripts/verify_hf_release.py](scripts/verify_hf_release.py). Latest recorded run passed 25 checks with 0 failures and 1 expected SFT warning against HF commit `7dcf02d87d07e4b173d7b375e9fc747605d80652`. ## Current Serving Baseline - Best current GGUF serving profile: llama.cpp `Q4_K_M` `tp=1` with `REASONING=off`. - Integrated MTP serving profile: `IQ4_XS-MTP-graft-headQ6` with adaptive throttling, `LLAMA_SPEC_MAX_DRAFTING_SLOTS=1`; this keeps low-concurrency MTP speedups and avoids the saturated c16 loss. - Earlier vLLM BF16 `tp=1` and SGLang BF16 `tp=1` profiles are preserved as baseline notes, but the publishable model package is the llama.cpp GGUF stack. - The default llama.cpp script uses `REASONING=off` because the model otherwise spends simple coding prompts in `reasoning_content` before producing final `content`. - Custom Q3_K_M `tp=1` is now validated as the lowest-memory GGUF option: 15.61 GiB on disk, 17.27 GiB loaded VRAM in the short-context benchmark profile, and 0 behavior-suite issues. Q4_K_M remains faster. - Custom IQ4_XS `tp=1` is validated as the middle-footprint GGUF option and is the base body for the integrated MTP graft artifact. - vLLM GGUF Q4_K_M currently loads but has broken output and is not benchmark-valid. - Profile notes: `runs/vllm-initial-profile.md` and `runs/sglang-initial-profile.md`. - Quantized control note: `runs/llamacpp-q4-control.md`. - GGUF/vLLM materialization audit: `runs/gguf-vllm-materialization-q4.md`. - Reusable benchmark summary: ```bash .venv-lab/bin/python scripts/summarize_benchmarks.py runs/*-seq-c*-256.jsonl ``` ## GPU1 Experiment Lane - Lane config: `configs/experiment_lanes.yaml`. - Quantization matrix: `configs/quantization_matrix.yaml`. - Custom Q3_K_M profile: `runs/llamacpp-q3-k-m-profile.md`. - Training plan: `configs/training_plan.yaml`. - Source-backed run note: `runs/gpu1-quant-train-plan.md`. ## Hardware Baseline - GPU0: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 97,887 MiB VRAM. - GPU1: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 97,887 MiB VRAM. - Driver: 580.159.03, CUDA runtime reported by driver: 13.0. - Host had no `nvcc` and no `cmake` at project start. A local `.venv-build` now provides CMake, Ninja, and CUDA 13.2 wheels for the llama.cpp CUDA build. ## Quick Start Create the lightweight lab environment: ```bash cd /home/ripper/ornith-35b-lab uv venv .venv-lab --python 3.12 source .venv-lab/bin/activate uv pip install -r requirements-lab.txt ``` Run the metadata audit without downloading full model weights: ```bash python scripts/audit_model.py --output runs/audit-ornith-35b.json .venv-lab/bin/python scripts/check_audit_gates.py runs/audit-ornith-35b.json ``` Run the combined reusable investigation report: ```bash .venv-lab/bin/python scripts/run_model_investigation.py \ --output runs/investigation-ornith-metadata.json .venv-lab/bin/python scripts/summarize_investigation.py \ runs/investigation-ornith-metadata.json \ --output runs/investigation-ornith-metadata.md ``` Add `--base-url` and `--served-model` when a live OpenAI-compatible backend is running to include the behavior suite in the same report. Verify the published HF release and local staged support-file hashes: ```bash .venv-lab/bin/python scripts/verify_hf_release.py \ --output runs/hf-release-verification.json ``` When running from a clone of the HF repo rather than this lab workspace, pass `--local-dir . --manifest ARTIFACT_MANIFEST.json`. Build the CPU-only release-readiness report: ```bash .venv-lab/bin/python scripts/build_release_readiness_report.py \ --allow-unreleased-sft \ --serving-profile-validation hf_upload/probes/serving-profile-validation.json \ --required-doc hf_upload/README.md \ --required-doc hf_upload/RELEASE_AUDIT.md \ --required-doc hf_upload/PROJECT_STATUS_AUDIT.md \ --required-doc hf_upload/MODEL_INVESTIGATION_HARNESS.md \ --required-doc hf_upload/SFT_REMEDIATION_PLAN.md \ --output runs/release-readiness-report.json \ --markdown-output runs/release-readiness-report.md \ --fail-on-serving-not-ready ``` The current readiness report passes the GGUF serving release and marks the full project incomplete because no SFT/post-training adapter is released. Validate the machine-readable serving profiles: ```bash .venv-lab/bin/python scripts/validate_serving_profiles.py \ --catalog configs/serving_profiles.yaml \ --manifest hf_upload/ARTIFACT_MANIFEST.json \ --hf-root hf_upload \ --output runs/serving-profile-validation.json \ --markdown-output runs/serving-profile-validation.md \ --fail-on-error ``` Run reusable behavior probes against any OpenAI-compatible backend: ```bash .venv-lab/bin/python scripts/probe_model_behavior.py \ --base-url http://localhost:8000/v1 \ --model Ornith-1.0-35B \ --output runs/behavior-vllm-bf16.json ``` The suite lives in `configs/model_behavior_suite.yaml` and is designed to be copied to other models. It turns model bugs into named, repeatable checks for coding correctness, JSON following, chat-template leakage, decode corruption, and optional tool-call handling. Run the GGUF/vLLM materialization audit for a quantized artifact: ```bash .venv-lab/bin/python scripts/audit_gguf_vllm_materialization.py \ --model deepreinforce-ai/Ornith-1.0-35B \ --gguf deepreinforce-ai/Ornith-1.0-35B-GGUF:Q4_K_M \ --output runs/gguf-vllm-materialization-q4.json ``` Start one serving backend on GPU0 after installing that backend in its own environment: ```bash CUDA_VISIBLE_DEVICES=0 scripts/serve_vllm_gpu0.sh CUDA_VISIBLE_DEVICES=0 scripts/serve_sglang_gpu0.sh CUDA_VISIBLE_DEVICES=0 scripts/serve_llamacpp_gpu0.sh ``` Serve the custom local Q3_K_M GGUF: ```bash QUANT=Q3_K_M PORT=8002 CTX_SIZE=8192 PARALLEL=16 \ scripts/serve_llamacpp_gpu0.sh ``` Run a smoke benchmark against an OpenAI-compatible server: ```bash python scripts/bench_openai.py \ --base-url http://localhost:8000/v1 \ --model Ornith-1.0-35B \ --concurrency 1 \ --max-tokens 256 \ --output runs/smoke-bench.jsonl ``` Probe the OpenAI-compatible server contract after each backend/profile change: ```bash .venv-lab/bin/python scripts/probe_openai_contract.py \ --base-url http://localhost:8000/v1 \ --model Ornith-1.0-35B \ --output runs/server-contract.json ``` Prepare and gate SFT data before launching GPU training: ```bash .venv-train/bin/python scripts/render_sft_jsonl.py \ --input data/train/coding_sft_messages.jsonl \ --output data/train/coding_sft.jsonl .venv-train/bin/python scripts/validate_sft_jsonl.py \ --train-jsonl data/train/coding_sft.jsonl \ --eval-jsonl data/eval/coding_sft_eval.jsonl \ --output runs/sft-data-validation.json ``` `scripts/train_sft_lora.py` runs the same validation before importing Torch and loading the 35B model unless `--skip-data-validation` is passed. The production SFT run was stopped at user request. Latest durable checkpoint: `artifacts/train/ornith-35b-coding-lora-prod-20k/checkpoint-8000`; stop record: `runs/ornith-35b-coding-lora-prod-20k.stopped.json`. CPU-side SFT remediation is prepared but not running. The repair path is defined in `configs/sft_repair_blend.yaml` and uses: - `scripts/build_sft_repair_set.py` - `scripts/build_sft_code_only_from_fences.py` - `scripts/build_sft_blend.py` - `data/train/coding_sft_prod_repair_strong.jsonl` - `data/eval/coding_sft_prod_repair_strong_eval.jsonl` - `runs/sft-data-validation-prod-repair-strong.json` - `runs/sft-format-audit-prod-repair-strong.json` - `runs/sft-format-gate-code-only-extract.json` - `runs/sft-format-gate-prod-repair-strong.json` - `runs/sft-resume-preflight-strong.json` - `runs/sft-format-gate-strict-repair.json` - `runs/sft-format-gate-prod-repair.json` The recommended strong repair split has 44,096 train rows, 2,312 eval rows, and no blocking validation issues. It reduces train message code fences to 54.43% and `Here's`/`Here is` preambles to 30.49%. The code-only extraction gate passes 8/8 checks, and the strong blended repair gate passes 10/10 checks. Do not resume SFT unless explicitly requested. Before any future resume, run the report-only guard: ```bash .venv-lab/bin/python scripts/preflight_sft_resume.py \ --output runs/sft-resume-preflight-strong.json ``` The current preflight passes 16/16 checks and includes the exact resume command preview without launching training. Run the production SFT monitor only for a resumed run: ```bash scripts/monitor_sft_prod.sh ``` Start the automatic checkpoint behavior gate watcher only for a resumed run: ```bash scripts/watch_sft_checkpoint_gate.sh ``` Evaluate a LoRA checkpoint against the reusable behavior suite: ```bash CUDA_VISIBLE_DEVICES=0 \ .venv-train/bin/python scripts/eval_lora_behavior.py \ --base-model deepreinforce-ai/Ornith-1.0-35B \ --adapter artifacts/train/ornith-35b-coding-lora-prod-20k/checkpoint- \ --suite configs/model_behavior_suite.yaml \ --output runs/eval-lora-behavior-checkpoint-.json ``` The LoRA evaluator defaults to final-answer mode, matching llama.cpp `REASONING=off`. Add `--enable-thinking` only for explicit reasoning-mode tests.