# SFT Preflight And Stopped-Run Status Date: 2026-06-26, updated 2026-06-27 ## Current State The production SFT data path was prepared and validated for a real 20k-step LoRA run on GPU1. The run was launched, evaluated at `checkpoint-8000`, and then stopped at user request on 2026-06-27. No adapter from this run has been released. Validated source: - Dataset: `bigcode/self-oss-instruct-sc2-exec-filter-50k` - License metadata: `odc-by` - Preset: `bigcode_self_oss` - Seed: `23` - Train rows: 44,000 - Eval rows: 2,000 Files: - Messages train: `data/train/coding_sft_prod_messages.jsonl` - Messages eval: `data/eval/coding_sft_prod_eval_messages.jsonl` - Rendered train: `data/train/coding_sft_prod.jsonl` - Rendered eval: `data/eval/coding_sft_prod_eval.jsonl` - Source prep report: `runs/coding-sft-prod-source-prep.json` - Validation report: `runs/sft-data-validation-bigcode-prod-44k.json` ## Validation Production split validation passed with no blocking issues: - JSONL parse: pass - rendered Ornith chat template markers: pass - nonempty rows: pass - obvious secret-pattern scan: pass - exact normalized duplicates: pass - license metadata present: pass - exact train/eval overlap: pass - minimum train rows: pass, 44,000 / 44,000 Warnings: - 114 train rows exceed 1,024 tokens. - 8 eval rows exceed 1,024 tokens. Token lengths: | Split | Rows | Mean | P50 | P95 | Max | |---|---:|---:|---:|---:|---:| | Train | 44,000 | 350.21 | 324 | 612 | 1,863 | | Eval | 2,000 | 350.55 | 324 | 620 | 1,493 | ## Production Run Launch: ```bash scripts/launch_sft_prod_20k_gpu1.sh ``` Monitor: ```bash scripts/monitor_sft_prod.sh ``` Default run parameters: - Output: `artifacts/train/ornith-35b-coding-lora-prod-20k` - Log: `runs/ornith-35b-coding-lora-prod-20k.log` - PID file: `runs/ornith-35b-coding-lora-prod-20k.pid` - Max steps: 20,000 - Micro batch: 1 - Gradient accumulation: 8 - Effective examples per optimizer step: 8 - Approximate examples seen: 160,000 - Approximate passes over 44k rows: 3.64 - LoRA rank/alpha/dropout: 16 / 32 / 0.05 - LR schedule: `1e-4`, cosine, warmup ratio `0.03` - Save/eval cadence: every 1,000 steps The script writes the exact command to `runs/ornith-35b-coding-lora-prod-20k.command.txt` and supports resume via: ```bash RESUME_FROM_CHECKPOINT=artifacts/train/ornith-35b-coding-lora-prod-20k/checkpoint- \ scripts/launch_sft_prod_20k_gpu1.sh ``` ## Checkpoint Behavior Gates Run local adapter behavior gates on GPU0 for any resumed or future checkpoint: ```bash CUDA_VISIBLE_DEVICES=0 \ .venv-train/bin/python scripts/eval_lora_behavior.py \ --base-model deepreinforce-ai/Ornith-1.0-35B \ --adapter artifacts/train/ornith-35b-coding-lora-prod-20k/checkpoint- \ --suite configs/model_behavior_suite.yaml \ --output runs/eval-lora-behavior-checkpoint-.json ``` Expected gate: - The command must exit `0` for high/critical behavior checks. - `summary.failed_checks` should be `0`. - If failures appear, inspect generated outputs before allowing the 20k run to become a release candidate. The eval harness now includes two PEFT/Transformers compatibility fixes: - `WeightConverter` accepts PEFT's newer `distributed_operation` and `quantization_operation` kwargs on this Transformers build. - Ornith LoRA checkpoints saved with `model.language_model.*` keys are loaded with an explicit PEFT `key_mapping` to the inference-time `model.*` module names. The `checkpoint-8000` rerun had no missing adapter-key warning. Automatic watcher for a future resumed run: ```bash CHECKPOINT_STEP= GPU_INDEX=0 \ scripts/watch_sft_checkpoint_gate.sh ``` Watcher outputs: - PID file: `runs/watch-sft-checkpoint-.pid` - Watch log: `runs/watch-sft-checkpoint-.log` - Eval output: `runs/eval-lora-behavior-checkpoint-.json` - Eval log: `runs/eval-lora-behavior-checkpoint-.log` - Gate log: `runs/eval-lora-behavior-checkpoint-.gate.log` - GPU gate: waits for GPU0 memory usage to be below 1,024 MiB before loading the base model and LoRA adapter. ## Stopped Run And Checkpoint 8000 Result Status on 2026-06-27: the 20k-step run was stopped at user request and both local GPUs were idled. No SFT adapter has been released. Durable state: - Latest durable checkpoint: `artifacts/train/ornith-35b-coding-lora-prod-20k/checkpoint-8000` - Training log reached: step `8673` - `checkpoint-9000`: not present - Recoverability: steps after 8000 were not saved as model state - Stop record: `probes/sft-prod-20k-stopped.json` Training metrics from `checkpoint-8000`: | Metric | Value | |---|---:| | Global step | 8,000 / 20,000 | | Epoch | 1.4545 | | Train loss | 0.48498 | | Train mean token accuracy | 0.84811 | | Train entropy | 0.47972 | | Eval loss | 0.47880 | | Eval mean token accuracy | 0.85095 | | Eval entropy | 0.47114 | | Tokens seen | 22,452,631 | Behavior gate command: ```bash CUDA_VISIBLE_DEVICES=0 \ .venv-train/bin/python scripts/eval_lora_behavior.py \ --base-model deepreinforce-ai/Ornith-1.0-35B \ --adapter artifacts/train/ornith-35b-coding-lora-prod-20k/checkpoint-8000 \ --suite configs/model_behavior_suite.yaml \ --output runs/eval-lora-behavior-checkpoint-8000-fixed.json ``` Gate result: 4 cases, 14 checks, 4 failed checks, 4 issues. Blocking failures: - `short_python_add`: `finish_reason='length'`, so the short coding prompt did not finish inside the gate budget. - `template_marker_leak`: the exact one-word response check failed; output started with `Here's a thinking process:`. Medium failures: - `json_instruction_following`: output did not parse as JSON. - `json_instruction_following`: output contained a markdown fence. Interpretation: `checkpoint-8000` is a valid loaded adapter checkpoint, but it is not a release candidate. It appears to overproduce reasoning/template text on strict formatting prompts, so later checkpoints need the same gate before any adapter publication decision.