benchmarks/sft-lora-smoke-bigcode.md

Ornith-1.0-35B Coding SFT LoRA Smoke

Date: 2026-06-26

Hardware: single RTX PRO 6000 Blackwell Max-Q 96GB on GPU1.

Base model: deepreinforce-ai/Ornith-1.0-35B.

This was a one-step LoRA smoke run to prove the local SFT stack, dataset rendering, validation gates, tokenizer path, TRL configuration, and adapter save path. It is not a capability claim for a completed fine-tuned model, and no capability-bearing adapter is released from this run.

Dataset

Source preset: bigcode_self_oss

Source dataset: bigcode/self-oss-instruct-sc2-exec-filter-50k

License metadata: odc-by

Selected rows:

Split	Rows	Mean tokens	P50 tokens	P95 tokens	Max tokens
Train	2,048	354.51	328	618	1,561
Eval	256	341.34	313	548	1,089

Validation gate: passed JSONL parse, rendered-template checks, license metadata checks, no obvious secret patterns, no exact train duplicates, no exact eval duplicates, and no exact normalized train/eval overlap. The only warnings were 4 train rows and 1 eval row above the 1,024-token smoke-run max length.

Training Command

CUDA_VISIBLE_DEVICES=1 TOKENIZERS_PARALLELISM=false \
.venv-train/bin/python scripts/train_sft_lora.py \
  --model deepreinforce-ai/Ornith-1.0-35B \
  --train-jsonl data/train/coding_sft.jsonl \
  --eval-jsonl data/eval/coding_sft_eval.jsonl \
  --output artifacts/train/ornith-35b-coding-lora-smoke \
  --max-length 1024 \
  --per-device-train-batch-size 1 \
  --gradient-accumulation-steps 1 \
  --learning-rate 2e-4 \
  --num-train-epochs 1 \
  --max-steps 1 \
  --save-steps 1 \
  --eval-steps 9999 \
  --logging-steps 1 \
  --save-total-limit 1 \
  --lora-r 8 \
  --lora-alpha 16 \
  --lora-dropout 0.05 \
  --loss-type nll \
  --router-aux-loss-coef 0.0 \
  --require-license \
  --min-train-rows 2048 \
  --validation-report runs/sft-train-validation-bigcode-pilot.json

Result

Metric	Value
Global steps	1
Train loss	1.1419
Grad norm	3.2746
Learning rate	0.0002
Train entropy	0.5914
Train mean token accuracy	0.7574
Eval loss	1.1048
Eval entropy	0.6682
Eval mean token accuracy	0.7442
Eval runtime seconds	19.8063
Eval samples/second	12.925

Saved adapter path: artifacts/train/ornith-35b-coding-lora-smoke.

Adapter size: 16,761,232 bytes for adapter_model.safetensors.

Issues Found And Fixed

• Passed an explicit AutoTokenizer as processing_class to SFTTrainer; this avoids TRL falling through to an AutoProcessor/image-processor path for the base model.
• Added explicit --loss-type nll and --router-aux-loss-coef 0.0; this avoids TRL's chunked MoE loss path expecting num_experts directly on the top-level config when Ornith exposes it under text_config.
• Added license-required data validation before training.

Next Training Step

Run a real adapter experiment with the validated BigCode pilot or a larger mixed coding corpus, sized as a production-scale SFT/post-training run. A release candidate should be on the order of 20k+ optimizer steps or an equivalent token budget, then compared against the base model on coding evals before publishing an adapter as an improved coding checkpoint.