# Ornith-1.0-35B Coding SFT LoRA Smoke

Date: 2026-06-26

Hardware: single RTX PRO 6000 Blackwell Max-Q 96GB on GPU1.

Base model: `deepreinforce-ai/Ornith-1.0-35B`.

This was a one-step LoRA smoke run to prove the local SFT stack, dataset
rendering, validation gates, tokenizer path, TRL configuration, and adapter save
path. It is not a capability claim for a completed fine-tuned model, and no
capability-bearing adapter is released from this run.

## Dataset

Source preset: `bigcode_self_oss`

Source dataset: `bigcode/self-oss-instruct-sc2-exec-filter-50k`

License metadata: `odc-by`

Selected rows:

| Split | Rows | Mean tokens | P50 tokens | P95 tokens | Max tokens |
|---|---:|---:|---:|---:|---:|
| Train | 2,048 | 354.51 | 328 | 618 | 1,561 |
| Eval | 256 | 341.34 | 313 | 548 | 1,089 |

Validation gate: passed JSONL parse, rendered-template checks, license metadata
checks, no obvious secret patterns, no exact train duplicates, no exact eval
duplicates, and no exact normalized train/eval overlap. The only warnings were
4 train rows and 1 eval row above the 1,024-token smoke-run max length.

## Training Command

```bash
CUDA_VISIBLE_DEVICES=1 TOKENIZERS_PARALLELISM=false \
.venv-train/bin/python scripts/train_sft_lora.py \
  --model deepreinforce-ai/Ornith-1.0-35B \
  --train-jsonl data/train/coding_sft.jsonl \
  --eval-jsonl data/eval/coding_sft_eval.jsonl \
  --output artifacts/train/ornith-35b-coding-lora-smoke \
  --max-length 1024 \
  --per-device-train-batch-size 1 \
  --gradient-accumulation-steps 1 \
  --learning-rate 2e-4 \
  --num-train-epochs 1 \
  --max-steps 1 \
  --save-steps 1 \
  --eval-steps 9999 \
  --logging-steps 1 \
  --save-total-limit 1 \
  --lora-r 8 \
  --lora-alpha 16 \
  --lora-dropout 0.05 \
  --loss-type nll \
  --router-aux-loss-coef 0.0 \
  --require-license \
  --min-train-rows 2048 \
  --validation-report runs/sft-train-validation-bigcode-pilot.json
```

## Result

| Metric | Value |
|---|---:|
| Global steps | 1 |
| Train loss | 1.1419 |
| Grad norm | 3.2746 |
| Learning rate | 0.0002 |
| Train entropy | 0.5914 |
| Train mean token accuracy | 0.7574 |
| Eval loss | 1.1048 |
| Eval entropy | 0.6682 |
| Eval mean token accuracy | 0.7442 |
| Eval runtime seconds | 19.8063 |
| Eval samples/second | 12.925 |

Saved adapter path: `artifacts/train/ornith-35b-coding-lora-smoke`.

Adapter size: 16,761,232 bytes for `adapter_model.safetensors`.

## Issues Found And Fixed

- Passed an explicit `AutoTokenizer` as `processing_class` to `SFTTrainer`; this
  avoids TRL falling through to an AutoProcessor/image-processor path for the
  base model.
- Added explicit `--loss-type nll` and `--router-aux-loss-coef 0.0`; this avoids
  TRL's chunked MoE loss path expecting `num_experts` directly on the top-level
  config when Ornith exposes it under `text_config`.
- Added license-required data validation before training.

## Next Training Step

Run a real adapter experiment with the validated BigCode pilot or a larger
mixed coding corpus, sized as a production-scale SFT/post-training run. A
release candidate should be on the order of 20k+ optimizer steps or an
equivalent token budget, then compared against the base model on coding evals
before publishing an adapter as an improved coding checkpoint.