# Training & reproduction

nano-case was trained in the [voidlab](https://github.com/vukrosic) research
harness — a small one-config LLM pipeline. Everything needed to reproduce the
numbers is here; the data is generated by code, so there is nothing to download.

## Environment

- Python 3.12, PyTorch 2.12 (CUDA), 1× NVIDIA RTX 5070 Ti (16 GB).
- Trains in a few minutes; the model is ~1M params and barely loads the GPU.

## Config (exactly as trained)

```yaml
model:   {vocab_size: 256, dim: 128, n_layers: 4, n_heads: 4, n_kv_heads: 2,
          head_dim: 32, ffn_mult: 4, max_seq_len: 64, attention: gqa,
          norm: rmsnorm, positional: rope, ffn: swiglu, rope_theta: 10000}
data:    {tokenizer: byte, dataset: cases, params: {n_train: 100000, n_val: 4000}}
optim:   {optimizer: adamw, schedule: cosine, lr: 3.0e-3, weight_decay: 0.1,
          betas: [0.9, 0.95], warmup_steps: 200}
train:   {trainer: sft, steps: 12000, batch_size: 64, seq_len: 64, seed: 0}
```

- **Objective:** SFT, prompt-masked cross-entropy — only the target identifier and
  the newline EOS are supervised; the `<case> | <messy input> => ` prompt is masked.
- **Seeds:** the headline number is reported as mean ± std over seeds 0/1/2.

## Reproduce the benchmark from the shipped weights

```bash
pip install -r requirements.txt
python eval_nano_case.py --n 4000      # model vs regex, overall + smushed slice
pytest test_nano_case.py -q            # labels / determinism / weights regression
python modeling_nano_case.py           # a few live conversions
```

## Honest notes

- The model's segmentation prior **is** the ~120-token training vocabulary. It
  nails smushing of *known* words; it does not generalise to out-of-vocabulary
  tokens or to chains far longer than seen in training (see the README's
  Limitations / OOD section). That is the expected ceiling of a 1M model on a
  vocabulary task, and it is reported, not hidden.