Training & reproduction
nano-case was trained in the voidlab research harness — a small one-config LLM pipeline. Everything needed to reproduce the numbers is here; the data is generated by code, so there is nothing to download.
Environment
- Python 3.12, PyTorch 2.12 (CUDA), 1× NVIDIA RTX 5070 Ti (16 GB).
- Trains in a few minutes; the model is ~1M params and barely loads the GPU.
Config (exactly as trained)
model: {vocab_size: 256, dim: 128, n_layers: 4, n_heads: 4, n_kv_heads: 2,
head_dim: 32, ffn_mult: 4, max_seq_len: 64, attention: gqa,
norm: rmsnorm, positional: rope, ffn: swiglu, rope_theta: 10000}
data: {tokenizer: byte, dataset: cases, params: {n_train: 100000, n_val: 4000}}
optim: {optimizer: adamw, schedule: cosine, lr: 3.0e-3, weight_decay: 0.1,
betas: [0.9, 0.95], warmup_steps: 200}
train: {trainer: sft, steps: 12000, batch_size: 64, seq_len: 64, seed: 0}
- Objective: SFT, prompt-masked cross-entropy — only the target identifier and
the newline EOS are supervised; the
<case> | <messy input> =>prompt is masked. - Seeds: the headline number is reported as mean ± std over seeds 0/1/2.
Reproduce the benchmark from the shipped weights
pip install -r requirements.txt
python eval_nano_case.py --n 4000 # model vs regex, overall + smushed slice
pytest test_nano_case.py -q # labels / determinism / weights regression
python modeling_nano_case.py # a few live conversions
Honest notes
- The model's segmentation prior is the ~120-token training vocabulary. It nails smushing of known words; it does not generalise to out-of-vocabulary tokens or to chains far longer than seen in training (see the README's Limitations / OOD section). That is the expected ceiling of a 1M model on a vocabulary task, and it is reported, not hidden.