🧠✨ TransformerLM (AR 2048, 4K vocab) — SimpleStories
This is the result of the code from https://github.com/triloy8/transformerlm, a minimal autoregressive Transformer LM trained on SimpleStories with a 2048-token context and an 4K vocab tokenizer. ✨
✅ Key Facts
- Model type: Autoregressive Transformer LM
- Dataset: SimpleStories
- Context length: 2048 tokens
- Tokenizer vocab size: 4,096
- Layers: 12
- Heads: 8
- d_model: 512
- d_ff: 1344
- Training setup: Single NVIDIA A40 48GB
- Runtime: ~24 hours ⏱️
📦 What’s Inside
- 60k steps from a 60k run, including:
- Optimizer state
- RNG state
- Safetensors weights
- Tokenizer config
- Run config
🚀 Reproducibility
To reproduce the run:
Exact commit that launched the train: https://github.com/triloy8/transformerlm/commit/7ae6b9ba829a3ad37d85d2d8d201c717a423fb73