Fixed-width GDN d256, 20k steps
This repository archives the canonical fixed-width Gated DeltaNet (GDN) language model with Q/K head dimension 256 and V head dimension 64. It is a PyTorch Lightning training checkpoint, not a directly loadable Transformers AutoModel package. Reconstruct the model with the bundled Hydra experiment config and the referenced HazyResearch/mlp-mixer code.
Checkpoint
- Training run ID:
m05d15y26-gdn-paper-d256-fp32res-gpt2init-20k-r1 - Slurm job:
53024698 - Optimizer step:
20000 - Expected repository filename:
last.ckpt - Size:
2,084,104,035bytes - SHA-256:
f42ba43f80a26aa991d5441fbfa3cab1113e5648df50f286c3d473d1dd016ed1 - PyTorch Lightning checkpoint version:
1.8.6
The archived source file was named step_20000.ckpt; upload uses last.ckpt without changing its bytes.
Architecture and training
The model has 12 layers, hidden size 768, 12 Q/K/V heads, fixed Q/K head dimension 256, V head dimension 64 (expand_v=0.25), sequence length 2048, vocabulary size 50,257, and approximately 173.65M parameters. It uses an fp32 residual stream and GPT-2-style residual-output initialization.
Training used the Pile, global batch size 256, AdamW (lr=5e-4, betas 0.9/0.95, weight decay 0.1), 200 warmup steps, and cosine decay to 5e-5 through 20,000 optimizer steps. The exact experiment YAML is in config/experiment.yaml.
Metrics
| Evaluation | Perplexity | Qualification |
|---|---|---|
| Final training-run validation log | 9.980 |
Rounded display in the step-20k training log; it is not the common-slice evaluator. |
| Common L2048 Pile evaluation | 9.87802532382 |
Independent shared 10% validation-slice protocol: 18,720 sequences / 38,338,560 tokens. |
These values use different evaluation pipelines and should not be treated as repeated estimates from one identical sample.
Code and loading
- Reconstruction code/config reference:
HazyResearch/mlp-mixer@d2b24d685bc12f6e974d3ce68ffe2563a477cea1 - Exact experiment path in that tree:
train/configs/experiment/mlp_mixer/gdn_paper_hdim256_fp32res_gpt2init_20k.yaml - The precise training-time source commit was not recorded in the checkpoint. The bundled config, checkpoint hyperparameters, tensor shapes, and content hash are the archival source of truth.
Load with torch.load("last.ckpt", map_location="cpu", mmap=True, weights_only=False) in a trusted environment with the matching Lightning/OmegaConf stack, then restore through the project Lightning module. The file contains optimizer and scheduler state as well as model weights.