GPT-2 124M — edu-FineWeb 10B
A 124M-parameter GPT-2 trained from scratch on 10B tokens of the edu-FineWeb sample.
- Architecture: GPT-2 (12 layers, 12 heads, 768 dim, 1024 context)
- Vocab: 50304 (GPT-2 BPE, padded)
- Training: 19,072 steps, ~0.5M tokens/batch (10B tokens total)
- Final val loss: 3.10
Files
ckpt_019072.pt— checkpoint (model,optimizer,scheduler,step,vocab_size)inference.py— self-contained model definition + generation script
Usage
huggingface-cli download Xerneas3318/gpt2-124m-edu-fineweb10b ckpt_019072.pt inference.py --local-dir .
python3 inference.py \
--ckpt ckpt_019072.pt \
--prompt "Once upon a time" \
--num-samples 4 --max-new-tokens 100 --temperature 0.9 --top-k 50
The script primes on the real prompt tokens and grows the context autoregressively,
with temperature + top-k sampling. Runs on GPU (CUDA) by default, CPU with --device cpu.