---
license: apache-2.0
datasets:
- trixyL/simplestories-4k-megatron
language:
- en
pipeline_tag: text-generation
tags:
- simplestories
- '2048'
- 4k
- transformerlm
---
# 🧠✨ TransformerLM (AR 2048, 4K vocab) — SimpleStories

This is the result of the code from https://github.com/triloy8/transformerlm, a minimal autoregressive Transformer LM trained on **SimpleStories** with a **2048-token context** and an **4K vocab tokenizer**. ✨

## ✅ Key Facts

- **Model type:** Autoregressive Transformer LM
- **Dataset:** SimpleStories
- **Context length:** 2048 tokens
- **Tokenizer vocab size:** 4,096
- **Layers:** 12
- **Heads:** 8
- **d_model:** 512
- **d_ff:** 1344
- **Training setup:** Single **NVIDIA A40 48GB**
- **Runtime:** ~24 hours ⏱️

## 📦 What’s Inside

- 60k steps from a 60k run, including:
  - Optimizer state
  - RNG state
  - Safetensors weights
- Tokenizer config
- Run config

## 🚀 Reproducibility

To reproduce the run:

Exact commit that launched the train: https://github.com/triloy8/transformerlm/commit/7ae6b9ba829a3ad37d85d2d8d201c717a423fb73