---
language: en
license: openrail
tags:
  - text-generation
  - causal-lm
  - gpt
  - small-language-model
  - tinystories
  - pytorch
pipeline_tag: text-generation
---

<p align="center">
  <img src="https://huggingface.co/ApyHTML19/ABADES-SLM-15M/resolve/main/abades-banner-animated.gif"
       alt="ABADES-SLM-15M — a 15M-parameter GPT trained on TinyStories"
       width="100%">
</p>

# ABADES-SLM-15M

A small GPT-style language model trained from scratch on the [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) dataset. Designed as a lightweight, educational SLM (Small Language Model).

---

## Model Details

| Property | Value |
|---|---|
| **Architecture** | GPT (decoder-only transformer) |
| **Parameters** | ~15M |
| **Vocab size** | 50,257 (GPT-2 tokenizer) |
| **Context length** | 128 tokens |
| **Layers** | 6 |
| **Attention heads** | 6 |
| **Embedding dim** | 384 |
| **Training dataset** | TinyStories |
| **Training iterations** | 45,000 |
| **License** | OpenRAIL |

---

## Usage

```python
import torch
import tiktoken
from model import GPT, GPTConfig  # your model file

# Load tokenizer
enc = tiktoken.get_encoding("gpt2")

# Load model
config = GPTConfig(
    vocab_size=50257,
    block_size=128,
    n_layer=6,
    n_head=6,
    n_embd=384,
    dropout=0.0,
    bias=True
)

model = GPT(config)
checkpoint = torch.load("ABADES-SLM-15M.pt", map_location="cpu")
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()

# Generate text
sentence = "Once upon a time there was a little girl"
context = torch.tensor(enc.encode_ordinary(sentence)).unsqueeze(0)

with torch.no_grad():
    output = model.generate(context, max_new_tokens=200, temperature=0.8, top_k=40)

print(enc.decode(output.squeeze().tolist()))
```

---

## Training Details

- **Tokenizer:** GPT-2 BPE (`tiktoken`)
- **Optimizer:** AdamW (`lr=1e-4`, `betas=(0.9, 0.95)`, `weight_decay=0.1`)
- **LR Schedule:** Linear warmup (1000 steps) → Cosine decay
- **Mixed precision:** bfloat16 / float16
- **Gradient accumulation:** 32 steps
- **Gradient clipping:** 0.5

---

## Example Outputs

**Prompt:** `"Once upon a time there was a pumpkin."`

> Once upon a time there was a pumpkin. It was big and orange and lived in a garden...

**Prompt:** `"A little girl went to the woods"`

> A little girl went to the woods with her dog. They were looking for something fun to do...

---

## Limitations

- Trained only on simple children's stories (TinyStories)
- Context window limited to 128 tokens
- Not suitable for complex reasoning or factual tasks
- May generate repetitive or incoherent text on out-of-domain prompts

---

## Author

**ApyHTML19** — built as a learning project to understand transformer training from scratch.

Inspired by [nanoGPT](https://github.com/karpathy/nanoGPT) by Andrej Karpathy.