Hugging Face's logo

ApyHTML19
/
ABADES-SLM-15M

Text Generation
PyTorch
English
causal-lm
gpt
small-language-model
tinystories
Model card Files
xet
 Community
ABADES-SLM-15M
File size: 2,836 Bytes
e1646b2
 
 
 
 
 
 
 
 
 
 
 
 
eacb3ea
 
 
 
 
a6ba989
 
 
e1646b2
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
a6ba989
e1646b2
 
 
a6ba989
e1646b2
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
a6ba989
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
---
language: en
license: openrail
tags:
  - text-generation
  - causal-lm
  - gpt
  - small-language-model
  - tinystories
  - pytorch
pipeline_tag: text-generation
---

<p align="center">
  <img src="https://huggingface.co/ApyHTML19/ABADES-SLM-15M/resolve/main/abades-banner-animated.gif"
       alt="ABADES-SLM-15M — a 15M-parameter GPT trained on TinyStories"
       width="100%">
</p>

# ABADES-SLM-15M

A small GPT-style language model trained from scratch on the [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) dataset. Designed as a lightweight, educational SLM (Small Language Model).

---

## Model Details

| Property | Value |
|---|---|
| **Architecture** | GPT (decoder-only transformer) |
| **Parameters** | ~15M |
| **Vocab size** | 50,257 (GPT-2 tokenizer) |
| **Context length** | 128 tokens |
| **Layers** | 6 |
| **Attention heads** | 6 |
| **Embedding dim** | 384 |
| **Training dataset** | TinyStories |
| **Training iterations** | 45,000 |
| **License** | OpenRAIL |

---

## Usage

```python
import torch
import tiktoken
from model import GPT, GPTConfig  # your model file

# Load tokenizer
enc = tiktoken.get_encoding("gpt2")

# Load model
config = GPTConfig(
    vocab_size=50257,
    block_size=128,
    n_layer=6,
    n_head=6,
    n_embd=384,
    dropout=0.0,
    bias=True
)

model = GPT(config)
checkpoint = torch.load("ABADES-SLM-15M.pt", map_location="cpu")
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()

# Generate text
sentence = "Once upon a time there was a little girl"
context = torch.tensor(enc.encode_ordinary(sentence)).unsqueeze(0)

with torch.no_grad():
    output = model.generate(context, max_new_tokens=200, temperature=0.8, top_k=40)

print(enc.decode(output.squeeze().tolist()))
```

---

## Training Details

- **Tokenizer:** GPT-2 BPE (`tiktoken`)
- **Optimizer:** AdamW (`lr=1e-4`, `betas=(0.9, 0.95)`, `weight_decay=0.1`)
- **LR Schedule:** Linear warmup (1000 steps) → Cosine decay
- **Mixed precision:** bfloat16 / float16
- **Gradient accumulation:** 32 steps
- **Gradient clipping:** 0.5

---

## Example Outputs

**Prompt:** `"Once upon a time there was a pumpkin."`

> Once upon a time there was a pumpkin. It was big and orange and lived in a garden...

**Prompt:** `"A little girl went to the woods"`

> A little girl went to the woods with her dog. They were looking for something fun to do...

---

## Limitations

- Trained only on simple children's stories (TinyStories)
- Context window limited to 128 tokens
- Not suitable for complex reasoning or factual tasks
- May generate repetitive or incoherent text on out-of-domain prompts

---

## Author

**ApyHTML19** — built as a learning project to understand transformer training from scratch.

Inspired by [nanoGPT](https://github.com/karpathy/nanoGPT) by Andrej Karpathy.