---
license: mit
tags:
- nanochat
- gpt
- language-model
- custom-architecture
---

# NanoChat GPT Model (d20, Step 21400)

This is a trained language model using the NanoChat framework with custom architecture features:
- Rotary Position Embeddings (RoPE)
- Query-Key Normalization (QK Norm)
- ReLU^2 activation in MLP layers
- Group Query Attention (GQA) support

## Model Details

- **Architecture**: NanoChat GPT
- **Layers**: 20
- **Hidden Size**: 1280
- **Attention Heads**: 10
- **KV Heads**: 10
- **Vocabulary Size**: 65536
- **Max Sequence Length**: 512
- **Training Step**: 21400
- **Validation BPB**: 0.8233

## Usage

This model uses a custom architecture and requires the [NanoChat](https://github.com/karpathy/nanochat) codebase to load and use.

### Loading the Model

```python
from nanochat.checkpoint_manager import build_model
from nanochat.common import get_base_dir

# Download or clone this repo, then:
checkpoint_dir = "path/to/this/repo"  # or use get_base_dir()
step = 21400
device = "cuda"  # or "cpu"

model, tokenizer, meta_data = build_model(
    checkpoint_dir, 
    step, 
    device, 
    phase="eval"
)
```

### Loading Weights Only

If you want to load just the weights:

```python
from safetensors.torch import load_file
import torch

weights = load_file("model.safetensors")
# Then load into your model architecture
```

## Training Details

See `training_metadata.json` for full training configuration and hyperparameters.

## Citation

If you use this model, please cite the NanoChat repository.