---
language: en
license: cc-by-nc-4.0
tags:
  - complexity-deep
  - dense-baseline
  - swiglu
  - checkpoint
  - resumable
---

# Dense SwiGLU Baseline (171M) — Training Checkpoint (Step 954)

Resumable training checkpoint with full optimizer state. Use this to continue training from step 954 (500M tokens).

## Contents

- `checkpoint.pt` - Model weights + training state
- `optimizer_rank0.pt` - AdamW optimizer state (GPU 0)
- `optimizer_rank1.pt` - AdamW optimizer state (GPU 1)
- `training_state.json` - Step counter, LR, etc.

## Model Config

- **Parameters**: 171M
- **Hidden**: 768, Layers: 18, Heads: 12, KV Heads: 4
- **MLP**: Dense SwiGLU, Intermediate: 2416
- **Training**: 500M tokens, AdamW lr=3e-4 (auto-scaled 6e-4), cosine 5% warmup

## Resume Training

```python
import torch

checkpoint = torch.load("checkpoint.pt", map_location="cpu")
model.load_state_dict(checkpoint["model"])

# Load optimizer for your GPU rank (0 or 1)
rank = torch.distributed.get_rank()
optimizer_state = torch.load(f"optimizer_rank{rank}.pt", map_location="cpu")
optimizer.load_state_dict(optimizer_state)

# Resume from step 954
```

## Pretrained Weights (inference)

For inference use the safetensors checkpoint in `../final/` instead.

## License

CC-BY-NC-4.0

Complexity-ML -- 2026