---
license: cc-by-nc-4.0
language:
- en
datasets:
- nvidia/ClimbMix
pipeline_tag: text-generation
library_name: transformers
tags:
- base-model
- pretraining
- research
- nanochat
- scaling-ladder
---

# Scaling Ladder — d26 (973M total parameters), seed 2

**Research artifact.** One of the base language models of a scaling ladder
(28 models: 8 seeds at d12 + 8 seeds at d16 + 8 seeds at d20 + 4 seeds at d26) trained to study seed-to-seed variance in
language-model pretraining. Every model is a plain next-token predictor
trained for 200 tokens per parameter. No instruction tuning, no safety
training.

*Tokens per parameter (TPP) is computed over scaling parameters throughout,
not total parameters — see Parameter counts below.*

This repository holds **size d26, seed 2**. The seed sets both the
weight initialization and the data order; everything else is identical
across the seeds of a size.

**This revision (`main`)** mirrors `TPP_200` — the completed ladder.

![validation loss across tokens per parameter](plots/bpb_across_tpp.png)

*Validation loss in bits per byte (lower is better) against realized tokens
per parameter, measured at each checkpoint save on the training run's
held-out validation split. One series is the annealed models (the `TPP_X`
revisions); the other is the un-annealed main-run (constant learning rate)
checkpoints they forked from (the `TPP_X_preanneal` revisions). A thin
connector joins each annealed model to the fork checkpoint its anneal
started from. The isolated point at the far left, when present, is the
`from_init` revision. This repository's single run only — no averaging
across seeds.*

## The ladder

| repos | seeds | layers | hidden size | total parameters | scaling parameters |
|---|---|---|---|---|---|
| d12_135m_seed1..8 | 8 | 12 | 768 | 135M | 110M |
| d16_268m_seed1..8 | 8 | 16 | 1024 | 268M | 235M |
| d20_477m_seed1..8 | 8 | 20 | 1280 | 477M | 435M |
| d26_973m_seed1..4 | 4 | 26 | 1664 | 973M | 918M |

All models of this ladder share one architecture family: a plain GPT with
none of nanochat's optional architecture mechanisms (see Architecture).
A sibling ladder trained with all of those mechanisms enabled is published
separately, under repo names carrying an `_optimized_` marker
(e.g. `d20_optimized_897m_seed1`); repo names without a marker, like this
one, always belong to the plain-architecture ladder.

## Parameter counts

The size in each repository name is the **total** parameter count of the
checkpoint — everything `model.safetensors` holds, including the input
token embedding: 135,266,354 / 268,435,514 / 477,102,146 / 972,947,534 for
d12 / d16 / d20 / d26. The **scaling-parameter** count — the weight
matrices plus the output head only, and the basis for tokens per parameter
and the scaling-law fits — is smaller: 110,100,480 / 234,881,024 /
435,159,040 / 918,421,504. The gap is the untied input token embedding
(vocabulary 32,768 x hidden size) plus a handful of scalar gains. All
counts are read from the training runs' own parameter-count logs.

## Revisions: checkpoints along training

Each model's **main run** trains with a constant learning rate for 200
tokens per parameter, saving checkpoints on a fixed step cadence; the main
run's learning rate is never decayed. Every annealed checkpoint comes from
a separate **anneal run**: it forks the main run at the saved checkpoint
closest to (mark minus 6.27B tokens) — before the mark — then
trains a fixed 6.27B tokens (for this size) while the learning
rate decays linearly to 5% of the constant value, landing at the mark.
The annealed model's total token count is therefore the mark itself, not
the mark plus the anneal. Because the fork snaps to the nearest saved
checkpoint, the realized total can deviate from the nominal mark; the
table below records it exactly, and the deviation is largest at the
lowest marks. Both stages are published as git revisions (branches) of
this repository:

- **`TPP_X`** (X = 10, 20, 30, 40, then every 20 up to 200): the annealed
  model at the X-tokens-per-parameter mark. Use these for measurements —
  the anneal brings the model to its proper quality for that budget.
- **`TPP_X_preanneal`**: the constant-learning-rate checkpoint of the
  main run that the TPP_X anneal forked from. Nominally the fork point
  sits about 6.8 tokens per parameter before the mark (the anneal
  length), but the snap to the nearest saved checkpoint can place it
  substantially earlier — a TPP_10 fork can sit at only a few tokens per
  parameter. The table below records every pre-anneal position exactly.
- **`from_init`**: the same anneal recipe applied from initialization —
  a standard warmup, then the full 6.27B-token linear decay,
  with no constant-learning-rate phase before it. It is the ladder's
  lowest-budget point.
- **`main`**: identical to `TPP_200` once it exists; while the ladder is
  still training, `main` holds the latest available `TPP_X`.

Marks are added incrementally while training continues, so a missing
revision only means it has not landed yet.

Currently available marks in this repository:

| mark | annealed step | annealed tokens/param | pre-anneal step | pre-anneal tokens/param |
|---|---|---|---|---|
| TPP_10 | 8477 | 9.68 | 2500 | 2.85 |
| TPP_20 | 17477 | 19.95 | 11500 | 13.13 |
| TPP_30 | 25977 | 29.66 | 20000 | 22.83 |
| TPP_40 | 34977 | 39.93 | 29000 | 33.11 |
| TPP_60 | 52477 | 59.91 | 46500 | 53.09 |
| TPP_80 | 69977 | 79.89 | 64000 | 73.07 |
| TPP_100 | 87477 | 99.87 | 81500 | 93.05 |
| TPP_120 | 104977 | 119.85 | 99000 | 113.03 |
| TPP_140 | 122477 | 139.83 | 116500 | 133.01 |
| TPP_160 | 139977 | 159.81 | 134000 | 152.99 |
| TPP_180 | 157477 | 179.79 | 151500 | 172.97 |
| TPP_200 | 174977 | 199.77 | 169000 | 192.95 |

"Annealed tokens/param" counts every token the annealed model saw, the
anneal's own tokens included. "Pre-anneal tokens/param" is the fork
point's position in the main run: fork step x tokens per step / scaling
parameters.

The `from_init` revision (step 6017, 6.87 tokens per parameter in total) has no pre-anneal row: it forked nothing — warmup plus the full decay, directly from initialization.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "jkminder/d26_973m_seed2"
revision = "TPP_100"  # or any revision above
tok = AutoTokenizer.from_pretrained(repo, revision=revision, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, revision=revision, trust_remote_code=True, dtype="bfloat16")
```

`trust_remote_code=True` is required: the architecture matches no stock
transformers class, so the modeling code ships in the repository
(`modeling_nanochat_gpt.py`, plain PyTorch). `generate()` is supported
with a KV cache, greedy, sampling and beam search alike. Assisted decoding
(an assistant model) is refused: it requires cache cropping, which the
implementation does not support.

## Architecture

A plain GPT — [nanochat](https://github.com/karpathy/nanochat) with all
six of its optional architecture mechanisms disabled: no value embeddings,
no re-injection of the input embedding at the layers, no per-layer
residual scaling, no learned previous-token mixing gate, no mid-network
subtraction of the stored input contribution, no query-key sharpening.
Attention is full-context in every layer (no sliding windows). The logit
softcap of 15 is kept. Dimensions at this size: depth 26, hidden
size 1664, 13 attention heads (head dimension 128), sequence
length 2048, vocabulary 32,768. Also: parameter-free RMSNorm, rotary
embeddings (base 100,000) with query-key RMS normalization after rotation,
relu(x)^2 MLP, no biases, untied embeddings.

Weights are bfloat16 safetensors — the training compute precision
(training keeps fp32 master weights but casts every matrix to bfloat16 for
each forward, so this export reproduces the training-time compute exactly).
Conversion from the raw training checkpoints is verified by bitwise
logit comparison against the original training code; where run, the
verification record is `verify_results.json` in the revision.

## Tokenizer

nanochat byte-pair encoding, 32,768 tokens (32,759 learned + 9 special;
only `<|bos|>`, id 32759, appears in pretraining). Trained once on
ClimbMix and pinned across every model of the study. Load it with
`trust_remote_code=True` as in the snippet (the config carries an
`auto_map`, and resolving it without the flag triggers an interactive
prompt).

## Training data

[ClimbMix](https://huggingface.co/datasets/nvidia/ClimbMix) (NVIDIA,
filtered English web text), pinned snapshot `climbmix_4100`, single pass,
sequences of 2048 tokens, 1,048,576 tokens per step at this size.
The d12 and d16 models train on the `climbmix_1201` snapshot; d20 and d26
need more tokens than one pass of it holds and train on the larger
`climbmix_4100` snapshot of the same source. The base data carries a
CC BY-NC 4.0, research-and-development-only license, which this model
mirrors.

## License

- Model weights: **cc-by-nc-4.0** (mirrors the ClimbMix training data;
  research use).
- Modeling/configuration code: MIT (derived from karpathy/nanochat; see
  the bundled LICENSE file).