---
license: other
library_name: transformers
pipeline_tag: text-generation
tags:
  - gpt2
  - childes
  - child-directed-speech
  - language-acquisition
  - developmental
  - sample-efficient-pretraining
---

# CHILDES-trained GPT-2 developmental ladder

A ladder of **GPT-2-small models trained from scratch on child-directed speech (CHILDES)**
at increasing amounts of input. It is the language-model side of a children-vs-LMs study of
how vocabulary is acquired across "development," and is meant as a research artifact for work
on developmental / sample-efficient language modeling.

## What's here

**180 models = 10 random seeds × 18 nested input budgets** (0.5M → 24M words).

```
development/seed{S}/rung{B}/
  S ∈ {0, 1, 2, 3, 4, 5, 7, 42, 99, 123}
  B ∈ {0.5M, 0.75M, 1M, 1.25M, 1.5M, 1.75M, 2M, 2.5M, 3M, 3.5M, 4M, 5M, 6M, 7M, 8M, 12M, 16M, 24M}
```

The budgets are **nested** (each larger budget contains the smaller), and a seed fixes both the
initialization and the input-stream order — so each seed is an "individual" accumulating input,
i.e. a developmental trajectory. Every model directory is self-contained (weights + tokenizer +
config + `provenance.json`).

## Load a model

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
sub = "development/seed42/rung24M"                     # pick any seed/budget
model = AutoModelForCausalLM.from_pretrained("mcxfrank/childes-gpt2-ladder", subfolder=sub, torch_dtype="auto")
tok   = AutoTokenizer.from_pretrained("mcxfrank/childes-gpt2-ladder", subfolder=sub)
```

Use `torch_dtype="auto"` to preserve the stored bf16 weights (otherwise they upcast to fp32).

## Training

- **Architecture:** GPT-2-small (125.8M params; 12 layers, 768-dim), trained **from scratch**.
- **Tokenizer:** `GPT2_CHILDES` byte-pair encoding (52k vocab), trained on CHILDES (Feng et al. / TinyDialogues).
- **Data:** English CHILDES (Feng et al. 2024 preparation). Budgets are counted in whitespace
  words (the developmental currency); each seed shuffles conversations and takes cumulative
  prefixes to each budget.
- **Optimization:** up to 40 epochs with **early stopping** (patience 4) on held-out validation
  loss; the lowest-validation-loss checkpoint is kept. AdamW, learning rate 1e-4 (linear decay,
  no warmup), weight decay 0, batch size 8, sequence length 1024, no in-epoch shuffling.
- **Precision:** stored as **bf16** safetensors.

## Provenance & reproduction

Each model carries a `provenance.json` (seed, budget, config). These weights were re-trained from
the exact ladder samples used in the analysis; re-trained per-word surprisal reproduces the
published values at **r = 0.988** (aggregate mean within ~0.04 nats). GPU nondeterminism makes
them faithful re-trainings, not bit-identical.

## Intended use & limitations

Research artifacts for studying developmental / sample-efficient language modeling and word
acquisition. These are **child-scale** models (≤24M words of input, English child-directed
speech only) and are **not** intended for production use or as general-purpose language models.

## Citation

Paper: *Children, but not language models, learn words via accelerating accumulation* (Frank, in prep).
Data & training pipeline: Feng et al. (2024); CHILDES (MacWhinney, TalkBank).

## License

**To be set by the repository owner.** The training data (CHILDES via TalkBank) is subject to
TalkBank's terms of use; please respect those when using or redistributing these models.