mcxfrank/childes-gpt2-ladder

CHILDES-trained GPT-2 developmental ladder

A ladder of GPT-2-small models trained from scratch on child-directed speech (CHILDES) at increasing amounts of input. It is the language-model side of a children-vs-LMs study of how vocabulary is acquired across "development," and is meant as a research artifact for work on developmental / sample-efficient language modeling.

What's here

180 models = 10 random seeds × 18 nested input budgets (0.5M → 24M words).

development/seed{S}/rung{B}/
  S ∈ {0, 1, 2, 3, 4, 5, 7, 42, 99, 123}
  B ∈ {0.5M, 0.75M, 1M, 1.25M, 1.5M, 1.75M, 2M, 2.5M, 3M, 3.5M, 4M, 5M, 6M, 7M, 8M, 12M, 16M, 24M}

The budgets are nested (each larger budget contains the smaller), and a seed fixes both the initialization and the input-stream order — so each seed is an "individual" accumulating input, i.e. a developmental trajectory. Every model directory is self-contained (weights + tokenizer + config + provenance.json).

Load a model

from transformers import AutoModelForCausalLM, AutoTokenizer
sub = "development/seed42/rung24M"                     # pick any seed/budget
model = AutoModelForCausalLM.from_pretrained("mcxfrank/childes-gpt2-ladder", subfolder=sub, torch_dtype="auto")
tok   = AutoTokenizer.from_pretrained("mcxfrank/childes-gpt2-ladder", subfolder=sub)

Use torch_dtype="auto" to preserve the stored bf16 weights (otherwise they upcast to fp32).

Training

• Architecture: GPT-2-small (125.8M params; 12 layers, 768-dim), trained from scratch.
• Tokenizer: GPT2_CHILDES byte-pair encoding (52k vocab), trained on CHILDES (Feng et al. / TinyDialogues).
• Data: English CHILDES (Feng et al. 2024 preparation). Budgets are counted in whitespace words (the developmental currency); each seed shuffles conversations and takes cumulative prefixes to each budget.
• Optimization: up to 40 epochs with early stopping (patience 4) on held-out validation loss; the lowest-validation-loss checkpoint is kept. AdamW, learning rate 1e-4 (linear decay, no warmup), weight decay 0, batch size 8, sequence length 1024, no in-epoch shuffling.
• Precision: stored as bf16 safetensors.

Provenance & reproduction

Each model carries a provenance.json (seed, budget, config). These weights were re-trained from the exact ladder samples used in the analysis; re-trained per-word surprisal reproduces the published values at r = 0.988 (aggregate mean within ~0.04 nats). GPU nondeterminism makes them faithful re-trainings, not bit-identical.

Intended use & limitations

Research artifacts for studying developmental / sample-efficient language modeling and word acquisition. These are child-scale models (≤24M words of input, English child-directed speech only) and are not intended for production use or as general-purpose language models.

Citation

Paper: Children, but not language models, learn words via accelerating accumulation (Frank, in prep). Data & training pipeline: Feng et al. (2024); CHILDES (MacWhinney, TalkBank).

License

To be set by the repository owner. The training data (CHILDES via TalkBank) is subject to TalkBank's terms of use; please respect those when using or redistributing these models.