# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("mcxfrank/childes-gpt2-ladder", device_map="auto")CHILDES-trained GPT-2 developmental ladder
A ladder of GPT-2-small models trained from scratch on child-directed speech (CHILDES) at increasing amounts of input. It is the language-model side of a children-vs-LMs study of how vocabulary is acquired across "development," and is meant as a research artifact for work on developmental / sample-efficient language modeling.
What's here
180 models = 10 random seeds × 18 nested input budgets (0.5M → 24M words).
development/seed{S}/rung{B}/
S ∈ {0, 1, 2, 3, 4, 5, 7, 42, 99, 123}
B ∈ {0.5M, 0.75M, 1M, 1.25M, 1.5M, 1.75M, 2M, 2.5M, 3M, 3.5M, 4M, 5M, 6M, 7M, 8M, 12M, 16M, 24M}
The budgets are nested (each larger budget contains the smaller), and a seed fixes both the
initialization and the input-stream order — so each seed is an "individual" accumulating input,
i.e. a developmental trajectory. Every model directory is self-contained (weights + tokenizer +
config + provenance.json).
Load a model
from transformers import AutoModelForCausalLM, AutoTokenizer
sub = "development/seed42/rung24M" # pick any seed/budget
model = AutoModelForCausalLM.from_pretrained("mcxfrank/childes-gpt2-ladder", subfolder=sub, torch_dtype="auto")
tok = AutoTokenizer.from_pretrained("mcxfrank/childes-gpt2-ladder", subfolder=sub)
Use torch_dtype="auto" to preserve the stored bf16 weights (otherwise they upcast to fp32).
Training
- Architecture: GPT-2-small (125.8M params; 12 layers, 768-dim), trained from scratch.
- Tokenizer:
GPT2_CHILDESbyte-pair encoding (52k vocab), trained on CHILDES (Feng et al. / TinyDialogues). - Data: English CHILDES (Feng et al. 2024 preparation). Budgets are counted in whitespace words (the developmental currency); each seed shuffles conversations and takes cumulative prefixes to each budget.
- Optimization: up to 40 epochs with early stopping (patience 4) on held-out validation loss; the lowest-validation-loss checkpoint is kept. AdamW, learning rate 1e-4 (linear decay, no warmup), weight decay 0, batch size 8, sequence length 1024, no in-epoch shuffling.
- Precision: stored as bf16 safetensors.
Provenance & reproduction
Each model carries a provenance.json (seed, budget, config). These weights were re-trained from
the exact ladder samples used in the analysis; re-trained per-word surprisal reproduces the
published values at r = 0.988 (aggregate mean within ~0.04 nats). GPU nondeterminism makes
them faithful re-trainings, not bit-identical.
Intended use & limitations
Research artifacts for studying developmental / sample-efficient language modeling and word acquisition. These are child-scale models (≤24M words of input, English child-directed speech only) and are not intended for production use or as general-purpose language models.
Citation
Paper: Children, but not language models, learn words via accelerating accumulation (Frank, in prep). Data & training pipeline: Feng et al. (2024); CHILDES (MacWhinney, TalkBank).
License
To be set by the repository owner. The training data (CHILDES via TalkBank) is subject to TalkBank's terms of use; please respect those when using or redistributing these models.
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="mcxfrank/childes-gpt2-ladder")