---
language: en
license: mit
tags:
- babylm
- babylm-2026
- strict-small
- gpt2
---

# babylm-2026-strict-small-childes-heavy

BabyLM Challenge 2026 — **Strict-Small track** (≤10M words). Part of
*Reweighting Child-Directed and Conversational Data for Sample-Efficient
BabyLM Pretraining* (strict_small_childes_heavy).

- **Architecture:** GPT-2-style decoder-only transformer, from-scratch init
  (no pretrained weights): 6 layers, 512 hidden, 8 heads,
  512-token context, 27.4M parameters (tied embeddings).
- **Objective:** causal language modeling.
- **Tokenizer:** byte-level BPE, 16k vocab, trained **only** on this run's
  ≤10M-word training mixture.
- **Training data:** 9,999,223 words (custom mixture resampled from the official 100M Strict pool). Word counts use
  whitespace tokenization matching the official BabyLM 2026 dataset cards.
  Manifest and datasheet in the [GitHub repo](https://github.com/qyxu1994/babylm-2026).
- **Training:** 10 epochs (~100M words exposure), packed
  512-token sequences, effective batch 128, lr 5e-4 cosine (3% warmup),
  weight decay 0.1, bf16, seed 42, single A100-80GB.

## Intermediate checkpoints

As required by BabyLM 2026, checkpoints at every 1M words seen (to 10M) and
every 10M (to 100M) are available as revisions `chck_1M` … `chck_100M`:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("qyxu1994/babylm-2026-strict-small-childes-heavy", revision="chck_5M")
tokenizer = AutoTokenizer.from_pretrained("qyxu1994/babylm-2026-strict-small-childes-heavy")
```

## Evaluation

Evaluated with the official [babylm-eval](https://github.com/babylm-org/babylm-eval)
pipeline (zero-shot: BLiMP, BLiMP-supplement, EWoK, entity tracking, COMPS,
reading; fine-tuning: BoolQ, MNLI, MRPC, MultiRC, QQP, RTE, WSC):

```bash
./eval_zero_shot.sh qyxu1994/babylm-2026-strict-small-childes-heavy causal evaluation_data/full_eval
```

## Mixture (strict_small_childes_heavy)

| Source | Words |
|---|---|
| childes | 4,999,995 |
| bnc_spoken | 1,499,975 |
| switchboard | 239,428 |
| open_subtitles | 1,499,945 |
| gutenberg | 999,991 |
| simple_wiki | 759,889 |

## Limitations

Single pretraining seed; 27M parameters; English only; trained on ≤10M words —
a research artifact for sample-efficiency study, not a production model.

## Citation

Paper forthcoming (BabyLM Challenge 2026). Please also cite the BabyLM 2026
call for papers (arXiv:2602.20092).