---
language: [en]
tags: [babylm-2026, strict-small, masked-diffusion, mdlm, llada, md_base]
license: apache-2.0
---

# Masked-Diffusion BabyLM (Strict-Small) — `MD_base` seed `42`

BabyLM 2026 **Strict-Small** (English) submission checkpoint. A LLaDA/MDLM-style
absorbing-state masked-diffusion language model on a GPT-2-scale bidirectional
Transformer, trained on <=10M unique words for <=10 epochs (<=100M words seen).

- **Method**: `MD_base` (masked diffusion)
- **Seed**: `42`
- **Words seen (final)**: `99996737`
- **Intermediate checkpoints**: `19` branches `chck_1M ... chck_100M`

## Usage

```python
from transformers import AutoModelForMaskedLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("amosluna/babylm-2026-strict-small-mdlm-seed42")
model = AutoModelForMaskedLM.from_pretrained("amosluna/babylm-2026-strict-small-mdlm-seed42", trust_remote_code=True)
# Intermediate checkpoint:
# model = AutoModelForMaskedLM.from_pretrained("amosluna/babylm-2026-strict-small-mdlm-seed42", revision="chck_10M", trust_remote_code=True)
```

## Evaluation

A masked-diffusion denoiser is scored like a masked LM (per-token
pseudo-log-likelihood), so the official BabyLM pipeline evaluates it with the
`mlm` backend:

```bash
cd strict
./eval_zero_shot.sh amosluna/babylm-2026-strict-small-mdlm-seed42 mlm
./eval_finetuning.sh --model_path amosluna/babylm-2026-strict-small-mdlm-seed42 --seed 42
bash scripts/collate_preds.sh amosluna/babylm-2026-strict-small-mdlm-seed42 mlm strict-small --fast
```

See the project's `docs/EVALUATION.md` for the diffusion-native ELBO scorer and
the inference-time layer-duplication variant.