| --- |
| license: apache-2.0 |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - babylm |
| - babylm-2026 |
| - causal-lm |
| - dual-attention-transformer |
| - nextlat |
| - ema |
| - custom-code |
| --- |
| |
| # DAT Strict NextLat Final |
|
|
| DAT Strict NextLat Final is a causal Dual Attention Transformer language model trained from scratch for the BabyLM 2026 Strict track. It combines self-attention, relational attention, position-relative symbols, and a NextLat auxiliary training objective. The repository contains the tokenizer and custom HuggingFace Transformers code required by the Auto classes. |
|
|
| ## Architecture |
|
|
| - 16 transformer layers |
| - Hidden dimension 1,024 |
| - 12 self-attention heads and 4 relational-attention heads |
| - Rotary token positional encoding |
| - Shared position-relative symbol retrieval without relative-symbol RoPE |
| - RCA relational attention |
| - SwiGLU feed-forward layers |
| - Vocabulary size 16,384 |
| - Maximum model sequence length 514 |
| - Untied token embedding and language-model head |
|
|
| ## Training |
|
|
| The model was trained for ten passes through the BabyLM 2026 Strict corpus under the 100 million word data limit. Training used batches of 42 sequences of 512 tokens, seed 1, Muon for hidden matrix parameters, and LambW for the remaining parameters. The Muon learning rate was 0.02, the auxiliary learning rate was 0.001, weight decay was 0.01, and the schedule used one percent warmup followed by cosine decay to ten percent of the initial rate. |
|
|
| NextLat used horizon 1 with cross-entropy weight 0.0, KL weight 0.5, and MSE weight 1.0. An exponential moving average with decay 0.999 was maintained during training. The published final model and checkpoint revisions use the EMA weights. No teacher model, synthetic data augmentation, or multimodal input was used. |
|
|
| ## Checkpoints |
|
|
| The final EMA model is stored on `main`. Intermediate EMA checkpoints use the official BabyLM Strict revision names from `chck_1M` through `chck_1000M`. These names represent cumulative words processed across repeated passes through the corpus. |
|
|
| ## Loading |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| repo_id = "abe123/babylm-dat-strict-nextlat-final" |
| tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True) |
| model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True) |
| ``` |
|
|
| Use an explicit immutable commit SHA for reproducible evaluation. |
|
|
| ## Intended use and limitations |
|
|
| This checkpoint is intended for BabyLM evaluation and language-model research. It uses custom model code and therefore requires `trust_remote_code=True` when loaded through HuggingFace Auto classes. |
|
|
