Update README.md

README.md

@@ -1,30 +1,18 @@
 ---
-license: cc-by-nc-nd-4.0
 language:
 - en
-library_name: transformers
-pipeline_tag: text-generation
+license: cc-by-nc-nd-4.0
 tags:
 - text-generation
 - from-scratch
 - tiny-language-model
 - tinystories
 - llama-style
-- small-language-model
 - research
 datasets:
 - roneneldan/TinyStories
 metrics:
 - perplexity
-- loss
-inference: false
-widget:
-- text: "Once upon a time, there was a little"
-  example_title: "Story Opening"
-- text: "One day, a girl named Lily found"
-  example_title: "Discovery"
-- text: "A little dog named Max wanted to"
-  example_title: "Animal"
 model-index:
 - name: child-12m
   results:
@@ -37,170 +25,141 @@ model-index:
       split: validation
     metrics:
     - type: perplexity
-      value: 3.45
+      value: 3.41
       name: Perplexity
     - type: loss
-      value: 1.2373
+      value: 1.2268
       name: Average Validation Loss
-    - type: bits_per_byte
-      value: 0.4285
-      name: Bits per Byte
 ---
 
 # child-12m
 
 A 12.3M parameter decoder-only transformer trained **entirely from scratch** on TinyStories. No pretrained checkpoint, no distillation, no transfer learning. Every weight learned from random initialization.
 
-**Headline metrics:** validation perplexity **3.45**, bits-per-byte **0.4285**, **98%** accuracy on a 50-pair grammar probe spanning 10 linguistic paradigms.
-
----
-
 ## Results
 
-| Metric                                     | Value                          |
-| ------------------------------------------ | ------------------------------ |
-| Parameters (total)                         | 12,256,256                     |
-| Parameters (non-embedding)                 | ~10.1M (82% of total)          |
-| Validation loss (nats/token)               | 1.2373                         |
-| Validation perplexity                      | 3.45                           |
-| Bits-per-byte                              | 0.4285                         |
-| Bytes per token                            | 4.17                           |
-| Grammar accuracy (50 pairs, 10 paradigms)  | 98.0%                          |
-| Training tokens seen                       | ~1.0 billion (2+ epochs)       |
-| Training time                              | ~4 hours on a single Kaggle T4 |
+| Metric | Value |
+|--------|-------|
+| Parameters (total) | 12,256,256 |
+| Parameters (non-embedding) | ~10.1M (82% of total) |
+| Validation loss (nats/token) | 1.2268 |
+| Validation perplexity | 3.41 |
+| Bits-per-byte | 0.4249 |
+| Bytes per token | 4.17 |
+| BLiMP accuracy (full, 67 paradigms) | 63.4% |
+| Training tokens seen | ~1.6 billion |
+| Total training steps | 24,919 |
 
 Bits-per-byte is reported alongside perplexity so results can be compared fairly against models that use different tokenizers.
 
----
-
-## Architecture
-
-A 14-layer Llama-style decoder-only transformer.
+## BLiMP Evaluation (Full, 67 Paradigms)
 
-| Component         | Specification                          |
-| ----------------- | -------------------------------------- |
-| Layers            | 14                                     |
-| `d_model`         | 256                                    |
-| Attention         | GQA (4 query heads, 2 KV heads)        |
-| Head dimension    | 64                                     |
-| FFN               | SwiGLU, hidden dim 688                 |
-| Normalization     | RMSNorm (pre-norm)                     |
-| Position encoding | RoPE (`theta=10000`)                   |
-| Features          | QK-norm, z-loss, weight-tied embeddings |
-| Vocabulary        | 8,192 ByteLevelBPE                     |
-| Max context       | 2,048                                  |
-| Dropout           | 0.0                                    |
-
-The architecture prioritizes depth (14 layers) over width (`d_model=256`) and uses a compact 8K vocabulary, allocating ~82% of the parameter budget to transformer computation rather than embeddings.
+63.4% overall on the full BLiMP benchmark (67 paradigms, 200 pairs each, 13,400 total evaluations). Within range of BabyLM 2024 Strict-Small baselines (60.6%ΓÇô69.8%) at half the parameter count, despite training exclusively on TinyStories' restricted register.
 
----
+**Reference points:**
+| Model | Params | BLiMP |
+|-------|--------|-------|
+| Random baseline | ΓÇö | 50.0% |
+| BabyLM 2024 LTG-BERT 10M | ~25M | 60.6% |
+| **child-12m (this model)** | **12.3M** | **63.4%** |
+| BabyLM 2024 BabyLlama 10M | ~24M | 69.8% |
+| GPT-2 Small (web-scale) | 124M | ~81% |
 
-## Grammar Evaluation
+**Notable finding:** The score distribution is strongly bimodal. Paradigms that test constructions present in TinyStories (agreement, reflexives, basic negation) score 85ΓÇô99%. Paradigms testing constructions absent from TinyStories (complex syntactic islands, NPI scope, distractor agreement) score at or below chance. This is a diagnostic of training distribution, not model capacity ΓÇö the architecture learns what it is shown.
 
-50 minimal pairs across 10 linguistic paradigms.
+**Top paradigms (>85%):** Principle A case (99.5%), sentential negation NPI licensing (99.0%), Principle A domain (98.5%), wh-vs-that no gap (95.0%), existential there quantifiers (91.0%), superlative quantifiers (87.5ΓÇô90.0%), anaphor number agreement (88.0%), irregular past participles (83.0ΓÇô88.0%), wh-subject gap long distance (87.0%)
 
-| Paradigm                  | Accuracy |
-| ------------------------- | -------- |
-| Subject-verb agreement    | 5/5      |
-| Determiner-noun agreement | 5/5      |
-| Verb tense consistency    | 5/5      |
-| Pronoun case              | 5/5      |
-| Word order                | 5/5      |
-| Negation                  | 5/5      |
-| Auxiliary verb            | 5/5      |
-| Comparative/superlative   | 4/5      |
-| Prepositional phrase      | 5/5      |
-| Sentence completeness     | 5/5      |
-| **Overall**               | **49/50 (98.0%)** |
-
-The probe is hand-crafted and does not substitute for a full BLiMP run. Treat it as a sanity check rather than a leaderboard claim.
-
----
+**Bottom paradigms (<25%):** matrix question NPI (8.0%), only NPI scope (9.5%), wh-vs-that with gap long distance (9.5%), only NPI licensor (19.5%), wh-vs-that with gap (21.0%), coordinate structure left branch (24.5%)
 
-## Generation Samples
+**Methodology note:** The gap between a hand-crafted grammar probe (100% on 50 pairs testing TinyStories-native constructions) and full BLiMP (63.4%) illustrates why custom eval suites for TinyStories-class models can dramatically overestimate grammatical competence. Future evaluations of TinyStories-trained models should use BLiMP or equivalent standardized benchmarks.
 
-**Prompt:** *"Once upon a time, there was a little"*
-
-> Once upon a time, there was a little boy named Timmy. Timmy loved to play with his toys all day long. One day, Timmy's mom asked him to help her in the kitchen. Timmy didn't want to stop playing, but...
+## Architecture
 
-**Prompt:** *"One day, a girl named Lily found"*
+A 14-layer Llama-style decoder-only transformer.
 
-> One day, a girl named Lily found a small bird in her yard. The bird was hurt and could not fly. Lily wanted to help the bird, so she asked her mom for help. "Mom, can we help the bird?" Lily asked.
+| Component | Specification |
+|-----------|--------------|
+| Layers | 14 |
+| d_model | 256 |
+| Attention | GQA (4 query heads, 2 KV heads) |
+| Head dimension | 64 |
+| FFN | SwiGLU, hidden dim 688 |
+| Normalization | RMSNorm (pre-norm) |
+| Position encoding | RoPE (theta=10000) |
+| Features | QK-norm, z-loss, weight-tied embeddings |
+| Vocabulary | 8,192 ByteLevelBPE |
+| Max context | 2,048 |
+| Dropout | 0.0 |
 
-**Prompt:** *"Mom said to the little boy,"*
+The architecture prioritizes depth (14 layers) over width (d_model=256) and uses a compact 8K vocabulary, allocating ~82% of the parameter budget to transformer computation rather than embeddings.
 
-> Mom said to the little boy, "Let's take a walk to the park, okay?" Little boy smiled and said, "Okay, mom!" They walked and walked until they arrived at the park. Little boy was so happy, he ran around...
+## Generation Samples (step 24,919)
 
-**Prompt:** *"A little dog named Max wanted to"*
+**Prompt:** "Once upon a time, there was a little"
+> Once upon a time, there was a little girl named Lily. She had a toy bear that she loved very much. One day, Lily went to the park to play with her bear. She saw a boy playing with a ball and asked him...
 
-> A little dog named Max wanted to try the lime. He picked it up with his mouth and started to chew. The lime was very sour and made his teeth feel funny. Max tried to chew the lime again, but he was stuck...
+**Prompt:** "One day, a girl named Lily found"
+> One day, a girl named Lily found a mysterious box in her room. She tried to open it, but it was locked. She tried to find the key, but it was locked. Lily was sad because the box was too hard to open.
 
-**Prompt:** *"Dad took his son to the"*
+**Prompt:** "Mom said to the little boy,"
+> Mom said to the little boy, "What's wrong? Why are you fighting?" The little boy said he wanted to play. He said, "No, silly. I don't want to share with you."
 
-> Dad took his son to the store. When they got there, his dad said, "Oh wow, it looks so expensive!" The son saw a big, shiny toy car. He said, "Dad, can we buy it?" His dad smiled and said, "Sure..."
+**Prompt:** "Papa said"
+> Papa said, "Let's go, let's go!" They went into the kitchen. It was dark and quiet. Papa said, "Let's make a fire and stay warm." Mama said, "Okay, let's make a fire and stay warm."
 
----
+**Prompt:** "She looked at the sky and"
+> She looked at the sky and saw a big rainbow. It was so big that she could not see the other side of it. She thought it was the most beautiful thing she had ever seen. Suddenly, a voice said, "What do..."
 
 ## Training
 
-| Field            | Value                                                |
-| ---------------- | ---------------------------------------------------- |
-| Dataset          | [roneneldan/TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) (2,119,719 stories) |
-| Tokens in dataset | 464,662,320                                         |
-| Tokens seen      | ~1.0 billion (2+ epochs)                             |
-| Optimizer        | AdamW (β₁=0.9, β₂=0.95, weight decay=0.1)            |
-| LR schedule      | 100-step warmup + cosine decay (3e-4 → 3e-5)         |
-| Total steps      | 15,387                                               |
-| Batch size       | 128 effective (32 × 4 gradient accumulation)         |
-| Sequence length  | 512                                                  |
-| Precision        | float16 (autocast + GradScaler)                      |
-| Gradient clip    | 1.0                                                  |
-| Hardware         | 1× Kaggle T4 GPU (16 GB)                             |
+| Field | Value |
+|-------|-------|
+| Dataset | roneneldan/TinyStories (2,119,719 stories) |
+| Tokens in dataset | 464,662,320 |
+| Tokens seen | ~1.6 billion (3+ epochs) |
+| Optimizer | AdamW (beta1=0.9, beta2=0.95, wd=0.1) |
+| LR schedule | WSD: warmup + stable at 2e-4 + cosine decay to 1e-5; then constant 5e-5 sessions |
+| Total steps | 24,919 |
+| Batch size | 128 effective (32 x 4 gradient accumulation) |
+| Sequence length | 512 |
+| Precision | float16 (autocast + GradScaler) |
+| Gradient clip | 1.0 |
+| Hardware | Kaggle T4 GPU (16 GB), single GPU, free tier |
 
 ### Training History
 
-| Checkpoint  | Step       | Train Loss | Val PPL  |
-| ----------- | ---------- | ---------- | -------- |
-| early       | 4,532      | 1.597      | 4.21     |
-| mid         | 5,734      | 1.579      | 4.07     |
-| **final**   | **15,387** | **1.437**  | **3.45** |
-
----
+| Checkpoint | Step | Train Loss | Val PPL | BPB |
+|-----------|------|-----------|---------|-----|
+| early | 4,532 | 1.597 | 4.21 | ΓÇö |
+| mid | 15,387 | 1.437 | 3.45 | 0.4285 |
+| final | 24,919 | 1.396 | 3.41 | 0.4249 |
 
 ## Limitations
 
-- Trained exclusively on TinyStories (3–4 year old vocabulary level)
-- No dialogue fine-tuning (base model only)
-- Grammar evaluation uses 50 hand-crafted pairs, not the full BLiMP suite
-- Single training run — no seed variance reported
+- Trained exclusively on TinyStories (3ΓÇô4 year old vocabulary level)
+- BLiMP performance is strongly distribution-dependent: high on TinyStories-native constructions, at-chance on absent ones
+- Single training run ΓÇö no seed variance reported
 - Not evaluated on general-purpose benchmarks
 - Not suitable for any production or safety-critical use
 
----
-
 ## Weights
 
 Weights are not publicly released. This model card documents architecture and results for research reference.
 
----
-
 ## Future Work
 
-- Dialogue fine-tuning on TinyDialogues
-- Full BLiMP / Zorro evaluation
+- Net2Net function-preserving expansion to 50M+ with diverse data (Simple Wikipedia)
+- Full BLiMP re-evaluation after data diversification to test whether bottom paradigms improve
 - Distortion-Reactive Plasticity (DRP) experiments
-- Net2Net function-preserving expansion to larger scales
-- Multi-seed runs to characterize training variance
-
----
+- Dialogue fine-tuning and identity training at larger scale
 
 ## Citation
 
-```bibtex
+```
 @misc{child12m2026,
   title  = {child-12m: A 12.3M Parameter Language Model Trained From Scratch on TinyStories},
   author = {Radji, Kraim and Claude (Anthropic Opus 4.6)},
   year   = {2026},
-  note   = {Perplexity 3.45, BPB 0.4285 on TinyStories validation. Weights not released.}
+  note   = {Perplexity 3.41, BPB 0.4249, BLiMP 63.4\% (full 67 paradigms). Weights not released.}
 }
 ```