Release Pollock 1.2 (r004)

Publish the verified r004 Transformers artifact, tokenizer, English benchmark results, model card, changelog, training history, and release manifest.

CHANGELOG.md

@@ -2,6 +2,14 @@
 
 Revision numbers identify published model states independently of release names.
 
+## r004 — Pollock 1.2
+
+- Changed the architecture from 12/14/896 to 14/13/832 while keeping nearly the same parameter count.
+- Kept the r003 tokenizer, dataset, context, effective batch, token budget, and 4e-4→4e-5 learning-rate range unchanged.
+- Improved LAMBADA and SciQ over r003; the other five reported English primary metrics declined.
+
+Full record: [`training-history/r004.md`](./training-history/r004.md)
+
 ## r003 — Pollock 1.1
 
 - Kept the r002 architecture, tokenizer, dataset, context, effective batch, and four-pass token budget unchanged.

README.md

@@ -8,35 +8,35 @@ license_link: https://huggingface.co/SlayerLab/pollock-mini-lm-125m/blob/main/LI
 datasets: [SlayerLab/minimal-en-corpus-2.5b]
 tags: [causal-lm, gpt2, nanogpt, bpe, educational, base-model]
 model-index:
-  - name: Pollock 1.1
+  - name: Pollock 1.2
     results:
       - task: {type: text-generation, name: Language modeling}
         dataset: {type: SlayerLab/minimal-en-corpus-2.5b, name: Minimal EN validation (sampled), split: validation}
-        metrics: [{type: loss, value: 2.5528755188, name: Final sampled validation loss}]
+        metrics: [{type: loss, value: 2.5352059555, name: Final sampled validation loss}]
       - task: {type: text-generation, name: Zero-shot evaluation}
         dataset: {type: blimp, name: BLiMP, split: train}
-        metrics: [{type: acc, value: 0.7773880597}]
+        metrics: [{type: acc, value: 0.7720149254}]
       - task: {type: text-generation, name: Zero-shot evaluation}
         dataset: {type: EleutherAI/lambada_openai, name: LAMBADA OpenAI, split: test}
-        metrics: [{type: acc, value: 0.2899281972}, {type: perplexity, value: 47.70114023}]
+        metrics: [{type: acc, value: 0.2928391228}, {type: perplexity, value: 47.56334054}]
       - task: {type: text-generation, name: Zero-shot evaluation}
         dataset: {type: hellaswag, name: HellaSwag, split: validation}
-        metrics: [{type: acc_norm, value: 0.3034256124}]
+        metrics: [{type: acc_norm, value: 0.3006373232}]
       - task: {type: text-generation, name: Zero-shot evaluation}
         dataset: {type: piqa, name: PIQA, split: validation}
-        metrics: [{type: acc_norm, value: 0.6109902067}]
+        metrics: [{type: acc_norm, value: 0.6077257889}]
       - task: {type: text-generation, name: Zero-shot evaluation}
         dataset: {type: sciq, name: SciQ, split: test}
-        metrics: [{type: acc_norm, value: 0.659}]
+        metrics: [{type: acc_norm, value: 0.67}]
       - task: {type: text-generation, name: Zero-shot evaluation}
         dataset: {type: allenai/ai2_arc, config: ARC-Easy, name: ARC-Easy, split: test}
-        metrics: [{type: acc_norm, value: 0.428030303}]
+        metrics: [{type: acc_norm, value: 0.4225589226}]
       - task: {type: text-generation, name: Zero-shot evaluation}
         dataset: {type: allenai/ai2_arc, config: ARC-Challenge, name: ARC-Challenge, split: test}
-        metrics: [{type: acc_norm, value: 0.2534129693}]
+        metrics: [{type: acc_norm, value: 0.2346416382}]
 ---
 
-# Pollock 1.1 — r003
+# Pollock 1.2 — r004
 
 ![Pollock avatar](./assets/pollock-mini-lm-avatar-320.png)
 
@@ -48,23 +48,23 @@ model-index:
 
 Pollock to niewielki, anglojęzyczny model bazowy typu decoder-only, wytrenowany od zera jako czytelny eksperyment edukacyjny. Implementacja bazuje na [nanoGPT](https://github.com/karpathy/nanoGPT) i własnym tokenizerze byte-level BPE. Jest to model do uzupełniania tekstu, nie asystent konwersacyjny.
 
-Nazwa luźno nawiązuje do gestu malarskiego Jacksona Pollocka: nanoGPT jest płótnem, na którym dane, konfiguracja i decyzje treningowe tworzą różne wzorce zachowania. Pełne dane techniczne tej wersji znajdują się w [`training-history/r003.md`](./training-history/r003.md), a różnice między wydaniami w [`CHANGELOG.md`](./CHANGELOG.md).
+Nazwa luźno nawiązuje do gestu malarskiego Jacksona Pollocka: nanoGPT jest płótnem, na którym dane, konfiguracja i decyzje treningowe tworzą różne wzorce zachowania. Pełne dane techniczne tej wersji znajdują się w [`training-history/r004.md`](./training-history/r004.md), a różnice między wydaniami w [`CHANGELOG.md`](./CHANGELOG.md).
 
 ### Architektura i tokenizer
 
 | Właściwość | Wartość |
 |---|---:|
-| Rewizja / wydanie | r003 / Pollock 1.1 |
+| Rewizja / wydanie | r004 / Pollock 1.2 |
 | Typ | decoder-only Transformer w stylu GPT-2 |
-| Warstwy / głowy / embedding | 12 / 14 / 896 |
+| Warstwy / głowy / embedding | 14 / 13 / 832 |
 | Maksymalny kontekst | 1024 tokeny |
 | Słownik | 12 288 tokenów |
-| Parametry nanoGPT | 126 637 952 |
-| Łączne unikalne parametry trenowalne | 127 555 456 |
+| Parametry nanoGPT | 126 541 376 |
+| Łączne unikalne parametry trenowalne | 127 393 344 |
 | Tokenizer | byte-level BPE, pretokenizacja w stylu GPT-2 |
 | Tokeny specjalne | <code>&lt;&#124;endoftext&#124;&gt;</code>, <code>&lt;&#124;im_start&#124;&gt;</code>, <code>&lt;&#124;im_end&#124;&gt;</code> |
 
-Artefakt Transformers ma 127 674 624 parametrów, w tym 119 168 zerowych parametrów bias dla zgodności z `GPT2LMHeadModel`. Natywny model był trenowany z `bias=False`.
+Artefakt Transformers ma 127 522 304 parametry, w tym 128 960 zerowych parametrów bias dla zgodności z `GPT2LMHeadModel`. Natywny model był trenowany z `bias=False`.
 
 ### Dane i trening
 
@@ -73,35 +73,35 @@ Model wytrenowano na [`SlayerLab/minimal-en-corpus-2.5b`](https://huggingface.co
 | Parametr | Wartość |
 |---|---:|
 | Tokeny treningowe / walidacyjne | 2 703 768 815 / 5 267 742 |
-| Finalny checkpoint | iteracja 22 004 |
+| Finalny checkpoint | aktualizacja 22 004 |
 | Przetworzone tokeny | 10 815 406 080 (około 4 epoki) |
 | Sekwencja / micro-batch na GPU | 1024 / 12 |
-| Akumulacja globalna / na GPU | 40 / 20 micro-stepów |
+| Akumulacja globalna / na GPU | 40 / 40 micro-stepów |
 | Effective batch | 491 520 tokenów |
 | Optymalizator | fused AdamW, betas 0.9/0.95 |
 | Learning rate | 4e-4 → 4e-5, cosine decay |
 | Warmup / weight decay / grad clip | 440 / 0.1 / 1.0 |
 | Precyzja | BF16 |
-| Sprzęt | 2× NVIDIA L40S |
+| Sprzęt | 1× NVIDIA GeForce RTX 5090 |
 | Framework | PyTorch 2.8.0+cu128, nanoGPT commit `3adf61e` |
 
 ### Ewaluacja
 
-Loss treningowy szacowano na stałych podzbiorach po 1 228 800 tokenów na split. Finalny checkpoint uzyskał validation loss **2.552876**; najlepszy wynik to **2.5362** w kroku 20 000. Nie należy porównywać tych wartości bezpośrednio z losowo próbkowaną walidacją r002.
+Loss treningowy szacowano na stałych podzbiorach po 1 228 800 tokenów na split. Finalny checkpoint uzyskał validation loss **2.535206**; najlepszy wynik to **2.535055** w kroku 22 000. Nie należy porównywać tych wartości bezpośrednio z r003, ponieważ rewizje używały różnych offsetów podzbioru walidacyjnego.
 
 Benchmarki wykonano zero-shot na pełnych splitach przy użyciu `lm-evaluation-harness` 0.4.12, batch size 8 i BF16.
 
 | Benchmark | Główna metryka | Wynik | Próbki |
 |---|---|---:|---:|
-| BLiMP | accuracy | 0.777388 | 67 000 |
-| LAMBADA OpenAI | accuracy | 0.289928 | 5 153 |
-| HellaSwag | normalized accuracy | 0.303426 | 10 042 |
-| PIQA | normalized accuracy | 0.610990 | 1 838 |
-| SciQ | normalized accuracy | 0.659000 | 1 000 |
-| ARC-Easy | normalized accuracy | 0.428030 | 2 376 |
-| ARC-Challenge | normalized accuracy | 0.253413 | 1 172 |
+| BLiMP | accuracy | 0.772015 | 67 000 |
+| LAMBADA OpenAI | accuracy | 0.292839 | 5 153 |
+| HellaSwag | normalized accuracy | 0.300637 | 10 042 |
+| PIQA | normalized accuracy | 0.607726 | 1 838 |
+| SciQ | normalized accuracy | 0.670000 | 1 000 |
+| ARC-Easy | normalized accuracy | 0.422559 | 2 376 |
+| ARC-Challenge | normalized accuracy | 0.234642 | 1 172 |
 
-LAMBADA osiągnęła perplexity 47.701140. Pełne metryki i protokół zapisano w historii r003.
+LAMBADA osiągnęła perplexity 47.563341. Pełne metryki i protokół zapisano w historii r004.
 
 ### Użycie z Transformers
 
@@ -119,7 +119,7 @@ output = model.generate(
 print(tokenizer.decode(output[0], skip_special_tokens=True))
 ```
 
-Model używa standardowego `GPT2LMHeadModel`; `trust_remote_code=True` nie jest potrzebne. Dla r003 nie opublikowano jeszcze próbek generacji. Historyczne próbki są przypisane do poprzednich wag w historii r001.
+Model używa standardowego `GPT2LMHeadModel`; `trust_remote_code=True` nie jest potrzebne. Dla r004 nie opublikowano jeszcze próbek generacji. Historyczne próbki są przypisane do poprzednich wag w historii r001.
 
 ### Ograniczenia i odpowiedzialne użycie
 
@@ -136,23 +136,23 @@ Model używa standardowego `GPT2LMHeadModel`; `trust_remote_code=True` nie jest
 
 Pollock is a small English decoder-only base language model trained from scratch as a readable educational experiment. It is based on [nanoGPT](https://github.com/karpathy/nanoGPT) and a custom byte-level BPE tokenizer. It is a completion model, not a conversational assistant.
 
-The name loosely refers to Jackson Pollock's painterly gesture: nanoGPT is the canvas on which data, configuration, and training decisions create different behavioral patterns. See [`training-history/r003.md`](./training-history/r003.md) for the complete technical record and [`CHANGELOG.md`](./CHANGELOG.md) for release-to-release changes.
+The name loosely refers to Jackson Pollock's painterly gesture: nanoGPT is the canvas on which data, configuration, and training decisions create different behavioral patterns. See [`training-history/r004.md`](./training-history/r004.md) for the complete technical record and [`CHANGELOG.md`](./CHANGELOG.md) for release-to-release changes.
 
 ### Architecture and tokenizer
 
 | Property | Value |
 |---|---:|
-| Revision / release | r003 / Pollock 1.1 |
+| Revision / release | r004 / Pollock 1.2 |
 | Type | GPT-2-style decoder-only Transformer |
-| Layers / heads / width | 12 / 14 / 896 |
+| Layers / heads / width | 14 / 13 / 832 |
 | Maximum context | 1,024 tokens |
 | Vocabulary | 12,288 tokens |
-| nanoGPT parameters | 126,637,952 |
-| Total unique trainable parameters | 127,555,456 |
+| nanoGPT parameters | 126,541,376 |
+| Total unique trainable parameters | 127,393,344 |
 | Tokenizer | byte-level BPE, GPT-2-style pretokenization |
 | Special tokens | `<|endoftext|>`, `<|im_start|>`, `<|im_end|>` |
 
-The Transformers artifact has 127,674,624 parameters, including 119,168 zero-valued compatibility bias parameters required by `GPT2LMHeadModel`. The native model was trained with `bias=False`.
+The Transformers artifact has 127,522,304 parameters, including 128,960 zero-valued compatibility bias parameters required by `GPT2LMHeadModel`. The native model was trained with `bias=False`.
 
 ### Data and training
 
@@ -161,39 +161,39 @@ The model was trained on [`SlayerLab/minimal-en-corpus-2.5b`](https://huggingfac
 | Setting | Value |
 |---|---:|
 | Training / validation tokens | 2,703,768,815 / 5,267,742 |
-| Final checkpoint | iteration 22,004 |
+| Final checkpoint | update 22,004 |
 | Token presentations | 10,815,406,080 (approximately 4 epochs) |
 | Sequence / micro-batch per GPU | 1,024 / 12 |
-| Global / per-GPU accumulation | 40 / 20 micro-steps |
+| Global / per-GPU accumulation | 40 / 40 micro-steps |
 | Effective batch | 491,520 tokens |
 | Optimizer | fused AdamW, betas 0.9/0.95 |
 | Learning rate | 4e-4 → 4e-5, cosine decay |
 | Warmup / weight decay / grad clip | 440 / 0.1 / 1.0 |
 | Precision | BF16 |
-| Hardware | 2× NVIDIA L40S |
+| Hardware | 1× NVIDIA GeForce RTX 5090 |
 | Framework | PyTorch 2.8.0+cu128, nanoGPT commit `3adf61e` |
 
 ### Evaluation
 
-Training-time loss was estimated on fixed subsets of 1,228,800 tokens per split. The final checkpoint achieved validation loss **2.552876**; the best result was **2.5362** at step 20,000. These values are not directly comparable with r002's randomly sampled training validation.
+Training-time loss was estimated on fixed subsets of 1,228,800 tokens per split. The final checkpoint achieved validation loss **2.535206**; the best result was **2.535055** at step 22,000. These values are not directly comparable with r003 because the revisions used different validation-subset offsets.
 
 Benchmarks used complete splits with `lm-evaluation-harness` 0.4.12, zero few-shot examples, batch size 8, and BF16.
 
 | Benchmark | Primary metric | Score | Samples |
 |---|---|---:|---:|
-| BLiMP | accuracy | 0.777388 | 67,000 |
-| LAMBADA OpenAI | accuracy | 0.289928 | 5,153 |
-| HellaSwag | normalized accuracy | 0.303426 | 10,042 |
-| PIQA | normalized accuracy | 0.610990 | 1,838 |
-| SciQ | normalized accuracy | 0.659000 | 1,000 |
-| ARC-Easy | normalized accuracy | 0.428030 | 2,376 |
-| ARC-Challenge | normalized accuracy | 0.253413 | 1,172 |
+| BLiMP | accuracy | 0.772015 | 67,000 |
+| LAMBADA OpenAI | accuracy | 0.292839 | 5,153 |
+| HellaSwag | normalized accuracy | 0.300637 | 10,042 |
+| PIQA | normalized accuracy | 0.607726 | 1,838 |
+| SciQ | normalized accuracy | 0.670000 | 1,000 |
+| ARC-Easy | normalized accuracy | 0.422559 | 2,376 |
+| ARC-Challenge | normalized accuracy | 0.234642 | 1,172 |
 
-LAMBADA perplexity was 47.701140. Full metrics and protocol details are recorded in the r003 history.
+LAMBADA perplexity was 47.563341. Full metrics and protocol details are recorded in the r004 history.
 
 ### Usage
 
-Use the Transformers example in the Polish section. The artifact uses standard `GPT2LMHeadModel`; `trust_remote_code=True` is unnecessary. No r003 generation samples have been published yet. Historical samples are explicitly attached to the predecessor weights in the r001 history.
+Use the Transformers example in the Polish section. The artifact uses standard `GPT2LMHeadModel`; `trust_remote_code=True` is unnecessary. No r004 generation samples have been published yet. Historical samples are explicitly attached to the predecessor weights in the r001 history.
 
 ### Limitations and responsible use
 

benchmarks/english.json

@@ -1,11 +1,11 @@
 {
   "schema_version": 1,
-  "revision_id": "r003",
-  "release": "Pollock 1.1",
-  "source_result": "runs/r003-lr4e-4-v4/benchmarks/results.json",
+  "revision_id": "r004",
+  "release": "Pollock 1.2",
+  "source_result": "runs/pollock-r004-shape/benchmarks/results.json",
   "checkpoint": {
-    "path": "runs/r003-lr4e-4-v4/checkpoints/ckpt-benchmark-compatible.pt",
-    "sha256": "0f53aa6b8454e856f8c4c489c7e6cbf207b8e124fe06bbb239315c017fe16fc0",
+    "path": "runs/pollock-r004-shape/checkpoints/ckpt-final.pt",
+    "sha256": "4e38589ca422597ff808bc93cc17ddcc441f734a4e0f671ee6d4527c545972e0",
     "source_iteration": 22004
   },
   "execution": {
@@ -19,37 +19,37 @@
   },
   "results": {
     "blimp": {
-      "acc": 0.7773880597014925,
+      "acc": 0.7720149253731343,
       "samples": 67000
     },
     "lambada_openai": {
-      "acc": 0.289928197166699,
-      "perplexity": 47.70114023017106,
+      "acc": 0.29283912284106345,
+      "perplexity": 47.5633405379111,
       "samples": 5153
     },
     "hellaswag": {
-      "acc_norm": 0.30342561242780325,
-      "acc": 0.28500298745269864,
+      "acc_norm": 0.300637323242382,
+      "acc": 0.2846046604262099,
       "samples": 10042
     },
     "piqa": {
-      "acc_norm": 0.6109902067464635,
-      "acc": 0.6147986942328618,
+      "acc_norm": 0.6077257889009793,
+      "acc": 0.6207834602829162,
       "samples": 1838
     },
     "sciq": {
-      "acc_norm": 0.659,
-      "acc": 0.766,
+      "acc_norm": 0.67,
+      "acc": 0.767,
       "samples": 1000
     },
     "arc_easy": {
-      "acc_norm": 0.42803030303030304,
-      "acc": 0.4861111111111111,
+      "acc_norm": 0.4225589225589226,
+      "acc": 0.4730639730639731,
       "samples": 2376
     },
     "arc_challenge": {
-      "acc_norm": 0.25341296928327645,
-      "acc": 0.22013651877133106,
+      "acc_norm": 0.23464163822525597,
+      "acc": 0.20648464163822525,
       "samples": 1172
     }
   }

config.json

@@ -12,12 +12,12 @@
   "initializer_range": 0.02,
   "layer_norm_epsilon": 1e-05,
   "model_type": "gpt2",
-  "model_version": "1.1",
+  "model_version": "1.2",
   "n_ctx": 1024,
-  "n_embd": 896,
-  "n_head": 14,
+  "n_embd": 832,
+  "n_head": 13,
   "n_inner": null,
-  "n_layer": 12,
+  "n_layer": 14,
   "n_positions": 1024,
   "pad_token_id": 12285,
   "reorder_and_upcast_attn": false,

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3a9964a0ba5f6d20ef6061d2a7e4442c1b001209ebe80f7b7148a25573decaa9
-size 510713512
+oid sha256:3c453f22d4bb70228e5cca79e183a425e0f0221e54bd782d12027c42c943c880
+size 510106688

release_manifest.json

@@ -1,21 +1,21 @@
 {
   "schema_version": 2,
-  "revision": 3,
-  "revision_id": "r003",
-  "release": "Pollock 1.1",
+  "revision": 4,
+  "revision_id": "r004",
+  "release": "Pollock 1.2",
   "model_id": "SlayerLab/pollock-mini-lm-125m",
   "source_checkpoint": {
-    "path_in_training_workspace": "runs/r003-lr4e-4-v4/checkpoints/ckpt.pt",
-    "sha256": "912483b44bb177c7a62fe19a53fc6db7b333cf881c57de15daf9dd7209d90a6b",
+    "path_in_training_workspace": "runs/pollock-r004-shape/checkpoints/ckpt-final.pt",
+    "sha256": "4e38589ca422597ff808bc93cc17ddcc441f734a4e0f671ee6d4527c545972e0",
     "iteration": 22004,
     "tokens_seen": 10815406080,
-    "native_nanogpt_parameters": 126637952,
-    "native_unique_trainable_parameters": 127555456
+    "native_nanogpt_parameters": 126541376,
+    "native_unique_trainable_parameters": 127393344
   },
   "architecture": {
-    "n_layer": 12,
-    "n_head": 14,
-    "n_embd": 896,
+    "n_layer": 14,
+    "n_head": 13,
+    "n_embd": 832,
     "block_size": 1024,
     "vocab_size": 12288,
     "dropout": 0.0,
@@ -27,7 +27,7 @@
     "init_from": "scratch",
     "micro_batch_per_gpu": 12,
     "gradient_accumulation_global": 40,
-    "ddp_world_size": 2,
+    "ddp_world_size": 1,
     "effective_batch_tokens": 491520,
     "optimizer": "fused AdamW",
     "learning_rate": 0.0004,
@@ -43,16 +43,18 @@
     "compile": true,
     "backend": "nccl",
     "seed": 1337,
-    "evaluation_seed": 4242,
-    "hardware": "2x NVIDIA L40S",
+    "evaluation_seed": 1337,
+    "hardware": "1x NVIDIA GeForce RTX 5090",
     "nanogpt_commit": "3adf61e154c3fe3fca428ad6bc3818b27a3b8291"
   },
   "evaluation": {
     "protocol": "fixed sampled subset",
     "subset_tokens_per_split": 1228800,
-    "final_validation_loss": 2.552875518798828,
-    "best_validation_loss": 2.5362,
-    "best_validation_step": 20000,
+    "final_train_loss": 2.4665083193778994,
+    "final_validation_loss": 2.535205955505371,
+    "best_validation_loss": 2.5350550031661987,
+    "best_validation_step": 22000,
+    "final_validation_perplexity": 12.61902953902636,
     "benchmark_harness": "lm-evaluation-harness 0.4.12",
     "benchmark_num_fewshot": 0,
     "benchmark_batch_size": 8,
@@ -62,19 +64,19 @@
   "conversion": {
     "target_class": "GPT2LMHeadModel",
     "transformers_version": "5.15.1",
-    "unique_serialized_parameters": 127674624,
-    "compatibility_zero_bias_parameters": 119168,
+    "unique_serialized_parameters": 127522304,
+    "compatibility_zero_bias_parameters": 128960,
     "validation_probe_shape": [2, 64],
     "max_absolute_logit_error": 0.0
   },
   "artifacts": {
-    "README.md": {"sha256": "5ab395ed94618c311257fed4c16c40ed558919e6b0b0f12177eca4250dc41f5a"},
-    "CHANGELOG.md": {"sha256": "00a60accbe6dfe0734326124c789f842141f61ecc26d7cd2f8291b148bfd125f"},
-    "training-history/r003.md": {"sha256": "4ae07419880227300a5c6c2ae259d7f609d928f33ab9c0c91cbce44dc6b56428"},
-    "benchmarks/english.json": {"sha256": "7dd602e81a79bbdad4393b9031d0f584148076310d85cbda71fb6867b1ad63d8"},
-    "config.json": {"sha256": "89816fc9a57e4915ef884864dfb7b0502dbc16e3da749e781ce41b2a2c7ce26d"},
+    "README.md": {"sha256": "64513cc6cf3cdc52544a02010293c97b58683c815ef952815954b44fa2b15a0f"},
+    "CHANGELOG.md": {"sha256": "5203517a875ca3f04747305a80f732aca54a05ca132a4a4b4fb1453d9d2557ea"},
+    "training-history/r004.md": {"sha256": "d9893a5a9f998cfc93e986d63eb36c9a39f519e1cb74f7a6bf6ac4d3cf6f6f12"},
+    "benchmarks/english.json": {"sha256": "ffe1e417bef440b1e6a84e05beccb96aa191f6f22141b388cc7ed08297318a7d"},
+    "config.json": {"sha256": "296c495f22edddb9828ff9c9ca486cddbacff510742f4aff6a04552d2da3b8d4"},
     "generation_config.json": {"sha256": "435beb27be51f0ed054f4a011e5109d125cdadc118b8799b18b155cc798d94d2"},
-    "model.safetensors": {"sha256": "3a9964a0ba5f6d20ef6061d2a7e4442c1b001209ebe80f7b7148a25573decaa9"},
+    "model.safetensors": {"sha256": "3c453f22d4bb70228e5cca79e183a425e0f0221e54bd782d12027c42c943c880"},
     "special_tokens_map.json": {"sha256": "8b2257a17ea997bb038f43b133aefec82344ad2b8abc2b8a02a6c0a994ed624e"},
     "tokenizer.json": {"sha256": "6cda4e5ec8293f3b821e02253f9c0e88e43ad4f7c6e1324763e1fb91749fef51"},
     "tokenizer_config.json": {"sha256": "4cdabe37dbdc1adfcc017ee9a1f86ab89bdf184827d2d9f05181cae0f8af19bf"}

training-history/r004.md

@@ -0,0 +1,142 @@
+# r004 - Pollock 1.2 training record
+
+## Identity and provenance
+
+| Field | Value |
+|---|---|
+| Revision / release | `r004` / Pollock 1.2 |
+| Model ID | `SlayerLab/pollock-mini-lm-125m` |
+| Source run | `runs/pollock-r004-shape` |
+| Final checkpoint | `checkpoints/ckpt-final.pt` |
+| Final checkpoint update / SHA-256 | 22,004 / `4e38589ca422597ff808bc93cc17ddcc441f734a4e0f671ee6d4527c545972e0` |
+| Best-validation checkpoint | update 22,000 / `181673699af41ebd9e0c2b6517a0a3a0b8eb7111d35fc2884152ca5c4d20815c` |
+| nanoGPT commit | `3adf61e154c3fe3fca428ad6bc3818b27a3b8291` (dirty workspace recorded) |
+| W&B run | [`7aa623ae`](https://wandb.ai/dawidmajewski-ezotic/minimal-en/runs/7aa623ae) |
+| Training environment | [DMPod GPT](https://github.com/dawidmajewski/dmpod-gpt) on RunPod |
+| Run status | completed, exit code 0, target budget completed |
+
+`ckpt-final.pt` and `ckpt-last.pt` are byte-identical and share the SHA-256 shown above. Benchmarks used the final checkpoint.
+
+## Experiment hypothesis
+
+R004 tests whether two additional Transformer blocks and a narrower residual stream improve language modeling at effectively the same parameter budget. R003's 12/14/896 shape is replaced by 14/13/832, retaining a 64-dimensional attention head. The parameter difference is 162,112, approximately 0.127%.
+
+## Architecture and parameter accounting
+
+| Setting | Value |
+|---|---:|
+| Type | GPT-2-style decoder-only Transformer |
+| Layers / heads / width | 14 / 13 / 832 |
+| Head dimension / MLP width | 64 / 3,328 |
+| Context / vocabulary | 1,024 / 12,288 |
+| Activation / dropout / bias | GELU / 0.0 / false |
+| Position representation | learned absolute embeddings |
+| Input/output token weights | tied |
+| Transformer excluding token/position embeddings | 116,317,760 |
+| Tied token embedding/output matrix | 10,223,616 |
+| nanoGPT reported count | 126,541,376 |
+| Learned position embedding | 851,968 |
+| Native unique trainable total | 127,393,344 |
+
+The Transformers artifact has 127,522,304 unique serialized parameters, including 128,960 zero-valued compatibility bias parameters.
+
+## Tokenizer and dataset
+
+The custom byte-level BPE tokenizer and packed dataset binaries are unchanged. Tokenizer SHA-256: `6cda4e5ec8293f3b821e02253f9c0e88e43ad4f7c6e1324763e1fb91749fef51`.
+
+| Split | BPE tokens | Binary SHA-256 |
+|---|---:|---|
+| Train | 2,703,768,815 | `5396834c25664080c14793a4880f86b86ae94074a6036f8f54a58686b02d30dc` |
+| Validation | 5,267,742 | `bf07bd1fb1f956a92ee7788f2a8110e1837f5b880ce432f6cd2a22d05f8ac135` |
+
+The backup intentionally excludes the dataset binaries. Their identities and hashes remain recorded in the run manifest.
+
+## Complete training configuration
+
+| Setting | Value |
+|---|---:|
+| Initialization | scratch |
+| Optimizer updates | 22,004 |
+| Sequence / micro-batch per GPU | 1,024 / 12 |
+| Global / per-GPU accumulation | 40 / 40 micro-steps |
+| GPU count / effective batch | 1 / 491,520 tokens |
+| Token presentations / passes | 10,815,406,080 / 4.000122 |
+| Optimizer / betas | fused AdamW / 0.9, 0.95 |
+| Maximum / minimum LR | 4e-4 / 4e-5 |
+| Schedule / warmup / decay | cosine / 440 / 22,003 updates |
+| Weight decay / grad clip | 0.1 / 1.0 |
+| Evaluation interval / batches | 500 / 100 per split |
+| Evaluation subset | fixed 1,228,800 tokens per split |
+| Training / data / evaluation seeds | 1,337 / 1,337 / 1,337 |
+| Precision / compilation / backend | BF16 / `torch.compile` / NCCL |
+| Maximum checkpoint interval | 15 minutes |
+| Hardware | 1x NVIDIA GeForce RTX 5090 |
+| PyTorch / CUDA | 2.8.0+cu128 / 12.8 |
+| Mean throughput | 191,468.895 tokens/s |
+| Runtime | 16.052 hours |
+
+Dataset, tokenizer, context, effective batch, token budget, optimizer settings, and learning-rate range match r003. Architecture, hardware, evaluation seed/frequency, and checkpoint frequency differ.
+
+## Training-time validation
+
+The table shows the start, warmup end, each 1,000-update point, the recorded minimum, and the final budget evaluation. The raw run evaluated every 500 updates.
+
+| Step | Train loss | Validation loss |
+|---:|---:|---:|
+| 0 | 9.544133 | 9.548397 |
+| 440 | 4.767136 | 4.817189 |
+| 1,000 | 3.415107 | 3.455676 |
+| 2,000 | 3.040967 | 3.082571 |
+| 3,000 | 2.898505 | 2.940333 |
+| 4,000 | 2.816346 | 2.859951 |
+| 5,000 | 2.757871 | 2.803232 |
+| 6,000 | 2.714673 | 2.763016 |
+| 7,000 | 2.682473 | 2.730678 |
+| 8,000 | 2.650815 | 2.701713 |
+| 9,000 | 2.625189 | 2.677834 |
+| 10,000 | 2.603702 | 2.657582 |
+| 11,000 | 2.584919 | 2.639393 |
+| 12,000 | 2.566821 | 2.622251 |
+| 13,000 | 2.548589 | 2.608235 |
+| 14,000 | 2.535984 | 2.594993 |
+| 15,000 | 2.522074 | 2.582463 |
+| 16,000 | 2.510117 | 2.571717 |
+| 17,000 | 2.498652 | 2.562398 |
+| 18,000 | 2.488918 | 2.554044 |
+| 19,000 | 2.480739 | 2.547309 |
+| 20,000 | 2.474230 | 2.541554 |
+| 21,000 | 2.470307 | 2.537608 |
+| 22,000 | 2.466657 | **2.535055** |
+| 22,004 | **2.466508** | 2.535206 |
+
+Final validation perplexity is 12.6190295390. R003 used evaluation seed 4,242 and different offset hashes, so its 2.552876 final value cannot be directly subtracted from this value as a controlled comparison.
+
+## English zero-shot benchmarks
+
+Tasks used complete splits with `lm-evaluation-harness` 0.4.12, zero few-shot examples, batch size 8, BF16, and 1,024 maximum context. As in r003, the adapter recorded 80 requests truncated to model context.
+
+| Benchmark | Primary | Additional | Samples | Delta vs r003 |
+|---|---:|---:|---:|---:|
+| BLiMP | acc 0.7720149254 | - | 67,000 | -0.005373 |
+| LAMBADA OpenAI | acc 0.2928391228 | perplexity 47.5633405379 | 5,153 | +0.002911 acc |
+| HellaSwag | acc_norm 0.3006373232 | acc 0.2846046604 | 10,042 | -0.002788 |
+| PIQA | acc_norm 0.6077257889 | acc 0.6207834603 | 1,838 | -0.003264 |
+| SciQ | acc_norm 0.6700000000 | acc 0.7670000000 | 1,000 | +0.011000 |
+| ARC-Easy | acc_norm 0.4225589226 | acc 0.4730639731 | 2,376 | -0.005471 |
+| ARC-Challenge | acc_norm 0.2346416382 | acc 0.2064846416 | 1,172 | -0.018771 |
+
+The deeper shape improves two primary metrics and reduces five. It does not support a general claim that additional depth is better at this parameter budget. Relative to r002, five primary metrics are higher and two are lower.
+
+Structured English results: [`../benchmarks/english.json`](../benchmarks/english.json).
+
+## Release preparation status
+
+- The Zstandard backup passed integrity testing and was extracted successfully.
+- All 11 run-manifest file hashes and all three checkpoint hashes were verified after extraction.
+- Final training status, target budget, summary metrics, and English benchmark results are present.
+- Transformers 5.15.1 conversion is complete; the `[2, 64]` deterministic parity probe produced a maximum absolute logit error of 0.0, and artifact hashes are recorded in `release_manifest.json`.
+- No files have been uploaded to Hugging Face for this revision.
+
+## Licensing and limitations
+
+This is an English, unaligned base model, not an instruction-following assistant. It may hallucinate, generate harmful or biased content, and reproduce training patterns. It is unsuitable for high-risk or production decisions without independent safeguards. The dataset combines sources with different terms; see the dataset card and the model repository's `LICENSE.md`.