Document LUMI-verified SFT recipe and token mixture

README.md

@@ -11,6 +11,10 @@ tags:
 - multilingual
 - instruction-tuning
 - sft
+datasets:
+- allenai/tulu-3-sft-mixture
+- utter-project/EuroBlocks-SFT-Synthetic-1124
+- nvidia/Nemotron-Post-Training-Dataset-v2
 language: [en, sv, de, fr, es, it, nl, pl, pt, cs, fi, da, el, bg, hr, hu, ro, sk, sl, et, lt, lv, ga, mt, eu, gl, is, nb, nn, sr, uk, ca, mk, sq, oc, lb, bs]
 ---
 
@@ -71,19 +75,29 @@ Long-context inference, especially near 256K, requires multiple GPUs for the KV
 | Field | Value |
 |---|---|
 | Base | `openeurollm/oellm-9b-256k-theta64m-prelude` |
-| Method | Full-parameter supervised fine-tuning with packed sequences and FlashAttention 2 |
+| Training library | Hugging Face TRL 0.28.0, using `SFTTrainer` and `SFTConfig` through a text-only wrapper |
+| Method | Full-parameter BF16 supervised fine-tuning; no LoRA or other PEFT adapter |
+| Distributed stack | Accelerate FSDP `FULL_SHARD`, FlashAttention 2, gradient checkpointing |
+| Framework versions | Transformers 5.12.1; PyTorch 2.9.1+ROCm 6.4; Datasets 5.0.0; Tokenizers 0.22.2 |
 | Data | 920,552 Tulu-3 rows + 161,644 EuroBlocks rows + a 250,000-row Nemotron-v2 math prefix |
 | Loaded examples | 1,332,196 source conversations before tokenization and packing |
-| Sequence length | 4,096 |
+| Tokenized corpus | 2,593,634,710 tokens before training-time sampling |
+| Sequence construction | Maximum length 4,096; best-fit-decreasing packing; 635,974 packed sequences at 99.57% mean fill |
+| Loss | Standard causal NLL over the full rendered conversation; `assistant_only_loss=false` |
 | Steps | 3,000 (about 0.604 epoch) |
-| Global batch | 128 |
-| Optimizer schedule | AdamW, peak LR 6e-6, cosine decay, 3% warmup |
+| Global batch | 128 packed sequences (32 ranks × batch 1 × gradient accumulation 4) |
+| Optimizer schedule | Fused AdamW, peak LR 6e-6, cosine decay, 3% warmup, weight decay 0, gradient clipping 1.0 |
+| Randomness | Random train sampler, seed 42; no held-out validation split |
 | Hardware | 4 LUMI-G nodes, 32 AMD MI250X GCDs |
 | Runtime | 14h 36m |
-| Final training loss | 0.7671 |
-| Final mean token accuracy | 0.7839 |
+| Tokens processed | 1,566,026,219 |
+| Training loss | 0.8034 aggregate; 0.7671 in the final 20-step logging window |
+| Final logged mean token accuracy | 0.7839 |
 
 The published weights are an unquantized BF16 export of the consolidated training checkpoint.
+The framework versions above come from the trainer-generated README stored with the completed
+checkpoint; the remaining values were checked against the submitted YAML, serialized `SFTConfig`,
+Slurm accounting, trainer state, logs, and cached Arrow datasets for LUMI job `20290451`.
 
 ## Training-data composition
 
@@ -105,6 +119,24 @@ The first two slices form the 1,082,196-row *general* Parquet. Within that Parqu
 `tulu3-euroblocks-85-15`. Adding the reasoning slice changes the overall ratio to the one shown
 above. These are row shares, not token shares.
 
+### Tokenized mixture
+
+Row shares substantially understate the contribution of the long Nemotron reasoning traces. The
+exact token counts below were measured from the tokenized Arrow cache produced by the training job,
+before best-fit-decreasing packing and random training-time sampling:
+
+| Effective training slice | Rows | Row share | Tokens | Token share | Mean tokens/row |
+|---|---:|---:|---:|---:|---:|
+| Tulu-3 commercial subset | 920,552 | 69.10% | 841,791,947 | 32.46% | 914 |
+| EuroBlocks subset | 161,644 | 12.13% | 81,355,816 | 3.14% | 503 |
+| Nemotron-v2 math prefix | 250,000 | 18.77% | 1,670,486,947 | 64.41% | 6,682 |
+| **Total** | **1,332,196** | **100%** | **2,593,634,710** | **100%** | **1,947** |
+
+Packing preserved all 2,593,634,710 tokens in 635,974 sequences. The run sampled 384,000 packed
+sequences over 3,000 optimizer steps and processed 1,566,026,219 tokens, corresponding to 0.6038
+epoch. Because source identities are not retained after cross-example packing, the table describes
+the exact tokenized pool rather than an exact per-source count of tokens drawn by the random sampler.
+
 ### Tulu-3 subset: exact source counts
 
 The run did not load the current public Tulu-3 mixture wholesale. It used a locally staged file
@@ -138,33 +170,15 @@ file. The staging label `commercial` describes the selected artifact; users shou
 Tulu collection license and every component's terms. The file has no normalized language column, so
 we do not infer exact per-language counts for this slice from prompt text.
 
-### EuroBlocks subset: language labels
-
-All 161,644 rows identify `EuroBlocks-SFT-Synthetic-1124` as their source. Their IDs run from
-`EuroBlocks-SFT-Synthetic-1124_1` through `_161644`; this is smaller than the 340,286-row train split
-described by the public EuroBlocks card. The staged rows contain the following language labels:
-
-| Language label | Rows | Language label | Rows |
-|---|---:|---|---:|
-| Unlabelled | 23,318 | Spanish | 17,428 |
-| Italian | 15,963 | French | 14,882 |
-| German | 14,081 | Portuguese | 13,966 |
-| Hindi | 7,982 | Dutch | 7,683 |
-| Swedish | 6,476 | Polish | 5,358 |
-| Ukrainian | 5,191 | Japanese | 4,735 |
-| Russian | 4,727 | Czech | 4,105 |
-| Hungarian | 4,010 | Romanian | 3,993 |
-| Korean | 2,905 | Finnish | 1,022 |
-| Slovak | 990 | Portuguese (Portugal) | 672 |
-| Greek | 582 | Norwegian | 534 |
-| Catalan | 223 | Bulgarian | 210 |
-| Slovenian | 201 | Estonian | 179 |
-| Latvian | 176 | Croatian | 30 |
-| Icelandic | 18 | Danish | 4 |
-
-The 23,318 unlabelled rows are reported as such rather than assigned a language after the fact.
-Spot checks found English examples among them, but no language-ID pass was run for this audit.
-EuroBlocks is synthetic and its dataset card documents mixed upstream sources and terms.
+### EuroBlocks subset: retained provenance
+
+The 161,644 effective rows have synthetic IDs from `EuroBlocks-SFT-Synthetic-1124_1` through
+`_161644`. This is smaller than the 340,286-line EuroBlocks staging file present on LUMI. The final
+training Parquet retains only `{id, messages}` and the source staging JSONL retains only `messages`;
+neither artifact records a normalized language label or the sampling/filtering mapping from the
+340,286-row source to the 161,644-row effective subset. Exact per-language counts therefore cannot
+be reconstructed from the frozen training artifacts without running a new language-identification
+pass. EuroBlocks is synthetic and its dataset card documents mixed upstream sources and terms.
 
 ### Reasoning slice: what was actually selected
 
@@ -185,8 +199,14 @@ row count is reported here because it is the file actually used by this run.
 
 - General rows were already stored as `{id, messages}`; reasoning rows were converted to
   `{messages}` by appending the `output` as an assistant turn.
-- Conversations were rendered with the tokenizer's included chat template and
-  `<end_of_turn>` EOS, then packed into sequences with a maximum length of 4,096 tokens.
+- Conversations were rendered with the tokenizer's included `<start_of_turn>` / `<end_of_turn>`
+  chat template, then packed with TRL's best-fit-decreasing strategy into sequences with a maximum
+  length of 4,096 tokens.
+- The serialized `SFTConfig` records `assistant_only_loss=false`, `completion_only_loss=null`, and
+  NLL loss. For these conversational `messages` rows, training loss therefore covered both user and
+  assistant tokens rather than masking to assistant responses only.
+- The 1,332,196 conversations became 1,663,938 sequence segments before/within packing because
+  records longer than 4,096 tokens were split. Of the packed rows, 436,108 were exactly 4,096 tokens.
 - This stage did **not** train on 256K-token sequences. The architectural 262,144-token setting
   comes from the base checkpoint and was evaluated after short-sequence SFT.
 - The SFT recipe records no additional post-concatenation deduplication or benchmark-
@@ -195,6 +215,19 @@ row count is reported here because it is the file actually used by this run.
 - Only a training split was loaded; no held-out validation split from these sources was used during
   optimization.
 
+### Data-quality observations from the frozen run
+
+- The Nemotron math slice was selected by prefix, not randomized. It consequently supplied 64.41%
+  of the tokenized corpus despite representing only 18.77% of source rows, while EuroBlocks supplied
+  3.14% of tokens.
+- Tulu row `ai2-adapt-dev/tulu_v3.9_aya_100k_21109` tokenized to 3,025,913 tokens. It contains a
+  roughly 3.3-million-character Somali dictionary duplicated verbatim as both the user and assistant
+  message; packing split this single outlier across hundreds of 4K segments.
+- The Tulu subset includes 236 rows from `ai2-adapt-dev/tulu_hard_coded_repeated_10`, an explicitly
+  repeated source component.
+- These observations describe the released checkpoint's actual training input. They were not fixed
+  retroactively, because doing so would describe a different run.
+
 Training-data terms are not replaced by the model repository's Apache-2.0 weight license. Tulu-3
 is an ODC-BY-1.0 collection with component-specific terms; EuroBlocks documents mixed upstream
 terms; and Nemotron-v2 documents per-sample licenses plus possible model-output terms. Review those
@@ -421,8 +454,9 @@ Raw reports: [word passphrases, 512-8K](./eval/niah-word-short.json),
 
 ## Secondary general behavior
 
-This model received a limited English math-reasoning slice but is not presented as a
-reasoning-specialized checkpoint. On a 1,368-example multilingual development suite it scored 23.2%
+Although English math-reasoning rows were only 18.77% of examples, their long traces made up 64.41%
+of the tokenized corpus. The resulting checkpoint is nevertheless not presented as a
+reasoning-specialized model. On a 1,368-example multilingual development suite it scored 23.2%
 overall, including 63.2% instruction following, 58.6% grounded QA, and 5.9% reasoning/math. A
 24-prompt Swedish generation probe had mean language-detection probability 0.96, but manual review
 found factual, arithmetic, grammar, and exact-constraint errors. These results are included to prevent
@@ -448,8 +482,11 @@ of both successful and incorrect responses.
 
 ## Reproduction
 
-Training and evaluation code: <https://github.com/BirgerMoell/qwen35-posttrain>, especially
-`docs/RUNBOOK_oellm9b_256k_posttrain.md`.
+Training and evaluation code: <https://github.com/BirgerMoell/qwen35-posttrain>. The most relevant
+files are [`scripts/sft_train.py`](https://github.com/BirgerMoell/qwen35-posttrain/blob/main/scripts/sft_train.py),
+[`configs/sft_oellm9b_256k.yaml`](https://github.com/BirgerMoell/qwen35-posttrain/blob/main/configs/sft_oellm9b_256k.yaml),
+[`configs/accelerate_fsdp.yaml`](https://github.com/BirgerMoell/qwen35-posttrain/blob/main/configs/accelerate_fsdp.yaml),
+and [`docs/RUNBOOK_oellm9b_256k_posttrain.md`](https://github.com/BirgerMoell/qwen35-posttrain/blob/main/docs/RUNBOOK_oellm9b_256k_posttrain.md).
 
 ## License