Training log (success)

logs/training_20260113_061956.log

@@ -0,0 +1,624 @@
+/venv/main/lib/python3.12/site-packages/trl/__init__.py:203: UserWarning: TRL currently supports vLLM versions: 0.10.2, 0.11.0, 0.11.1, 0.11.2. You have version 0.13.0 installed. We recommend installing a supported version to avoid compatibility issues.
+  if is_vllm_available():
+Loading config from: configs/runtime_config.yaml
+Random seed set to: 42
+============================================================
+Training Configuration:
+  Framework: accelerate
+  Method: sft
+============================================================
+W&B Project: aimo3-exp-1-train
+W&B Run: qwen3-0.6b-sft-lora-20260113_061956-001
+wandb: Currently logged in as: vinoku (vinokuteam) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
+wandb: WARNING Using a boolean value for 'reinit' is deprecated. Use 'return_previous' or 'finish_previous' instead.
+wandb: setting up run 3fkbe45r
+wandb: Tracking run with wandb version 0.23.1
+wandb: Run data is saved locally in /workspace/distributed-sft-rl-inference-pipeline/wandb/run-20260112_222013-3fkbe45r
+wandb: Run `wandb offline` to turn off syncing.
+wandb: Syncing run qwen3-0.6b-sft-lora-20260113_061956-001
+wandb: ⭐️ View project at https://wandb.ai/vinokuteam/aimo3-exp-1-train
+wandb: 🚀 View run at https://wandb.ai/vinokuteam/aimo3-exp-1-train/runs/3fkbe45r
+wandb: Detected [openai] in use.
+wandb: Use W&B Weave for improved LLM call tracing. Install Weave with `pip install weave` then add `import weave` to the top of your script.
+wandb: For more information, check out the docs at: https://weave-docs.wandb.ai/
+W&B Run: qwen3-0.6b-sft-lora-20260113_061956-001
+W&B URL: https://wandb.ai/vinokuteam/aimo3-exp-1-train/runs/3fkbe45r
+W&B logging enabled (ID: 3fkbe45r)
+Loading model: Qwen/Qwen3-0.6B
+ZeRO-0 mode (no HfDeepSpeedConfig needed)
+`torch_dtype` is deprecated! Use `dtype` instead!
+Building datasets...
+Loading dataset from HuggingFace: HAD653/GSM8K-OpenMath-MathReason-13k
+Train size: 13857, Test size: 416
+Dataset size after dropping NaN: 13857
+Dataset size after dropping NaN: 416
+Final train size: 13857, test size: 416
+
+============================================================
+SAMPLE TRAINING DATA (first example):
+============================================================
+<|im_start|>system
+You are a helpful and reflective maths assistant. Please reason step by step to put the answer in \boxed{}.<|im_end|>
+<|im_start|>user
+Albert is wondering how much pizza he can eat in one day. He buys 2 large pizzas and 2 small pizzas. A large pizza has 16 slices and a small pizza has 8 slices. If he eats it all, how many pieces does he eat that day?<|im_end|>
+<|im_start|>assistant
+<think>
+Problem:
+Albert buys 2 large pizzas (16 slices each) and 2 small pizzas (8 slices each). Find total slices eaten.
+
+Reasoning:
+1. Large pizza slices = 2*16 = 32.
+2. Small pizza slices = 2*8 = 16.
+3. Total slices = 32+16 = 48.
+
+Answer:
+48
+</think>
+
+\boxed{48}<|im_end|>
+============================================================
+
+Creating LoRA configuration...
+Using Accelerate + SFT training pipeline
+Initializing SFTTrainer...
+
+Adding EOS to train dataset:   0%|          | 0/13857 [00:00<?, ? examples/s]
+Adding EOS to train dataset:  30%|██▉       | 4112/13857 [00:00<00:00, 39716.06 examples/s]
+Adding EOS to train dataset:  60%|██████    | 8353/13857 [00:00<00:00, 41271.86 examples/s]
+Adding EOS to train dataset:  90%|█████████ | 12523/13857 [00:00<00:00, 41465.24 examples/s]
+Adding EOS to train dataset: 100%|██████████| 13857/13857 [00:00<00:00, 41056.84 examples/s]
+
+Tokenizing train dataset:   0%|          | 0/13857 [00:00<?, ? examples/s]
+Tokenizing train dataset:   1%|▏         | 187/13857 [00:00<00:07, 1852.55 examples/s]
+Tokenizing train dataset:   3%|▎         | 378/13857 [00:00<00:07, 1877.99 examples/s]
+Tokenizing train dataset:   4%|▍         | 573/13857 [00:00<00:06, 1902.48 examples/s]
+Tokenizing train dataset:   6%|▌         | 772/13857 [00:00<00:06, 1933.09 examples/s]
+Tokenizing train dataset:   7%|▋         | 977/13857 [00:00<00:06, 1971.70 examples/s]
+Tokenizing train dataset:   9%|▊         | 1208/13857 [00:00<00:07, 1780.65 examples/s]
+Tokenizing train dataset:  10%|█         | 1402/13857 [00:00<00:06, 1822.62 examples/s]
+Tokenizing train dataset:  12%|█▏        | 1598/13857 [00:00<00:06, 1859.51 examples/s]
+Tokenizing train dataset:  13%|█▎        | 1801/13857 [00:00<00:06, 1905.53 examples/s]
+Tokenizing train dataset:  14%|█▍        | 1999/13857 [00:01<00:06, 1925.89 examples/s]
+Tokenizing train dataset:  16%|█▌        | 2232/13857 [00:01<00:06, 1779.53 examples/s]
+Tokenizing train dataset:  18%|█▊        | 2439/13857 [00:01<00:06, 1854.49 examples/s]
+Tokenizing train dataset:  19%|█▉        | 2643/13857 [00:01<00:05, 1902.63 examples/s]
+Tokenizing train dataset:  21%|██        | 2850/13857 [00:01<00:05, 1946.52 examples/s]
+Tokenizing train dataset:  22%|██▏       | 3093/13857 [00:01<00:06, 1793.63 examples/s]
+Tokenizing train dataset:  24%|██▎       | 3279/13857 [00:01<00:05, 1809.71 examples/s]
+Tokenizing train dataset:  25%|██▌       | 3471/13857 [00:01<00:05, 1837.39 examples/s]
+Tokenizing train dataset:  26%|██▋       | 3663/13857 [00:01<00:05, 1856.62 examples/s]
+Tokenizing train dataset:  28%|██▊       | 3932/13857 [00:02<00:05, 1830.47 examples/s]
+Tokenizing train dataset:  30%|███       | 4196/13857 [00:02<00:05, 1729.06 examples/s]
+Tokenizing train dataset:  32%|███▏      | 4392/13857 [00:02<00:05, 1781.85 examples/s]
+Tokenizing train dataset:  33%|███▎      | 4595/13857 [00:02<00:05, 1842.05 examples/s]
+Tokenizing train dataset:  35%|███▍      | 4798/13857 [00:02<00:04, 1888.49 examples/s]
+Tokenizing train dataset:  36%|███▌      | 5000/13857 [00:02<00:05, 1750.89 examples/s]
+Tokenizing train dataset:  38%|███▊      | 5208/13857 [00:02<00:04, 1835.53 examples/s]
+Tokenizing train dataset:  39%|███▉      | 5415/13857 [00:02<00:04, 1896.83 examples/s]
+Tokenizing train dataset:  41%|████      | 5625/13857 [00:03<00:04, 1951.48 examples/s]
+Tokenizing train dataset:  42%|████▏     | 5830/13857 [00:03<00:04, 1977.78 examples/s]
+Tokenizing train dataset:  44%|████▍     | 6102/13857 [00:03<00:04, 1852.31 examples/s]
+Tokenizing train dataset:  46%|████▌     | 6307/13857 [00:03<00:03, 1901.90 examples/s]
+Tokenizing train dataset:  48%|████▊     | 6601/13857 [00:03<00:03, 1919.99 examples/s]
+Tokenizing train dataset:  49%|████▉     | 6803/13857 [00:03<00:03, 1942.15 examples/s]
+Tokenizing train dataset:  51%|█████     | 7000/13857 [00:03<00:03, 1805.41 examples/s]
+Tokenizing train dataset:  52%|█████▏    | 7206/13857 [00:03<00:03, 1870.01 examples/s]
+Tokenizing train dataset:  54%|█████▍    | 7451/13857 [00:04<00:03, 1783.37 examples/s]
+Tokenizing train dataset:  55%|█████▌    | 7683/13857 [00:04<00:03, 1700.33 examples/s]
+Tokenizing train dataset:  57%|█████▋    | 7926/13857 [00:04<00:03, 1668.62 examples/s]
+Tokenizing train dataset:  59%|█████▉    | 8145/13857 [00:04<00:03, 1481.19 examples/s]
+Tokenizing train dataset:  60%|██████    | 8359/13857 [00:04<00:03, 1457.07 examples/s]
+Tokenizing train dataset:  61%|██████▏   | 8511/13857 [00:04<00:03, 1469.17 examples/s]
+Tokenizing train dataset:  63%|██████▎   | 8666/13857 [00:04<00:03, 1486.34 examples/s]
+Tokenizing train dataset:  64%|██████▍   | 8897/13857 [00:05<00:03, 1500.20 examples/s]
+Tokenizing train dataset:  66%|██████▌   | 9080/13857 [00:05<00:03, 1382.65 examples/s]
+Tokenizing train dataset:  67%|██████▋   | 9241/13857 [00:05<00:03, 1432.59 examples/s]
+Tokenizing train dataset:  68%|██████▊   | 9404/13857 [00:05<00:03, 1478.86 examples/s]
+Tokenizing train dataset:  69%|██████▉   | 9558/13857 [00:05<00:02, 1493.16 examples/s]
+Tokenizing train dataset:  70%|███████   | 9721/13857 [00:05<00:02, 1527.97 examples/s]
+Tokenizing train dataset:  71%|███████▏  | 9881/13857 [00:05<00:02, 1544.61 examples/s]
+Tokenizing train dataset:  73%|███████▎  | 10079/13857 [00:05<00:02, 1331.48 examples/s]
+Tokenizing train dataset:  74%|███████▍  | 10234/13857 [00:05<00:02, 1379.87 examples/s]
+Tokenizing train dataset:  75%|███████▌  | 10401/13857 [00:06<00:02, 1453.78 examples/s]
+Tokenizing train dataset:  76%|███████▌  | 10559/13857 [00:06<00:02, 1484.99 examples/s]
+Tokenizing train dataset:  77%|███████▋  | 10719/13857 [00:06<00:02, 1515.15 examples/s]
+Tokenizing train dataset:  79%|███████▊  | 10882/13857 [00:06<00:01, 1543.14 examples/s]
+Tokenizing train dataset:  80%|███████▉  | 11074/13857 [00:06<00:02, 1366.29 examples/s]
+Tokenizing train dataset:  81%|████████  | 11246/13857 [00:06<00:01, 1454.07 examples/s]
+Tokenizing train dataset:  82%|████████▏ | 11402/13857 [00:06<00:01, 1480.89 examples/s]
+Tokenizing train dataset:  84%|████████▍ | 11631/13857 [00:06<00:01, 1482.48 examples/s]
+Tokenizing train dataset:  85%|████████▌ | 11793/13857 [00:07<00:01, 1512.29 examples/s]
+Tokenizing train dataset:  86%|████████▌ | 11951/13857 [00:07<00:01, 1528.71 examples/s]
+Tokenizing train dataset:  88%|████████▊ | 12164/13857 [00:07<00:01, 1403.81 examples/s]
+Tokenizing train dataset:  89%|████████▉ | 12318/13857 [00:07<00:01, 1434.06 examples/s]
+Tokenizing train dataset:  90%|████████▉ | 12470/13857 [00:07<00:00, 1453.17 examples/s]
+Tokenizing train dataset:  91%|█████████ | 12628/13857 [00:07<00:00, 1486.39 examples/s]
+Tokenizing train dataset:  92%|█████████▏| 12783/13857 [00:07<00:00, 1500.55 examples/s]
+Tokenizing train dataset:  94%|█████████▍| 13000/13857 [00:07<00:00, 1349.52 examples/s]
+Tokenizing train dataset:  95%|█████████▍| 13162/13857 [00:07<00:00, 1413.09 examples/s]
+Tokenizing train dataset:  96%|█████████▌| 13322/13857 [00:08<00:00, 1458.65 examples/s]
+Tokenizing train dataset:  97%|█████████▋| 13479/13857 [00:08<00:00, 1488.47 examples/s]
+Tokenizing train dataset:  98%|█████████▊| 13645/13857 [00:08<00:00, 1532.14 examples/s]
+Tokenizing train dataset: 100%|██████████| 13857/13857 [00:08<00:00, 1397.86 examples/s]
+Tokenizing train dataset: 100%|██████████| 13857/13857 [00:08<00:00, 1639.61 examples/s]
+
+Truncating train dataset:   0%|          | 0/13857 [00:00<?, ? examples/s]
+Truncating train dataset: 100%|██████████| 13857/13857 [00:00<00:00, 437489.43 examples/s]
+
+Adding EOS to eval dataset:   0%|          | 0/416 [00:00<?, ? examples/s]
+Adding EOS to eval dataset: 100%|██████████| 416/416 [00:00<00:00, 37833.22 examples/s]
+
+Tokenizing eval dataset:   0%|          | 0/416 [00:00<?, ? examples/s]
+Tokenizing eval dataset:  44%|████▍     | 184/416 [00:00<00:00, 1813.09 examples/s]
+Tokenizing eval dataset:  89%|████████▉ | 371/416 [00:00<00:00, 1839.65 examples/s]
+Tokenizing eval dataset: 100%|██████████| 416/416 [00:00<00:00, 1703.92 examples/s]
+
+Truncating eval dataset:   0%|          | 0/416 [00:00<?, ? examples/s]
+Truncating eval dataset: 100%|██████████| 416/416 [00:00<00:00, 211034.16 examples/s]
+Starting training...
+The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}.
+[2026-01-12 22:20:47] WARNING accelerator.py:2164: Gradient accumulation steps mismatch: GradientAccumulationPlugin has 1, DeepSpeed config has 4. Using DeepSpeed's value.
+
+  0%|          | 0/10 [00:00<?, ?it/s]
+ 10%|█         | 1/10 [00:01<00:14,  1.65s/it]
+ 20%|██        | 2/10 [00:02<00:10,  1.31s/it]
+ 30%|███       | 3/10 [00:03<00:08,  1.21s/it]
+ 40%|████      | 4/10 [00:04<00:06,  1.17s/it]
+ 50%|█████     | 5/10 [00:06<00:05,  1.14s/it]
+ 60%|██████    | 6/10 [00:07<00:04,  1.12s/it]
+ 70%|███████   | 7/10 [00:08<00:03,  1.11s/it]
+ 80%|████████  | 8/10 [00:09<00:02,  1.10s/it]
+ 90%|█████████ | 9/10 [00:10<00:01,  1.10s/it]
+100%|██████████| 10/10 [00:11<00:00,  1.09s/it]
+                                               
+{'loss': 2.0751, 'grad_norm': 0.1237564780666222, 'learning_rate': 4.000000000000001e-06, 'entropy': 0.94697265625, 'num_tokens': 21152.0, 'mean_token_accuracy': 0.66548752784729, 'epoch': 0.01}
+
+100%|██████████| 10/10 [00:11<00:00,  1.09s/it]
+
+  0%|          | 0/208 [00:00<?, ?it/s]
+
+  1%|▏         | 3/208 [00:00<00:09, 22.64it/s]
+
+  3%|▎         | 6/208 [00:00<00:11, 17.45it/s]
+
+  4%|▍         | 8/208 [00:00<00:11, 16.70it/s]
+
+  5%|▍         | 10/208 [00:00<00:12, 15.79it/s]
+
+  6%|▌         | 12/208 [00:00<00:13, 15.02it/s]
+
+  7%|▋         | 14/208 [00:00<00:13, 14.57it/s]
+
+  8%|▊         | 16/208 [00:01<00:13, 14.26it/s]
+
+  9%|▊         | 18/208 [00:01<00:13, 14.28it/s]
+
+ 10%|▉         | 20/208 [00:01<00:13, 14.30it/s]
+
+ 11%|█         | 22/208 [00:01<00:12, 14.31it/s]
+
+ 12%|█▏        | 24/208 [00:01<00:12, 14.32it/s]
+
+ 12%|█▎        | 26/208 [00:01<00:12, 14.09it/s]
+
+ 13%|█▎        | 28/208 [00:01<00:12, 14.11it/s]
+
+ 14%|█▍        | 30/208 [00:02<00:12, 13.94it/s]
+
+ 15%|█▌        | 32/208 [00:02<00:12, 13.75it/s]
+
+ 16%|█▋        | 34/208 [00:02<00:12, 13.85it/s]
+
+ 17%|█▋        | 36/208 [00:02<00:12, 13.90it/s]
+
+ 18%|█▊        | 38/208 [00:02<00:12, 13.82it/s]
+
+ 19%|█▉        | 40/208 [00:02<00:12, 13.94it/s]
+
+ 20%|██        | 42/208 [00:02<00:11, 13.94it/s]
+
+ 21%|██        | 44/208 [00:03<00:11, 13.95it/s]
+
+ 22%|██▏       | 46/208 [00:03<00:11, 14.14it/s]
+
+ 23%|██▎       | 48/208 [00:03<00:11, 13.99it/s]
+
+ 24%|██▍       | 50/208 [00:03<00:11, 14.01it/s]
+
+ 25%|██▌       | 52/208 [00:03<00:11, 13.84it/s]
+
+ 26%|██▌       | 54/208 [00:03<00:11, 13.98it/s]
+
+ 27%|██▋       | 56/208 [00:03<00:10, 13.95it/s]
+
+ 28%|██▊       | 58/208 [00:04<00:10, 14.36it/s]
+
+ 29%|██▉       | 60/208 [00:04<00:10, 14.56it/s]
+
+ 30%|██▉       | 62/208 [00:04<00:09, 15.04it/s]
+
+ 31%|███       | 64/208 [00:04<00:09, 15.22it/s]
+
+ 32%|███▏      | 66/208 [00:04<00:09, 15.40it/s]
+
+ 33%|███▎      | 68/208 [00:04<00:09, 15.34it/s]
+
+ 34%|███▎      | 70/208 [00:04<00:08, 15.40it/s]
+
+ 35%|███▍      | 72/208 [00:04<00:08, 15.34it/s]
+
+ 36%|███▌      | 74/208 [00:05<00:08, 15.54it/s]
+
+ 37%|███▋      | 76/208 [00:05<00:08, 15.59it/s]
+
+ 38%|███▊      | 78/208 [00:05<00:08, 15.54it/s]
+
+ 38%|███▊      | 80/208 [00:05<00:08, 15.50it/s]
+
+ 39%|███▉      | 82/208 [00:05<00:08, 15.50it/s]
+
+ 40%|████      | 84/208 [00:05<00:07, 15.54it/s]
+
+ 41%|████▏     | 86/208 [00:05<00:07, 15.72it/s]
+
+ 42%|████▏     | 88/208 [00:05<00:07, 15.67it/s]
+
+ 43%|████▎     | 90/208 [00:06<00:07, 15.47it/s]
+
+ 44%|████▍     | 92/208 [00:06<00:07, 15.43it/s]
+
+ 45%|████▌     | 94/208 [00:06<00:07, 15.54it/s]
+
+ 46%|████▌     | 96/208 [00:06<00:07, 15.28it/s]
+
+ 47%|████▋     | 98/208 [00:06<00:07, 15.26it/s]
+
+ 48%|████▊     | 100/208 [00:06<00:07, 15.30it/s]
+
+ 49%|████▉     | 102/208 [00:06<00:06, 15.40it/s]
+
+ 50%|█████     | 104/208 [00:06<00:06, 15.42it/s]
+
+ 51%|█████     | 106/208 [00:07<00:06, 15.34it/s]
+
+ 52%|█████▏    | 108/208 [00:07<00:06, 15.20it/s]
+
+ 53%|█████▎    | 110/208 [00:07<00:06, 15.37it/s]
+
+ 54%|█████▍    | 112/208 [00:07<00:06, 15.37it/s]
+
+ 55%|█████▍    | 114/208 [00:07<00:06, 15.26it/s]
+
+ 56%|█████▌    | 116/208 [00:07<00:06, 15.31it/s]
+
+ 57%|█████▋    | 118/208 [00:07<00:05, 15.36it/s]
+
+ 58%|█████▊    | 120/208 [00:08<00:05, 15.59it/s]
+
+ 59%|█████▊    | 122/208 [00:08<00:05, 15.74it/s]
+
+ 60%|█████▉    | 124/208 [00:08<00:05, 15.95it/s]
+
+ 61%|██████    | 126/208 [00:08<00:05, 15.91it/s]
+
+ 62%|██████▏   | 128/208 [00:08<00:05, 15.92it/s]
+
+ 62%|██████▎   | 130/208 [00:08<00:04, 15.84it/s]
+
+ 63%|██████▎   | 132/208 [00:08<00:04, 15.69it/s]
+
+ 64%|██████▍   | 134/208 [00:08<00:04, 15.77it/s]
+
+ 65%|██████▌   | 136/208 [00:09<00:04, 15.69it/s]
+
+ 66%|██████▋   | 138/208 [00:09<00:04, 15.52it/s]
+
+ 67%|██████▋   | 140/208 [00:09<00:04, 15.59it/s]
+
+ 68%|██████▊   | 142/208 [00:09<00:04, 15.64it/s]
+
+ 69%|██████▉   | 144/208 [00:09<00:04, 15.57it/s]
+
+ 70%|███████   | 146/208 [00:09<00:03, 15.62it/s]
+
+ 71%|███████   | 148/208 [00:09<00:03, 15.52it/s]
+
+ 72%|███████▏  | 150/208 [00:09<00:03, 15.50it/s]
+
+ 73%|███████▎  | 152/208 [00:10<00:03, 15.43it/s]
+
+ 74%|███████▍  | 154/208 [00:10<00:03, 15.49it/s]
+
+ 75%|███████▌  | 156/208 [00:10<00:03, 15.32it/s]
+
+ 76%|███████▌  | 158/208 [00:10<00:03, 15.13it/s]
+
+ 77%|███████▋  | 160/208 [00:10<00:03, 15.22it/s]
+
+ 78%|███████▊  | 162/208 [00:10<00:02, 15.40it/s]
+
+ 79%|███████▉  | 164/208 [00:10<00:02, 15.52it/s]
+
+ 80%|███████▉  | 166/208 [00:10<00:02, 15.65it/s]
+
+ 81%|████████  | 168/208 [00:11<00:02, 15.87it/s]
+
+ 82%|████████▏ | 170/208 [00:11<00:02, 15.81it/s]
+
+ 83%|████████▎ | 172/208 [00:11<00:02, 15.72it/s]
+
+ 84%|████████▎ | 174/208 [00:11<00:02, 15.80it/s]
+
+ 85%|████████▍ | 176/208 [00:11<00:02, 15.86it/s]
+
+ 86%|████████▌ | 178/208 [00:11<00:01, 15.57it/s]
+
+ 87%|████████▋ | 180/208 [00:11<00:01, 15.54it/s]
+
+ 88%|████████▊ | 182/208 [00:12<00:01, 15.37it/s]
+
+ 88%|████████▊ | 184/208 [00:12<00:01, 15.29it/s]
+
+ 89%|████████▉ | 186/208 [00:12<00:01, 15.37it/s]
+
+ 90%|█████████ | 188/208 [00:12<00:01, 15.36it/s]
+
+ 91%|█████████▏| 190/208 [00:12<00:01, 15.28it/s]
+
+ 92%|█████████▏| 192/208 [00:12<00:01, 15.38it/s]
+
+ 93%|█████████▎| 194/208 [00:12<00:00, 15.66it/s]
+
+ 94%|█████████▍| 196/208 [00:12<00:00, 15.59it/s]
+
+ 95%|█████████▌| 198/208 [00:13<00:00, 15.06it/s]
+
+ 96%|█████████▌| 200/208 [00:13<00:00, 15.14it/s]
+
+ 97%|█████████▋| 202/208 [00:13<00:00, 15.13it/s]
+
+ 98%|█████████▊| 204/208 [00:13<00:00, 14.80it/s]
+
+ 99%|█████████▉| 206/208 [00:13<00:00, 14.49it/s]
+
+100%|██████████| 208/208 [00:13<00:00, 14.51it/s]
+                                               
+
+                                                 
+{'eval_loss': 2.0517611503601074, 'eval_runtime': 13.824, 'eval_samples_per_second': 30.093, 'eval_steps_per_second': 15.046, 'eval_entropy': 0.9272273137019231, 'eval_num_tokens': 21152.0, 'eval_mean_token_accuracy': 0.6686571506926646, 'epoch': 0.01}
+
+100%|██████████| 10/10 [00:25<00:00,  1.09s/it]
+
+100%|██████████| 208/208 [00:13<00:00, 14.51it/s]
+
+                                                 
+                                               
+{'train_runtime': 27.5551, 'train_samples_per_second': 2.903, 'train_steps_per_second': 0.363, 'train_loss': 2.0750574111938476, 'epoch': 0.01}
+
+100%|██████████| 10/10 [00:27<00:00,  1.09s/it]
+100%|██████████| 10/10 [00:27<00:00,  2.75s/it]
+Saving final model to: ./checkpoints/final_model
+Training metrics: {'val_loss': 2.0517611503601074, 'train_loss': 2.0751}
+Number of training samples: 13857
+Saved metrics.json to checkpoints/final_model/metrics.json
+Uploading to vinoku89/aimo3-exp-1 (main branch)...
+
+Processing Files (0 / 0)      : |          |  0.00B /  0.00B            
+
+New Data Upload               : |          |  0.00B /  0.00B            
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+
+  ...adapter_model.safetensors:   2%|▏         |  433kB / 20.2MB            
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+
+  ...adapter_model.safetensors:   2%|▏         |  433kB / 20.2MB            
+Processing Files (1 / 2)      :  37%|███▋      | 11.9MB / 31.7MB,   ???B/s  
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+
+  ...adapter_model.safetensors:   2%|▏         |  433kB / 20.2MB            
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+
+  ...adapter_model.safetensors:   2%|▏         |  433kB / 20.2MB            
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+
+  ...adapter_model.safetensors:   7%|▋         | 1.50MB / 20.2MB            
+Processing Files (1 / 2)      :  41%|████      | 12.9MB / 31.7MB, 1.78MB/s  
+
+New Data Upload               :   3%|▎         |  608kB / 19.3MB, 1.01MB/s  
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+
+  ...adapter_model.safetensors:   7%|▋         | 1.50MB / 20.2MB            
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+
+  ...adapter_model.safetensors:   7%|▋         | 1.50MB / 20.2MB            
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+
+  ...adapter_model.safetensors:  16%|█▋        | 3.33MB / 20.2MB            
+Processing Files (1 / 2)      :  47%|████▋     | 14.7MB / 31.7MB, 2.41MB/s  
+
+New Data Upload               :  13%|█▎        | 2.43MB / 19.3MB, 2.03MB/s  
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+
+  ...adapter_model.safetensors:  34%|███▍      | 6.97MB / 20.2MB            
+Processing Files (1 / 2)      :  58%|█████▊    | 18.4MB / 31.7MB, 4.67MB/s  
+
+New Data Upload               :  31%|███▏      | 6.08MB / 19.3MB, 4.34MB/s  
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+
+  ...adapter_model.safetensors:  86%|████████▌ | 17.3MB / 20.2MB            
+Processing Files (1 / 2)      :  91%|█████████ | 28.7MB / 31.7MB, 10.6MB/s  
+
+New Data Upload               :  85%|████████▍ | 16.4MB / 19.3MB, 10.3MB/s  
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+
+  ...adapter_model.safetensors:  98%|█████████▊| 19.7MB / 20.2MB            
+Processing Files (1 / 2)      :  98%|█████████▊| 31.2MB / 31.7MB, 10.7MB/s  
+
+New Data Upload               :  97%|█████████▋| 18.8MB / 19.3MB, 10.5MB/s  
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+
+  ...adapter_model.safetensors:  98%|█████████▊| 19.7MB / 20.2MB            
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+
+  ...adapter_model.safetensors:  98%|█████████▊| 19.7MB / 20.2MB            
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+
+  ...adapter_model.safetensors:  98%|█████████▊| 19.7MB / 20.2MB            
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+
+  ...adapter_model.safetensors: 100%|██████████| 20.2MB / 20.2MB            
+Processing Files (2 / 2)      : 100%|██████████| 31.7MB / 31.7MB, 7.62MB/s  
+
+New Data Upload               : 100%|██████████| 19.3MB / 19.3MB, 7.44MB/s  
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+
+  ...adapter_model.safetensors: 100%|██████████| 20.2MB / 20.2MB            
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+
+  ...adapter_model.safetensors: 100%|██████████| 20.2MB / 20.2MB            
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+
+  ...adapter_model.safetensors: 100%|██████████| 20.2MB / 20.2MB            
+Processing Files (2 / 2)      : 100%|██████████| 31.7MB / 31.7MB, 6.60MB/s  
+
+New Data Upload               : 100%|██████████| 19.3MB / 19.3MB, 6.45MB/s  
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+  ...adapter_model.safetensors: 100%|██████████| 20.2MB / 20.2MB            
+Warning: Could not create run tag: 409 Client Error: Conflict for url: https://huggingface.co/api/models/vinoku89/aimo3-exp-1/tag/main (Request ID: Root=1-696573e6-143cc58e016376b56072dfc8;2c01acd5-4334-4a67-912b-ea1bd7169a04)
+
+Tag reference exists already
+Updated 'latest' tag
+Updated 'best' tag (val_loss: 2.0518 < 2.0519)
+Upload complete: https://huggingface.co/vinoku89/aimo3-exp-1
+Uploaded logs/runtime_config_20260113_061956.yaml to vinoku89/aimo3-exp-1 (main branch)
+
+============================================================
+Model pushed successfully!
+============================================================
+  URL: https://huggingface.co/vinoku89/aimo3-exp-1
+  Run Tag: qwen0.6b-sft-e1-13.9k-loss2.05-20260112-001
+  Tags: qwen0.6b-sft-e1-13.9k-loss2.05-20260112-001, latest, best (if lowest loss)
+
+  To load this specific run:
+    from peft import PeftModel
+    model = PeftModel.from_pretrained('base_model', 'vinoku89/aimo3-exp-1', revision='qwen0.6b-sft-e1-13.9k-loss2.05-20260112-001')
+
+  To load the best run:
+    model = PeftModel.from_pretrained('base_model', 'vinoku89/aimo3-exp-1', revision='best')
+
+  To load the latest run:
+    model = PeftModel.from_pretrained('base_model', 'vinoku89/aimo3-exp-1', revision='latest')
+============================================================
+
+wandb: updating run metadata
+wandb: uploading history steps 1-2, summary, console lines 52-52; uploading output.log; uploading wandb-summary.json
+wandb: uploading output.log; uploading config.yaml
+wandb: uploading console lines 54-89
+wandb: 
+wandb: Run history:
+wandb:             eval/entropy ▁
+wandb:                eval/loss ▁
+wandb: eval/mean_token_accuracy ▁
+wandb:          eval/num_tokens ▁
+wandb:             eval/runtime ▁
+wandb:  eval/samples_per_second ▁
+wandb:    eval/steps_per_second ▁
+wandb:            train/entropy ▁
+wandb:              train/epoch ▁▁▁
+wandb:        train/global_step ▁▁▁
+wandb:                       +5 ...
+wandb: 
+wandb: Run summary:
+wandb:             eval/entropy 0.92723
+wandb:                eval/loss 2.05176
+wandb: eval/mean_token_accuracy 0.66866
+wandb:          eval/num_tokens 21152
+wandb:             eval/runtime 13.824
+wandb:  eval/samples_per_second 30.093
+wandb:    eval/steps_per_second 15.046
+wandb:               total_flos 67838115053568.0
+wandb:            train/entropy 0.94697
+wandb:              train/epoch 0.00577
+wandb:                      +10 ...
+wandb: 
+wandb: 🚀 View run qwen3-0.6b-sft-lora-20260113_061956-001 at: https://wandb.ai/vinokuteam/aimo3-exp-1-train/runs/3fkbe45r
+wandb: ⭐️ View project at: https://wandb.ai/vinokuteam/aimo3-exp-1-train
+wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
+wandb: Find logs at: ./wandb/run-20260112_222013-3fkbe45r/logs
+Training completed successfully!
+[W112 22:21:34.601229775 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator())
+[W112 22:21:36.393460385 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator())