Training log (success)

logs/training_20260113_063547.log

@@ -0,0 +1,645 @@
+/venv/main/lib/python3.12/site-packages/trl/__init__.py:203: UserWarning: TRL currently supports vLLM versions: 0.10.2, 0.11.0, 0.11.1, 0.11.2. You have version 0.13.0 installed. We recommend installing a supported version to avoid compatibility issues.
+  if is_vllm_available():
+Loading config from: configs/runtime_config.yaml
+Random seed set to: 42
+============================================================
+Training Configuration:
+  Framework: accelerate
+  Method: sft
+============================================================
+W&B Project: aimo3-exp-1-train
+W&B Run: qwen3-0.6b-sft-lora-20260113_063547-003
+wandb: Currently logged in as: vinoku (vinokuteam) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
+wandb: WARNING Using a boolean value for 'reinit' is deprecated. Use 'return_previous' or 'finish_previous' instead.
+wandb: setting up run 0pkyx3cq
+wandb: Tracking run with wandb version 0.23.1
+wandb: Run data is saved locally in /workspace/distributed-sft-rl-inference-pipeline/wandb/run-20260112_223606-0pkyx3cq
+wandb: Run `wandb offline` to turn off syncing.
+wandb: Syncing run qwen3-0.6b-sft-lora-20260113_063547-003
+wandb: ⭐️ View project at https://wandb.ai/vinokuteam/aimo3-exp-1-train
+wandb: 🚀 View run at https://wandb.ai/vinokuteam/aimo3-exp-1-train/runs/0pkyx3cq
+wandb: Detected [openai] in use.
+wandb: Use W&B Weave for improved LLM call tracing. Install Weave with `pip install weave` then add `import weave` to the top of your script.
+wandb: For more information, check out the docs at: https://weave-docs.wandb.ai/
+W&B Run: qwen3-0.6b-sft-lora-20260113_063547-003
+W&B URL: https://wandb.ai/vinokuteam/aimo3-exp-1-train/runs/0pkyx3cq
+W&B logging enabled (ID: 0pkyx3cq)
+Loading model: Qwen/Qwen3-0.6B
+ZeRO-0 mode (no HfDeepSpeedConfig needed)
+`torch_dtype` is deprecated! Use `dtype` instead!
+Building datasets...
+Loading dataset from HuggingFace: HAD653/GSM8K-OpenMath-MathReason-13k
+Train size: 13857, Test size: 416
+Dataset size after dropping NaN: 13857
+Dataset size after dropping NaN: 416
+Final train size: 13857, test size: 416
+
+============================================================
+SAMPLE TRAINING DATA (first example):
+============================================================
+<|im_start|>system
+You are a helpful and reflective maths assistant. Please reason step by step to put the answer in \boxed{}.<|im_end|>
+<|im_start|>user
+Albert is wondering how much pizza he can eat in one day. He buys 2 large pizzas and 2 small pizzas. A large pizza has 16 slices and a small pizza has 8 slices. If he eats it all, how many pieces does he eat that day?<|im_end|>
+<|im_start|>assistant
+<think>
+Problem:
+Albert buys 2 large pizzas (16 slices each) and 2 small pizzas (8 slices each). Find total slices eaten.
+
+Reasoning:
+1. Large pizza slices = 2*16 = 32.
+2. Small pizza slices = 2*8 = 16.
+3. Total slices = 32+16 = 48.
+
+Answer:
+48
+</think>
+
+\boxed{48}<|im_end|>
+============================================================
+
+Creating LoRA configuration...
+Using Accelerate + SFT training pipeline
+Initializing SFTTrainer...
+
+Adding EOS to train dataset:   0%|          | 0/13857 [00:00<?, ? examples/s]
+Adding EOS to train dataset:  29%|██▉       | 4000/13857 [00:00<00:00, 39763.60 examples/s]
+Adding EOS to train dataset:  60%|██████    | 8359/13857 [00:00<00:00, 42001.96 examples/s]
+Adding EOS to train dataset:  91%|█████████ | 12594/13857 [00:00<00:00, 42156.41 examples/s]
+Adding EOS to train dataset: 100%|██████████| 13857/13857 [00:00<00:00, 41823.49 examples/s]
+
+Tokenizing train dataset:   0%|          | 0/13857 [00:00<?, ? examples/s]
+Tokenizing train dataset:   1%|▏         | 191/13857 [00:00<00:07, 1890.51 examples/s]
+Tokenizing train dataset:   3%|▎         | 389/13857 [00:00<00:06, 1940.43 examples/s]
+Tokenizing train dataset:   4%|▍         | 596/13857 [00:00<00:06, 1995.13 examples/s]
+Tokenizing train dataset:   6%|▌         | 809/13857 [00:00<00:06, 2044.60 examples/s]
+Tokenizing train dataset:   8%|▊         | 1102/13857 [00:00<00:06, 1846.58 examples/s]
+Tokenizing train dataset:   9%|▉         | 1308/13857 [00:00<00:06, 1904.63 examples/s]
+Tokenizing train dataset:  11%|█         | 1506/13857 [00:00<00:06, 1922.84 examples/s]
+Tokenizing train dataset:  12%|█▏        | 1715/13857 [00:00<00:06, 1968.12 examples/s]
+Tokenizing train dataset:  14%|█▍        | 1917/13857 [00:00<00:06, 1981.04 examples/s]
+Tokenizing train dataset:  16%|█▌        | 2179/13857 [00:01<00:06, 1760.60 examples/s]
+Tokenizing train dataset:  17%|█▋        | 2383/13857 [00:01<00:06, 1830.22 examples/s]
+Tokenizing train dataset:  19%|█▊        | 2589/13857 [00:01<00:05, 1889.05 examples/s]
+Tokenizing train dataset:  20%|██        | 2797/13857 [00:01<00:05, 1940.31 examples/s]
+Tokenizing train dataset:  22%|██▏       | 3000/13857 [00:01<00:05, 1813.69 examples/s]
+Tokenizing train dataset:  23%|██▎       | 3199/13857 [00:01<00:05, 1859.46 examples/s]
+Tokenizing train dataset:  25%|██▍       | 3402/13857 [00:01<00:05, 1902.25 examples/s]
+Tokenizing train dataset:  26%|██▌       | 3607/13857 [00:01<00:05, 1940.17 examples/s]
+Tokenizing train dataset:  27%|██▋       | 3809/13857 [00:01<00:05, 1961.99 examples/s]
+Tokenizing train dataset:  30%|██▉       | 4096/13857 [00:02<00:05, 1832.11 examples/s]
+Tokenizing train dataset:  31%|███       | 4297/13857 [00:02<00:05, 1874.63 examples/s]
+Tokenizing train dataset:  32%|███▏      | 4496/13857 [00:02<00:04, 1902.27 examples/s]
+Tokenizing train dataset:  34%|███▍      | 4701/13857 [00:02<00:04, 1939.24 examples/s]
+Tokenizing train dataset:  35%|███▌      | 4901/13857 [00:02<00:04, 1954.40 examples/s]
+Tokenizing train dataset:  37%|███▋      | 5105/13857 [00:02<00:04, 1797.40 examples/s]
+Tokenizing train dataset:  38%|███▊      | 5309/13857 [00:02<00:04, 1861.65 examples/s]
+Tokenizing train dataset:  40%|███▉      | 5507/13857 [00:02<00:04, 1891.21 examples/s]
+Tokenizing train dataset:  41%|████      | 5714/13857 [00:03<00:04, 1937.16 examples/s]
+Tokenizing train dataset:  43%|████▎     | 5912/13857 [00:03<00:04, 1946.11 examples/s]
+Tokenizing train dataset:  45%|████▍     | 6205/13857 [00:03<00:04, 1812.33 examples/s]
+Tokenizing train dataset:  46%|████▌     | 6401/13857 [00:03<00:04, 1847.31 examples/s]
+Tokenizing train dataset:  48%|████▊     | 6595/13857 [00:03<00:03, 1870.17 examples/s]
+Tokenizing train dataset:  49%|████▉     | 6797/13857 [00:03<00:03, 1907.11 examples/s]
+Tokenizing train dataset:  51%|█████     | 6998/13857 [00:03<00:03, 1932.64 examples/s]
+Tokenizing train dataset:  52%|█████▏    | 7235/13857 [00:03<00:03, 1795.00 examples/s]
+Tokenizing train dataset:  54%|█████▍    | 7459/13857 [00:03<00:03, 1685.28 examples/s]
+Tokenizing train dataset:  55%|█████▌    | 7687/13857 [00:04<00:03, 1627.60 examples/s]
+Tokenizing train dataset:  57%|█████▋    | 7927/13857 [00:04<00:03, 1614.76 examples/s]
+Tokenizing train dataset:  59%|█████▉    | 8158/13857 [00:04<00:03, 1455.26 examples/s]
+Tokenizing train dataset:  60%|██████    | 8380/13857 [00:04<00:03, 1460.88 examples/s]
+Tokenizing train dataset:  62%|██████▏   | 8532/13857 [00:04<00:03, 1470.97 examples/s]
+Tokenizing train dataset:  63%|██████▎   | 8684/13857 [00:04<00:03, 1480.53 examples/s]
+Tokenizing train dataset:  64%|██████▍   | 8839/13857 [00:04<00:03, 1493.90 examples/s]
+Tokenizing train dataset:  65%|██████▍   | 8997/13857 [00:05<00:03, 1514.36 examples/s]
+Tokenizing train dataset:  66%|██████▌   | 9159/13857 [00:05<00:03, 1352.37 examples/s]
+Tokenizing train dataset:  67%|██████▋   | 9315/13857 [00:05<00:03, 1403.67 examples/s]
+Tokenizing train dataset:  69%|██████▉   | 9531/13857 [00:05<00:03, 1412.98 examples/s]
+Tokenizing train dataset:  70%|██████▉   | 9689/13857 [00:05<00:02, 1452.32 examples/s]
+Tokenizing train dataset:  71%|███████   | 9842/13857 [00:05<00:02, 1470.22 examples/s]
+Tokenizing train dataset:  72%|███████▏  | 10000/13857 [00:05<00:03, 1238.11 examples/s]
+Tokenizing train dataset:  73%|███████▎  | 10156/13857 [00:05<00:02, 1313.97 examples/s]
+Tokenizing train dataset:  74%|███████▍  | 10308/13857 [00:06<00:02, 1362.84 examples/s]
+Tokenizing train dataset:  75%|███████▌  | 10462/13857 [00:06<00:02, 1407.76 examples/s]
+Tokenizing train dataset:  77%|███████▋  | 10615/13857 [00:06<00:02, 1438.19 examples/s]
+Tokenizing train dataset:  78%|███████▊  | 10772/13857 [00:06<00:02, 1470.99 examples/s]
+Tokenizing train dataset:  79%|███████▉  | 10925/13857 [00:06<00:01, 1483.63 examples/s]
+Tokenizing train dataset:  80%|███████▉  | 11076/13857 [00:06<00:02, 1320.95 examples/s]
+Tokenizing train dataset:  81%|████████  | 11247/13857 [00:06<00:01, 1422.45 examples/s]
+Tokenizing train dataset:  82%|████████▏ | 11405/13857 [00:06<00:01, 1462.65 examples/s]
+Tokenizing train dataset:  83%|████████▎ | 11562/13857 [00:06<00:01, 1490.55 examples/s]
+Tokenizing train dataset:  85%|████████▌ | 11788/13857 [00:07<00:01, 1492.53 examples/s]
+Tokenizing train dataset:  86%|████████▌ | 11944/13857 [00:07<00:01, 1508.49 examples/s]
+Tokenizing train dataset:  88%|████████▊ | 12170/13857 [00:07<00:01, 1386.13 examples/s]
+Tokenizing train dataset:  89%|████████▉ | 12320/13857 [00:07<00:01, 1412.79 examples/s]
+Tokenizing train dataset:  90%|█████████ | 12473/13857 [00:07<00:00, 1438.11 examples/s]
+Tokenizing train dataset:  91%|█████████ | 12631/13857 [00:07<00:00, 1473.12 examples/s]
+Tokenizing train dataset:  92%|█████████▏| 12781/13857 [00:07<00:00, 1477.85 examples/s]
+Tokenizing train dataset:  94%|█████████▍| 13000/13857 [00:07<00:00, 1325.28 examples/s]
+Tokenizing train dataset:  95%|█████████▍| 13154/13857 [00:08<00:00, 1374.38 examples/s]
+Tokenizing train dataset:  96%|█████████▌| 13313/13857 [00:08<00:00, 1426.68 examples/s]
+Tokenizing train dataset:  97%|█████████▋| 13466/13857 [00:08<00:00, 1448.56 examples/s]
+Tokenizing train dataset:  98%|█████████▊| 13625/13857 [00:08<00:00, 1483.41 examples/s]
+Tokenizing train dataset:  99%|█████████▉| 13778/13857 [00:08<00:00, 1494.49 examples/s]
+Tokenizing train dataset: 100%|██████████| 13857/13857 [00:08<00:00, 1626.11 examples/s]
+
+Truncating train dataset:   0%|          | 0/13857 [00:00<?, ? examples/s]
+Truncating train dataset: 100%|██████████| 13857/13857 [00:00<00:00, 342437.05 examples/s]
+
+Adding EOS to eval dataset:   0%|          | 0/416 [00:00<?, ? examples/s]
+Adding EOS to eval dataset: 100%|██████████| 416/416 [00:00<00:00, 38467.98 examples/s]
+
+Tokenizing eval dataset:   0%|          | 0/416 [00:00<?, ? examples/s]
+Tokenizing eval dataset:  43%|████▎     | 177/416 [00:00<00:00, 1753.62 examples/s]
+Tokenizing eval dataset: 100%|██████████| 416/416 [00:00<00:00, 1568.02 examples/s]
+Tokenizing eval dataset: 100%|██████████| 416/416 [00:00<00:00, 1580.56 examples/s]
+
+Truncating eval dataset:   0%|          | 0/416 [00:00<?, ? examples/s]
+Truncating eval dataset: 100%|██████████| 416/416 [00:00<00:00, 217777.14 examples/s]
+Starting training...
+The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}.
+[2026-01-12 22:36:44] WARNING accelerator.py:2164: Gradient accumulation steps mismatch: GradientAccumulationPlugin has 1, DeepSpeed config has 4. Using DeepSpeed's value.
+
+  0%|          | 0/10 [00:00<?, ?it/s]
+ 10%|█         | 1/10 [00:01<00:15,  1.67s/it]
+ 20%|██        | 2/10 [00:02<00:10,  1.34s/it]
+ 30%|███       | 3/10 [00:03<00:08,  1.24s/it]
+ 40%|████      | 4/10 [00:05<00:07,  1.19s/it]
+ 50%|█████     | 5/10 [00:06<00:05,  1.16s/it]
+ 60%|██████    | 6/10 [00:07<00:04,  1.14s/it]
+ 70%|███████   | 7/10 [00:08<00:03,  1.12s/it]
+ 80%|████████  | 8/10 [00:09<00:02,  1.11s/it]
+ 90%|█████████ | 9/10 [00:10<00:01,  1.10s/it]
+100%|██████████| 10/10 [00:11<00:00,  1.10s/it]
+                                               
+{'loss': 2.0744, 'grad_norm': 0.12263150345158981, 'learning_rate': 4.000000000000001e-06, 'entropy': 0.9474609375, 'num_tokens': 21152.0, 'mean_token_accuracy': 0.6660121768712998, 'epoch': 0.01}
+
+100%|██████████| 10/10 [00:11<00:00,  1.10s/it]
+
+  0%|          | 0/208 [00:00<?, ?it/s]
+
+  1%|▏         | 3/208 [00:00<00:09, 22.78it/s]
+
+  3%|▎         | 6/208 [00:00<00:11, 17.53it/s]
+
+  4%|▍         | 8/208 [00:00<00:12, 16.51it/s]
+
+  5%|▍         | 10/208 [00:00<00:12, 15.75it/s]
+
+  6%|▌         | 12/208 [00:00<00:12, 15.41it/s]
+
+  7%|▋         | 14/208 [00:00<00:12, 15.61it/s]
+
+  8%|▊         | 16/208 [00:00<00:12, 15.55it/s]
+
+  9%|▊         | 18/208 [00:01<00:12, 15.62it/s]
+
+ 10%|▉         | 20/208 [00:01<00:12, 15.58it/s]
+
+ 11%|█         | 22/208 [00:01<00:11, 15.69it/s]
+
+ 12%|█▏        | 24/208 [00:01<00:11, 15.95it/s]
+
+ 12%|█▎        | 26/208 [00:01<00:11, 16.03it/s]
+
+ 13%|█▎        | 28/208 [00:01<00:11, 15.68it/s]
+
+ 14%|█▍        | 30/208 [00:01<00:11, 15.51it/s]
+
+ 15%|█▌        | 32/208 [00:02<00:11, 15.46it/s]
+
+ 16%|█▋        | 34/208 [00:02<00:11, 15.43it/s]
+
+ 17%|█▋        | 36/208 [00:02<00:11, 15.41it/s]
+
+ 18%|█▊        | 38/208 [00:02<00:11, 15.42it/s]
+
+ 19%|█▉        | 40/208 [00:02<00:10, 15.41it/s]
+
+ 20%|██        | 42/208 [00:02<00:10, 15.25it/s]
+
+ 21%|██        | 44/208 [00:02<00:10, 15.26it/s]
+
+ 22%|██▏       | 46/208 [00:02<00:10, 15.33it/s]
+
+ 23%|██▎       | 48/208 [00:03<00:10, 15.15it/s]
+
+ 24%|██▍       | 50/208 [00:03<00:10, 15.32it/s]
+
+ 25%|██▌       | 52/208 [00:03<00:10, 15.22it/s]
+
+ 26%|██▌       | 54/208 [00:03<00:10, 15.31it/s]
+
+ 27%|██▋       | 56/208 [00:03<00:09, 15.27it/s]
+
+ 28%|██▊       | 58/208 [00:03<00:09, 15.30it/s]
+
+ 29%|██▉       | 60/208 [00:03<00:09, 15.28it/s]
+
+ 30%|██▉       | 62/208 [00:03<00:09, 15.53it/s]
+
+ 31%|███       | 64/208 [00:04<00:09, 15.59it/s]
+
+ 32%|███▏      | 66/208 [00:04<00:09, 15.02it/s]
+
+ 33%|███▎      | 68/208 [00:04<00:09, 14.82it/s]
+
+ 34%|███▎      | 70/208 [00:04<00:09, 14.82it/s]
+
+ 35%|███▍      | 72/208 [00:04<00:09, 14.82it/s]
+
+ 36%|███▌      | 74/208 [00:04<00:09, 14.87it/s]
+
+ 37%|███▋      | 76/208 [00:04<00:08, 14.95it/s]
+
+ 38%|███▊      | 78/208 [00:05<00:08, 14.97it/s]
+
+ 38%|███▊      | 80/208 [00:05<00:08, 15.02it/s]
+
+ 39%|███▉      | 82/208 [00:05<00:08, 15.07it/s]
+
+ 40%|████      | 84/208 [00:05<00:08, 15.13it/s]
+
+ 41%|████▏     | 86/208 [00:05<00:07, 15.28it/s]
+
+ 42%|████▏     | 88/208 [00:05<00:07, 15.36it/s]
+
+ 43%|████▎     | 90/208 [00:05<00:07, 15.22it/s]
+
+ 44%|████▍     | 92/208 [00:05<00:07, 15.30it/s]
+
+ 45%|████▌     | 94/208 [00:06<00:07, 15.19it/s]
+
+ 46%|████▌     | 96/208 [00:06<00:07, 15.19it/s]
+
+ 47%|████▋     | 98/208 [00:06<00:07, 15.42it/s]
+
+ 48%|████▊     | 100/208 [00:06<00:07, 15.43it/s]
+
+ 49%|████▉     | 102/208 [00:06<00:06, 15.51it/s]
+
+ 50%|█████     | 104/208 [00:06<00:06, 15.57it/s]
+
+ 51%|█████     | 106/208 [00:06<00:06, 15.32it/s]
+
+ 52%|█████▏    | 108/208 [00:07<00:06, 15.22it/s]
+
+ 53%|█████▎    | 110/208 [00:07<00:06, 15.25it/s]
+
+ 54%|█████▍    | 112/208 [00:07<00:06, 15.31it/s]
+
+ 55%|█████▍    | 114/208 [00:07<00:06, 15.18it/s]
+
+ 56%|█████▌    | 116/208 [00:07<00:06, 15.14it/s]
+
+ 57%|█████▋    | 118/208 [00:07<00:05, 15.15it/s]
+
+ 58%|█████▊    | 120/208 [00:07<00:05, 15.26it/s]
+
+ 59%|█████▊    | 122/208 [00:07<00:05, 15.42it/s]
+
+ 60%|█████▉    | 124/208 [00:08<00:05, 15.58it/s]
+
+ 61%|██████    | 126/208 [00:08<00:05, 15.48it/s]
+
+ 62%|██████▏   | 128/208 [00:08<00:05, 15.67it/s]
+
+ 62%|██████▎   | 130/208 [00:08<00:04, 15.61it/s]
+
+ 63%|██████▎   | 132/208 [00:08<00:04, 15.48it/s]
+
+ 64%|██████▍   | 134/208 [00:08<00:04, 15.52it/s]
+
+ 65%|██████▌   | 136/208 [00:08<00:04, 15.40it/s]
+
+ 66%|██████▋   | 138/208 [00:08<00:04, 15.47it/s]
+
+ 67%|██████▋   | 140/208 [00:09<00:04, 15.65it/s]
+
+ 68%|██████▊   | 142/208 [00:09<00:04, 15.62it/s]
+
+ 69%|██████▉   | 144/208 [00:09<00:04, 15.55it/s]
+
+ 70%|███████   | 146/208 [00:09<00:03, 15.71it/s]
+
+ 71%|███████   | 148/208 [00:09<00:03, 15.65it/s]
+
+ 72%|███████▏  | 150/208 [00:09<00:03, 15.64it/s]
+
+ 73%|███████▎  | 152/208 [00:09<00:03, 15.66it/s]
+
+ 74%|███████▍  | 154/208 [00:09<00:03, 15.69it/s]
+
+ 75%|███████▌  | 156/208 [00:10<00:03, 15.53it/s]
+
+ 76%|███████▌  | 158/208 [00:10<00:03, 15.50it/s]
+
+ 77%|███████▋  | 160/208 [00:10<00:03, 15.50it/s]
+
+ 78%|███████▊  | 162/208 [00:10<00:02, 15.52it/s]
+
+ 79%|███████▉  | 164/208 [00:10<00:02, 15.53it/s]
+
+ 80%|███████▉  | 166/208 [00:10<00:02, 15.57it/s]
+
+ 81%|████████  | 168/208 [00:10<00:02, 15.74it/s]
+
+ 82%|████████▏ | 170/208 [00:11<00:02, 15.59it/s]
+
+ 83%|████████▎ | 172/208 [00:11<00:02, 15.50it/s]
+
+ 84%|████████▎ | 174/208 [00:11<00:02, 15.56it/s]
+
+ 85%|████████▍ | 176/208 [00:11<00:02, 15.62it/s]
+
+ 86%|████████▌ | 178/208 [00:11<00:01, 15.40it/s]
+
+ 87%|████████▋ | 180/208 [00:11<00:01, 15.37it/s]
+
+ 88%|████████▊ | 182/208 [00:11<00:01, 15.45it/s]
+
+ 88%|████████▊ | 184/208 [00:11<00:01, 15.39it/s]
+
+ 89%|████████▉ | 186/208 [00:12<00:01, 15.26it/s]
+
+ 90%|█████████ | 188/208 [00:12<00:01, 15.30it/s]
+
+ 91%|█████████▏| 190/208 [00:12<00:01, 15.37it/s]
+
+ 92%|█████████▏| 192/208 [00:12<00:01, 15.34it/s]
+
+ 93%|█████████▎| 194/208 [00:12<00:00, 15.58it/s]
+
+ 94%|█████████▍| 196/208 [00:12<00:00, 15.48it/s]
+
+ 95%|█████████▌| 198/208 [00:12<00:00, 15.41it/s]
+
+ 96%|█████████▌| 200/208 [00:12<00:00, 15.51it/s]
+
+ 97%|█████████▋| 202/208 [00:13<00:00, 15.12it/s]
+
+ 98%|█████████▊| 204/208 [00:13<00:00, 14.83it/s]
+
+ 99%|█████████▉| 206/208 [00:13<00:00, 14.90it/s]
+
+100%|██████████| 208/208 [00:13<00:00, 14.85it/s]
+                                               
+
+                                                 
+{'eval_loss': 2.0531225204467773, 'eval_runtime': 13.5735, 'eval_samples_per_second': 30.648, 'eval_steps_per_second': 15.324, 'eval_entropy': 0.9273212139423077, 'eval_num_tokens': 21152.0, 'eval_mean_token_accuracy': 0.6695072605059698, 'epoch': 0.01}
+
+100%|██████████| 10/10 [00:25<00:00,  1.10s/it]
+
+100%|██████████| 208/208 [00:13<00:00, 14.85it/s]
+
+                                                 
+                                               
+{'train_runtime': 27.5135, 'train_samples_per_second': 2.908, 'train_steps_per_second': 0.363, 'train_loss': 2.074407958984375, 'epoch': 0.01}
+
+100%|██████████| 10/10 [00:27<00:00,  1.10s/it]
+100%|██████████| 10/10 [00:27<00:00,  2.75s/it]
+Saving final model to: ./checkpoints/final_model
+Training metrics: {'val_loss': 2.0531225204467773, 'train_loss': 2.0744}
+Number of training samples: 13857
+Saved metrics.json to checkpoints/final_model/metrics.json
+Uploading to vinoku89/aimo3-exp-1 (main branch)...
+
+Processing Files (0 / 0)      : |          |  0.00B /  0.00B            
+
+New Data Upload               : |          |  0.00B /  0.00B            
+
+
+  ...adapter_model.safetensors:   2%|▏         |  343kB / 20.2MB            
+
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+  ...adapter_model.safetensors:   2%|▏         |  343kB / 20.2MB            
+
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 2)      :  37%|███▋      | 11.8MB / 31.7MB,   ???B/s  
+
+
+  ...adapter_model.safetensors:   2%|▏         |  343kB / 20.2MB            
+
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+  ...adapter_model.safetensors:   2%|▏         |  343kB / 20.2MB            
+
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+  ...adapter_model.safetensors:   7%|▋         | 1.45MB / 20.2MB            
+
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 2)      :  41%|████      | 12.9MB / 31.7MB, 1.84MB/s  
+
+New Data Upload               :   3%|▎         |  608kB / 19.4MB, 1.01MB/s  
+
+
+  ...adapter_model.safetensors:   7%|▋         | 1.45MB / 20.2MB            
+
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+  ...adapter_model.safetensors:   7%|▋         | 1.45MB / 20.2MB            
+
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+  ...adapter_model.safetensors:   7%|▋         | 1.45MB / 20.2MB            
+
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+  ...adapter_model.safetensors:  16%|█▌        | 3.27MB / 20.2MB            
+
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 2)      :  46%|████▋     | 14.7MB / 31.7MB, 2.09MB/s  
+
+New Data Upload               :  13%|█▎        | 2.43MB / 19.4MB, 1.74MB/s  
+
+
+  ...adapter_model.safetensors:  28%|██▊       | 5.71MB / 20.2MB            
+
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 2)      :  54%|█████▍    | 17.1MB / 31.7MB, 3.35MB/s  
+
+New Data Upload               :  25%|██▌       | 4.87MB / 19.4MB, 3.04MB/s  
+
+
+  ...adapter_model.safetensors:  61%|██████▏   | 12.4MB / 20.2MB            
+
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 2)      :  75%|███████▌  | 23.8MB / 31.7MB, 6.70MB/s  
+
+New Data Upload               :  60%|█████▉    | 11.6MB / 19.4MB, 6.42MB/s  
+
+
+  ...adapter_model.safetensors:  97%|█████████▋| 19.7MB / 20.2MB            
+
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (1 / 2)      :  98%|█████████▊| 31.1MB / 31.7MB, 9.68MB/s  
+
+New Data Upload               :  97%|█████████▋| 18.9MB / 19.4MB, 9.43MB/s  
+
+
+  ...adapter_model.safetensors:  97%|█████████▋| 19.7MB / 20.2MB            
+
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+  ...adapter_model.safetensors:  97%|█████████▋| 19.7MB / 20.2MB            
+
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+  ...adapter_model.safetensors:  97%|█████████▋| 19.7MB / 20.2MB            
+
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+  ...adapter_model.safetensors:  97%|█████████▋| 19.7MB / 20.2MB            
+
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+  ...adapter_model.safetensors:  97%|█████████▋| 19.7MB / 20.2MB            
+
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+  ...adapter_model.safetensors: 100%|██████████| 20.2MB / 20.2MB            
+
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (2 / 2)      : 100%|██████████| 31.7MB / 31.7MB, 6.22MB/s  
+
+New Data Upload               : 100%|██████████| 19.4MB / 19.4MB, 6.06MB/s  
+
+
+  ...adapter_model.safetensors: 100%|██████████| 20.2MB / 20.2MB            
+
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+  ...adapter_model.safetensors: 100%|██████████| 20.2MB / 20.2MB            
+
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+
+
+  ...adapter_model.safetensors: 100%|██████████| 20.2MB / 20.2MB            
+
+
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Processing Files (2 / 2)      : 100%|██████████| 31.7MB / 31.7MB, 5.53MB/s  
+
+New Data Upload               : 100%|██████████| 19.4MB / 19.4MB, 5.39MB/s  
+
+  ...adapter_model.safetensors: 100%|██████████| 20.2MB / 20.2MB            
+
+  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+Created tag: qwen0.6b-sft-e1-13.9k-loss2.05-20260112-003
+Updated 'latest' tag
+Kept existing 'best' tag (current: 2.0531 >= best: 2.0518)
+Upload complete: https://huggingface.co/vinoku89/aimo3-exp-1
+Uploaded logs/runtime_config_20260113_063547.yaml to vinoku89/aimo3-exp-1 (main branch)
+
+============================================================
+Model pushed successfully!
+============================================================
+  URL: https://huggingface.co/vinoku89/aimo3-exp-1
+  Run Tag: qwen0.6b-sft-e1-13.9k-loss2.05-20260112-003
+  Tags: qwen0.6b-sft-e1-13.9k-loss2.05-20260112-003, latest, best (if lowest loss)
+
+  To load this specific run:
+    from peft import PeftModel
+    model = PeftModel.from_pretrained('base_model', 'vinoku89/aimo3-exp-1', revision='qwen0.6b-sft-e1-13.9k-loss2.05-20260112-003')
+
+  To load the best run:
+    model = PeftModel.from_pretrained('base_model', 'vinoku89/aimo3-exp-1', revision='best')
+
+  To load the latest run:
+    model = PeftModel.from_pretrained('base_model', 'vinoku89/aimo3-exp-1', revision='latest')
+============================================================
+
+wandb: updating run metadata
+wandb: uploading output.log; uploading wandb-summary.json
+wandb: uploading output.log
+wandb: uploading config.yaml
+wandb: uploading summary, console lines 54-87
+wandb: 
+wandb: Run history:
+wandb:             eval/entropy ▁
+wandb:                eval/loss ▁
+wandb: eval/mean_token_accuracy ▁
+wandb:          eval/num_tokens ▁
+wandb:             eval/runtime ▁
+wandb:  eval/samples_per_second ▁
+wandb:    eval/steps_per_second ▁
+wandb:            train/entropy ▁
+wandb:              train/epoch ▁▁▁
+wandb:        train/global_step ▁▁▁
+wandb:                       +5 ...
+wandb: 
+wandb: Run summary:
+wandb:             eval/entropy 0.92732
+wandb:                eval/loss 2.05312
+wandb: eval/mean_token_accuracy 0.66951
+wandb:          eval/num_tokens 21152
+wandb:             eval/runtime 13.5735
+wandb:  eval/samples_per_second 30.648
+wandb:    eval/steps_per_second 15.324
+wandb:               total_flos 67838115053568.0
+wandb:            train/entropy 0.94746
+wandb:              train/epoch 0.00577
+wandb:                      +10 ...
+wandb: 
+wandb: 🚀 View run qwen3-0.6b-sft-lora-20260113_063547-003 at: https://wandb.ai/vinokuteam/aimo3-exp-1-train/runs/0pkyx3cq
+wandb: ⭐️ View project at: https://wandb.ai/vinokuteam/aimo3-exp-1-train
+wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
+wandb: Find logs at: ./wandb/run-20260112_223606-0pkyx3cq/logs
+Training completed successfully!
+[W112 22:37:40.785124394 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator())
+[W112 22:37:42.435659668 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator())