/venv/main/lib/python3.12/site-packages/trl/__init__.py:203: UserWarning: TRL currently supports vLLM versions: 0.10.2, 0.11.0, 0.11.1, 0.11.2. You have version 0.13.0 installed. We recommend installing a supported version to avoid compatibility issues.
  if is_vllm_available():
Loading config from: configs/runtime_config.yaml
Random seed set to: 42
============================================================
Training Configuration:
  Framework: accelerate
  Method: sft
============================================================
W&B Project: aimo3-exp-1-train
W&B Run: qwen3-0.6b-sft-lora-20260113_063547-003
wandb: Currently logged in as: vinoku (vinokuteam) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: WARNING Using a boolean value for 'reinit' is deprecated. Use 'return_previous' or 'finish_previous' instead.
wandb: setting up run 0pkyx3cq
wandb: Tracking run with wandb version 0.23.1
wandb: Run data is saved locally in /workspace/distributed-sft-rl-inference-pipeline/wandb/run-20260112_223606-0pkyx3cq
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run qwen3-0.6b-sft-lora-20260113_063547-003
wandb: ⭐️ View project at https://wandb.ai/vinokuteam/aimo3-exp-1-train
wandb: 🚀 View run at https://wandb.ai/vinokuteam/aimo3-exp-1-train/runs/0pkyx3cq
wandb: Detected [openai] in use.
wandb: Use W&B Weave for improved LLM call tracing. Install Weave with `pip install weave` then add `import weave` to the top of your script.
wandb: For more information, check out the docs at: https://weave-docs.wandb.ai/
W&B Run: qwen3-0.6b-sft-lora-20260113_063547-003
W&B URL: https://wandb.ai/vinokuteam/aimo3-exp-1-train/runs/0pkyx3cq
W&B logging enabled (ID: 0pkyx3cq)
Loading model: Qwen/Qwen3-0.6B
ZeRO-0 mode (no HfDeepSpeedConfig needed)
`torch_dtype` is deprecated! Use `dtype` instead!
Building datasets...
Loading dataset from HuggingFace: HAD653/GSM8K-OpenMath-MathReason-13k
Train size: 13857, Test size: 416
Dataset size after dropping NaN: 13857
Dataset size after dropping NaN: 416
Final train size: 13857, test size: 416

============================================================
SAMPLE TRAINING DATA (first example):
============================================================
<|im_start|>system
You are a helpful and reflective maths assistant. Please reason step by step to put the answer in \boxed{}.<|im_end|>
<|im_start|>user
Albert is wondering how much pizza he can eat in one day. He buys 2 large pizzas and 2 small pizzas. A large pizza has 16 slices and a small pizza has 8 slices. If he eats it all, how many pieces does he eat that day?<|im_end|>
<|im_start|>assistant
<think>
Problem:
Albert buys 2 large pizzas (16 slices each) and 2 small pizzas (8 slices each). Find total slices eaten.

Reasoning:
1. Large pizza slices = 2*16 = 32.
2. Small pizza slices = 2*8 = 16.
3. Total slices = 32+16 = 48.

Answer:
48
</think>

\boxed{48}<|im_end|>
============================================================

Creating LoRA configuration...
Using Accelerate + SFT training pipeline
Initializing SFTTrainer...

Adding EOS to train dataset:   0%|          | 0/13857 [00:00<?, ? examples/s]
Adding EOS to train dataset:  29%|██▉       | 4000/13857 [00:00<00:00, 39763.60 examples/s]
Adding EOS to train dataset:  60%|██████    | 8359/13857 [00:00<00:00, 42001.96 examples/s]
Adding EOS to train dataset:  91%|█████████ | 12594/13857 [00:00<00:00, 42156.41 examples/s]
Adding EOS to train dataset: 100%|██████████| 13857/13857 [00:00<00:00, 41823.49 examples/s]

Tokenizing train dataset:   0%|          | 0/13857 [00:00<?, ? examples/s]
Tokenizing train dataset:   1%|▏         | 191/13857 [00:00<00:07, 1890.51 examples/s]
Tokenizing train dataset:   3%|▎         | 389/13857 [00:00<00:06, 1940.43 examples/s]
Tokenizing train dataset:   4%|▍         | 596/13857 [00:00<00:06, 1995.13 examples/s]
Tokenizing train dataset:   6%|▌         | 809/13857 [00:00<00:06, 2044.60 examples/s]
Tokenizing train dataset:   8%|▊         | 1102/13857 [00:00<00:06, 1846.58 examples/s]
Tokenizing train dataset:   9%|▉         | 1308/13857 [00:00<00:06, 1904.63 examples/s]
Tokenizing train dataset:  11%|█         | 1506/13857 [00:00<00:06, 1922.84 examples/s]
Tokenizing train dataset:  12%|█▏        | 1715/13857 [00:00<00:06, 1968.12 examples/s]
Tokenizing train dataset:  14%|█▍        | 1917/13857 [00:00<00:06, 1981.04 examples/s]
Tokenizing train dataset:  16%|█▌        | 2179/13857 [00:01<00:06, 1760.60 examples/s]
Tokenizing train dataset:  17%|█▋        | 2383/13857 [00:01<00:06, 1830.22 examples/s]
Tokenizing train dataset:  19%|█▊        | 2589/13857 [00:01<00:05, 1889.05 examples/s]
Tokenizing train dataset:  20%|██        | 2797/13857 [00:01<00:05, 1940.31 examples/s]
Tokenizing train dataset:  22%|██▏       | 3000/13857 [00:01<00:05, 1813.69 examples/s]
Tokenizing train dataset:  23%|██▎       | 3199/13857 [00:01<00:05, 1859.46 examples/s]
Tokenizing train dataset:  25%|██▍       | 3402/13857 [00:01<00:05, 1902.25 examples/s]
Tokenizing train dataset:  26%|██▌       | 3607/13857 [00:01<00:05, 1940.17 examples/s]
Tokenizing train dataset:  27%|██▋       | 3809/13857 [00:01<00:05, 1961.99 examples/s]
Tokenizing train dataset:  30%|██▉       | 4096/13857 [00:02<00:05, 1832.11 examples/s]
Tokenizing train dataset:  31%|███       | 4297/13857 [00:02<00:05, 1874.63 examples/s]
Tokenizing train dataset:  32%|███▏      | 4496/13857 [00:02<00:04, 1902.27 examples/s]
Tokenizing train dataset:  34%|███▍      | 4701/13857 [00:02<00:04, 1939.24 examples/s]
Tokenizing train dataset:  35%|███▌      | 4901/13857 [00:02<00:04, 1954.40 examples/s]
Tokenizing train dataset:  37%|███▋      | 5105/13857 [00:02<00:04, 1797.40 examples/s]
Tokenizing train dataset:  38%|███▊      | 5309/13857 [00:02<00:04, 1861.65 examples/s]
Tokenizing train dataset:  40%|███▉      | 5507/13857 [00:02<00:04, 1891.21 examples/s]
Tokenizing train dataset:  41%|████      | 5714/13857 [00:03<00:04, 1937.16 examples/s]
Tokenizing train dataset:  43%|████▎     | 5912/13857 [00:03<00:04, 1946.11 examples/s]
Tokenizing train dataset:  45%|████▍     | 6205/13857 [00:03<00:04, 1812.33 examples/s]
Tokenizing train dataset:  46%|████▌     | 6401/13857 [00:03<00:04, 1847.31 examples/s]
Tokenizing train dataset:  48%|████▊     | 6595/13857 [00:03<00:03, 1870.17 examples/s]
Tokenizing train dataset:  49%|████▉     | 6797/13857 [00:03<00:03, 1907.11 examples/s]
Tokenizing train dataset:  51%|█████     | 6998/13857 [00:03<00:03, 1932.64 examples/s]
Tokenizing train dataset:  52%|█████▏    | 7235/13857 [00:03<00:03, 1795.00 examples/s]
Tokenizing train dataset:  54%|█████▍    | 7459/13857 [00:03<00:03, 1685.28 examples/s]
Tokenizing train dataset:  55%|█████▌    | 7687/13857 [00:04<00:03, 1627.60 examples/s]
Tokenizing train dataset:  57%|█████▋    | 7927/13857 [00:04<00:03, 1614.76 examples/s]
Tokenizing train dataset:  59%|█████▉    | 8158/13857 [00:04<00:03, 1455.26 examples/s]
Tokenizing train dataset:  60%|██████    | 8380/13857 [00:04<00:03, 1460.88 examples/s]
Tokenizing train dataset:  62%|██████▏   | 8532/13857 [00:04<00:03, 1470.97 examples/s]
Tokenizing train dataset:  63%|██████▎   | 8684/13857 [00:04<00:03, 1480.53 examples/s]
Tokenizing train dataset:  64%|██████▍   | 8839/13857 [00:04<00:03, 1493.90 examples/s]
Tokenizing train dataset:  65%|██████▍   | 8997/13857 [00:05<00:03, 1514.36 examples/s]
Tokenizing train dataset:  66%|██████▌   | 9159/13857 [00:05<00:03, 1352.37 examples/s]
Tokenizing train dataset:  67%|██████▋   | 9315/13857 [00:05<00:03, 1403.67 examples/s]
Tokenizing train dataset:  69%|██████▉   | 9531/13857 [00:05<00:03, 1412.98 examples/s]
Tokenizing train dataset:  70%|██████▉   | 9689/13857 [00:05<00:02, 1452.32 examples/s]
Tokenizing train dataset:  71%|███████   | 9842/13857 [00:05<00:02, 1470.22 examples/s]
Tokenizing train dataset:  72%|███████▏  | 10000/13857 [00:05<00:03, 1238.11 examples/s]
Tokenizing train dataset:  73%|███████▎  | 10156/13857 [00:05<00:02, 1313.97 examples/s]
Tokenizing train dataset:  74%|███████▍  | 10308/13857 [00:06<00:02, 1362.84 examples/s]
Tokenizing train dataset:  75%|███████▌  | 10462/13857 [00:06<00:02, 1407.76 examples/s]
Tokenizing train dataset:  77%|███████▋  | 10615/13857 [00:06<00:02, 1438.19 examples/s]
Tokenizing train dataset:  78%|███████▊  | 10772/13857 [00:06<00:02, 1470.99 examples/s]
Tokenizing train dataset:  79%|███████▉  | 10925/13857 [00:06<00:01, 1483.63 examples/s]
Tokenizing train dataset:  80%|███████▉  | 11076/13857 [00:06<00:02, 1320.95 examples/s]
Tokenizing train dataset:  81%|████████  | 11247/13857 [00:06<00:01, 1422.45 examples/s]
Tokenizing train dataset:  82%|████████▏ | 11405/13857 [00:06<00:01, 1462.65 examples/s]
Tokenizing train dataset:  83%|████████▎ | 11562/13857 [00:06<00:01, 1490.55 examples/s]
Tokenizing train dataset:  85%|████████▌ | 11788/13857 [00:07<00:01, 1492.53 examples/s]
Tokenizing train dataset:  86%|████████▌ | 11944/13857 [00:07<00:01, 1508.49 examples/s]
Tokenizing train dataset:  88%|████████▊ | 12170/13857 [00:07<00:01, 1386.13 examples/s]
Tokenizing train dataset:  89%|████████▉ | 12320/13857 [00:07<00:01, 1412.79 examples/s]
Tokenizing train dataset:  90%|█████████ | 12473/13857 [00:07<00:00, 1438.11 examples/s]
Tokenizing train dataset:  91%|█████████ | 12631/13857 [00:07<00:00, 1473.12 examples/s]
Tokenizing train dataset:  92%|█████████▏| 12781/13857 [00:07<00:00, 1477.85 examples/s]
Tokenizing train dataset:  94%|█████████▍| 13000/13857 [00:07<00:00, 1325.28 examples/s]
Tokenizing train dataset:  95%|█████████▍| 13154/13857 [00:08<00:00, 1374.38 examples/s]
Tokenizing train dataset:  96%|█████████▌| 13313/13857 [00:08<00:00, 1426.68 examples/s]
Tokenizing train dataset:  97%|█████████▋| 13466/13857 [00:08<00:00, 1448.56 examples/s]
Tokenizing train dataset:  98%|█████████▊| 13625/13857 [00:08<00:00, 1483.41 examples/s]
Tokenizing train dataset:  99%|█████████▉| 13778/13857 [00:08<00:00, 1494.49 examples/s]
Tokenizing train dataset: 100%|██████████| 13857/13857 [00:08<00:00, 1626.11 examples/s]

Truncating train dataset:   0%|          | 0/13857 [00:00<?, ? examples/s]
Truncating train dataset: 100%|██████████| 13857/13857 [00:00<00:00, 342437.05 examples/s]

Adding EOS to eval dataset:   0%|          | 0/416 [00:00<?, ? examples/s]
Adding EOS to eval dataset: 100%|██████████| 416/416 [00:00<00:00, 38467.98 examples/s]

Tokenizing eval dataset:   0%|          | 0/416 [00:00<?, ? examples/s]
Tokenizing eval dataset:  43%|████▎     | 177/416 [00:00<00:00, 1753.62 examples/s]
Tokenizing eval dataset: 100%|██████████| 416/416 [00:00<00:00, 1568.02 examples/s]
Tokenizing eval dataset: 100%|██████████| 416/416 [00:00<00:00, 1580.56 examples/s]

Truncating eval dataset:   0%|          | 0/416 [00:00<?, ? examples/s]
Truncating eval dataset: 100%|██████████| 416/416 [00:00<00:00, 217777.14 examples/s]
Starting training...
The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}.
[2026-01-12 22:36:44] WARNING accelerator.py:2164: Gradient accumulation steps mismatch: GradientAccumulationPlugin has 1, DeepSpeed config has 4. Using DeepSpeed's value.

  0%|          | 0/10 [00:00<?, ?it/s]
 10%|█         | 1/10 [00:01<00:15,  1.67s/it]
 20%|██        | 2/10 [00:02<00:10,  1.34s/it]
 30%|███       | 3/10 [00:03<00:08,  1.24s/it]
 40%|████      | 4/10 [00:05<00:07,  1.19s/it]
 50%|█████     | 5/10 [00:06<00:05,  1.16s/it]
 60%|██████    | 6/10 [00:07<00:04,  1.14s/it]
 70%|███████   | 7/10 [00:08<00:03,  1.12s/it]
 80%|████████  | 8/10 [00:09<00:02,  1.11s/it]
 90%|█████████ | 9/10 [00:10<00:01,  1.10s/it]
100%|██████████| 10/10 [00:11<00:00,  1.10s/it]
                                               
{'loss': 2.0744, 'grad_norm': 0.12263150345158981, 'learning_rate': 4.000000000000001e-06, 'entropy': 0.9474609375, 'num_tokens': 21152.0, 'mean_token_accuracy': 0.6660121768712998, 'epoch': 0.01}

100%|██████████| 10/10 [00:11<00:00,  1.10s/it]

  0%|          | 0/208 [00:00<?, ?it/s]

  1%|▏         | 3/208 [00:00<00:09, 22.78it/s]

  3%|▎         | 6/208 [00:00<00:11, 17.53it/s]

  4%|▍         | 8/208 [00:00<00:12, 16.51it/s]

  5%|▍         | 10/208 [00:00<00:12, 15.75it/s]

  6%|▌         | 12/208 [00:00<00:12, 15.41it/s]

  7%|▋         | 14/208 [00:00<00:12, 15.61it/s]

  8%|▊         | 16/208 [00:00<00:12, 15.55it/s]

  9%|▊         | 18/208 [00:01<00:12, 15.62it/s]

 10%|▉         | 20/208 [00:01<00:12, 15.58it/s]

 11%|█         | 22/208 [00:01<00:11, 15.69it/s]

 12%|█▏        | 24/208 [00:01<00:11, 15.95it/s]

 12%|█▎        | 26/208 [00:01<00:11, 16.03it/s]

 13%|█▎        | 28/208 [00:01<00:11, 15.68it/s]

 14%|█▍        | 30/208 [00:01<00:11, 15.51it/s]

 15%|█▌        | 32/208 [00:02<00:11, 15.46it/s]

 16%|█▋        | 34/208 [00:02<00:11, 15.43it/s]

 17%|█▋        | 36/208 [00:02<00:11, 15.41it/s]

 18%|█▊        | 38/208 [00:02<00:11, 15.42it/s]

 19%|█▉        | 40/208 [00:02<00:10, 15.41it/s]

 20%|██        | 42/208 [00:02<00:10, 15.25it/s]

 21%|██        | 44/208 [00:02<00:10, 15.26it/s]

 22%|██▏       | 46/208 [00:02<00:10, 15.33it/s]

 23%|██▎       | 48/208 [00:03<00:10, 15.15it/s]

 24%|██▍       | 50/208 [00:03<00:10, 15.32it/s]

 25%|██▌       | 52/208 [00:03<00:10, 15.22it/s]

 26%|██▌       | 54/208 [00:03<00:10, 15.31it/s]

 27%|██▋       | 56/208 [00:03<00:09, 15.27it/s]

 28%|██▊       | 58/208 [00:03<00:09, 15.30it/s]

 29%|██▉       | 60/208 [00:03<00:09, 15.28it/s]

 30%|██▉       | 62/208 [00:03<00:09, 15.53it/s]

 31%|███       | 64/208 [00:04<00:09, 15.59it/s]

 32%|███▏      | 66/208 [00:04<00:09, 15.02it/s]

 33%|███▎      | 68/208 [00:04<00:09, 14.82it/s]

 34%|███▎      | 70/208 [00:04<00:09, 14.82it/s]

 35%|███▍      | 72/208 [00:04<00:09, 14.82it/s]

 36%|███▌      | 74/208 [00:04<00:09, 14.87it/s]

 37%|███▋      | 76/208 [00:04<00:08, 14.95it/s]

 38%|███▊      | 78/208 [00:05<00:08, 14.97it/s]

 38%|███▊      | 80/208 [00:05<00:08, 15.02it/s]

 39%|███▉      | 82/208 [00:05<00:08, 15.07it/s]

 40%|████      | 84/208 [00:05<00:08, 15.13it/s]

 41%|████▏     | 86/208 [00:05<00:07, 15.28it/s]

 42%|████▏     | 88/208 [00:05<00:07, 15.36it/s]

 43%|████▎     | 90/208 [00:05<00:07, 15.22it/s]

 44%|████▍     | 92/208 [00:05<00:07, 15.30it/s]

 45%|████▌     | 94/208 [00:06<00:07, 15.19it/s]

 46%|████▌     | 96/208 [00:06<00:07, 15.19it/s]

 47%|████▋     | 98/208 [00:06<00:07, 15.42it/s]

 48%|████▊     | 100/208 [00:06<00:07, 15.43it/s]

 49%|████▉     | 102/208 [00:06<00:06, 15.51it/s]

 50%|█████     | 104/208 [00:06<00:06, 15.57it/s]

 51%|█████     | 106/208 [00:06<00:06, 15.32it/s]

 52%|█████▏    | 108/208 [00:07<00:06, 15.22it/s]

 53%|█████▎    | 110/208 [00:07<00:06, 15.25it/s]

 54%|█████▍    | 112/208 [00:07<00:06, 15.31it/s]

 55%|█████▍    | 114/208 [00:07<00:06, 15.18it/s]

 56%|█████▌    | 116/208 [00:07<00:06, 15.14it/s]

 57%|█████▋    | 118/208 [00:07<00:05, 15.15it/s]

 58%|█████▊    | 120/208 [00:07<00:05, 15.26it/s]

 59%|█████▊    | 122/208 [00:07<00:05, 15.42it/s]

 60%|█████▉    | 124/208 [00:08<00:05, 15.58it/s]

 61%|██████    | 126/208 [00:08<00:05, 15.48it/s]

 62%|██████▏   | 128/208 [00:08<00:05, 15.67it/s]

 62%|██████▎   | 130/208 [00:08<00:04, 15.61it/s]

 63%|██████▎   | 132/208 [00:08<00:04, 15.48it/s]

 64%|██████▍   | 134/208 [00:08<00:04, 15.52it/s]

 65%|██████▌   | 136/208 [00:08<00:04, 15.40it/s]

 66%|██████▋   | 138/208 [00:08<00:04, 15.47it/s]

 67%|██████▋   | 140/208 [00:09<00:04, 15.65it/s]

 68%|██████▊   | 142/208 [00:09<00:04, 15.62it/s]

 69%|██████▉   | 144/208 [00:09<00:04, 15.55it/s]

 70%|███████   | 146/208 [00:09<00:03, 15.71it/s]

 71%|███████   | 148/208 [00:09<00:03, 15.65it/s]

 72%|███████▏  | 150/208 [00:09<00:03, 15.64it/s]

 73%|███████▎  | 152/208 [00:09<00:03, 15.66it/s]

 74%|███████▍  | 154/208 [00:09<00:03, 15.69it/s]

 75%|███████▌  | 156/208 [00:10<00:03, 15.53it/s]

 76%|███████▌  | 158/208 [00:10<00:03, 15.50it/s]

 77%|███████▋  | 160/208 [00:10<00:03, 15.50it/s]

 78%|███████▊  | 162/208 [00:10<00:02, 15.52it/s]

 79%|███████▉  | 164/208 [00:10<00:02, 15.53it/s]

 80%|███████▉  | 166/208 [00:10<00:02, 15.57it/s]

 81%|████████  | 168/208 [00:10<00:02, 15.74it/s]

 82%|████████▏ | 170/208 [00:11<00:02, 15.59it/s]

 83%|████████▎ | 172/208 [00:11<00:02, 15.50it/s]

 84%|████████▎ | 174/208 [00:11<00:02, 15.56it/s]

 85%|████████▍ | 176/208 [00:11<00:02, 15.62it/s]

 86%|████████▌ | 178/208 [00:11<00:01, 15.40it/s]

 87%|████████▋ | 180/208 [00:11<00:01, 15.37it/s]

 88%|████████▊ | 182/208 [00:11<00:01, 15.45it/s]

 88%|████████▊ | 184/208 [00:11<00:01, 15.39it/s]

 89%|████████▉ | 186/208 [00:12<00:01, 15.26it/s]

 90%|█████████ | 188/208 [00:12<00:01, 15.30it/s]

 91%|█████████▏| 190/208 [00:12<00:01, 15.37it/s]

 92%|█████████▏| 192/208 [00:12<00:01, 15.34it/s]

 93%|█████████▎| 194/208 [00:12<00:00, 15.58it/s]

 94%|█████████▍| 196/208 [00:12<00:00, 15.48it/s]

 95%|█████████▌| 198/208 [00:12<00:00, 15.41it/s]

 96%|█████████▌| 200/208 [00:12<00:00, 15.51it/s]

 97%|█████████▋| 202/208 [00:13<00:00, 15.12it/s]

 98%|█████████▊| 204/208 [00:13<00:00, 14.83it/s]

 99%|█████████▉| 206/208 [00:13<00:00, 14.90it/s]

100%|██████████| 208/208 [00:13<00:00, 14.85it/s]
                                               

                                                 
{'eval_loss': 2.0531225204467773, 'eval_runtime': 13.5735, 'eval_samples_per_second': 30.648, 'eval_steps_per_second': 15.324, 'eval_entropy': 0.9273212139423077, 'eval_num_tokens': 21152.0, 'eval_mean_token_accuracy': 0.6695072605059698, 'epoch': 0.01}

100%|██████████| 10/10 [00:25<00:00,  1.10s/it]

100%|██████████| 208/208 [00:13<00:00, 14.85it/s]

                                                 
                                               
{'train_runtime': 27.5135, 'train_samples_per_second': 2.908, 'train_steps_per_second': 0.363, 'train_loss': 2.074407958984375, 'epoch': 0.01}

100%|██████████| 10/10 [00:27<00:00,  1.10s/it]
100%|██████████| 10/10 [00:27<00:00,  2.75s/it]
Saving final model to: ./checkpoints/final_model
Training metrics: {'val_loss': 2.0531225204467773, 'train_loss': 2.0744}
Number of training samples: 13857
Saved metrics.json to checkpoints/final_model/metrics.json
Uploading to vinoku89/aimo3-exp-1 (main branch)...

Processing Files (0 / 0)      : |          |  0.00B /  0.00B            

New Data Upload               : |          |  0.00B /  0.00B            


  ...adapter_model.safetensors:   2%|▏         |  343kB / 20.2MB            



  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            


  ...adapter_model.safetensors:   2%|▏         |  343kB / 20.2MB            



  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
Processing Files (1 / 2)      :  37%|███▋      | 11.8MB / 31.7MB,   ???B/s  


  ...adapter_model.safetensors:   2%|▏         |  343kB / 20.2MB            



  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            


  ...adapter_model.safetensors:   2%|▏         |  343kB / 20.2MB            



  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            


  ...adapter_model.safetensors:   7%|▋         | 1.45MB / 20.2MB            



  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
Processing Files (1 / 2)      :  41%|████      | 12.9MB / 31.7MB, 1.84MB/s  

New Data Upload               :   3%|▎         |  608kB / 19.4MB, 1.01MB/s  


  ...adapter_model.safetensors:   7%|▋         | 1.45MB / 20.2MB            



  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            


  ...adapter_model.safetensors:   7%|▋         | 1.45MB / 20.2MB            



  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            


  ...adapter_model.safetensors:   7%|▋         | 1.45MB / 20.2MB            



  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            


  ...adapter_model.safetensors:  16%|█▌        | 3.27MB / 20.2MB            



  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
Processing Files (1 / 2)      :  46%|████▋     | 14.7MB / 31.7MB, 2.09MB/s  

New Data Upload               :  13%|█▎        | 2.43MB / 19.4MB, 1.74MB/s  


  ...adapter_model.safetensors:  28%|██▊       | 5.71MB / 20.2MB            



  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
Processing Files (1 / 2)      :  54%|█████▍    | 17.1MB / 31.7MB, 3.35MB/s  

New Data Upload               :  25%|██▌       | 4.87MB / 19.4MB, 3.04MB/s  


  ...adapter_model.safetensors:  61%|██████▏   | 12.4MB / 20.2MB            



  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
Processing Files (1 / 2)      :  75%|███████▌  | 23.8MB / 31.7MB, 6.70MB/s  

New Data Upload               :  60%|█████▉    | 11.6MB / 19.4MB, 6.42MB/s  


  ...adapter_model.safetensors:  97%|█████████▋| 19.7MB / 20.2MB            



  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
Processing Files (1 / 2)      :  98%|█████████▊| 31.1MB / 31.7MB, 9.68MB/s  

New Data Upload               :  97%|█████████▋| 18.9MB / 19.4MB, 9.43MB/s  


  ...adapter_model.safetensors:  97%|█████████▋| 19.7MB / 20.2MB            



  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            


  ...adapter_model.safetensors:  97%|█████████▋| 19.7MB / 20.2MB            



  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            


  ...adapter_model.safetensors:  97%|█████████▋| 19.7MB / 20.2MB            



  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            


  ...adapter_model.safetensors:  97%|█████████▋| 19.7MB / 20.2MB            



  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            


  ...adapter_model.safetensors:  97%|█████████▋| 19.7MB / 20.2MB            



  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            


  ...adapter_model.safetensors: 100%|██████████| 20.2MB / 20.2MB            



  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
Processing Files (2 / 2)      : 100%|██████████| 31.7MB / 31.7MB, 6.22MB/s  

New Data Upload               : 100%|██████████| 19.4MB / 19.4MB, 6.06MB/s  


  ...adapter_model.safetensors: 100%|██████████| 20.2MB / 20.2MB            



  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            


  ...adapter_model.safetensors: 100%|██████████| 20.2MB / 20.2MB            



  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            


  ...adapter_model.safetensors: 100%|██████████| 20.2MB / 20.2MB            



  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
Processing Files (2 / 2)      : 100%|██████████| 31.7MB / 31.7MB, 5.53MB/s  

New Data Upload               : 100%|██████████| 19.4MB / 19.4MB, 5.39MB/s  

  ...adapter_model.safetensors: 100%|██████████| 20.2MB / 20.2MB            

  ...inal_model/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
Created tag: qwen0.6b-sft-e1-13.9k-loss2.05-20260112-003
Updated 'latest' tag
Kept existing 'best' tag (current: 2.0531 >= best: 2.0518)
Upload complete: https://huggingface.co/vinoku89/aimo3-exp-1
Uploaded logs/runtime_config_20260113_063547.yaml to vinoku89/aimo3-exp-1 (main branch)

============================================================
Model pushed successfully!
============================================================
  URL: https://huggingface.co/vinoku89/aimo3-exp-1
  Run Tag: qwen0.6b-sft-e1-13.9k-loss2.05-20260112-003
  Tags: qwen0.6b-sft-e1-13.9k-loss2.05-20260112-003, latest, best (if lowest loss)

  To load this specific run:
    from peft import PeftModel
    model = PeftModel.from_pretrained('base_model', 'vinoku89/aimo3-exp-1', revision='qwen0.6b-sft-e1-13.9k-loss2.05-20260112-003')

  To load the best run:
    model = PeftModel.from_pretrained('base_model', 'vinoku89/aimo3-exp-1', revision='best')

  To load the latest run:
    model = PeftModel.from_pretrained('base_model', 'vinoku89/aimo3-exp-1', revision='latest')
============================================================

wandb: updating run metadata
wandb: uploading output.log; uploading wandb-summary.json
wandb: uploading output.log
wandb: uploading config.yaml
wandb: uploading summary, console lines 54-87
wandb: 
wandb: Run history:
wandb:             eval/entropy ▁
wandb:                eval/loss ▁
wandb: eval/mean_token_accuracy ▁
wandb:          eval/num_tokens ▁
wandb:             eval/runtime ▁
wandb:  eval/samples_per_second ▁
wandb:    eval/steps_per_second ▁
wandb:            train/entropy ▁
wandb:              train/epoch ▁▁▁
wandb:        train/global_step ▁▁▁
wandb:                       +5 ...
wandb: 
wandb: Run summary:
wandb:             eval/entropy 0.92732
wandb:                eval/loss 2.05312
wandb: eval/mean_token_accuracy 0.66951
wandb:          eval/num_tokens 21152
wandb:             eval/runtime 13.5735
wandb:  eval/samples_per_second 30.648
wandb:    eval/steps_per_second 15.324
wandb:               total_flos 67838115053568.0
wandb:            train/entropy 0.94746
wandb:              train/epoch 0.00577
wandb:                      +10 ...
wandb: 
wandb: 🚀 View run qwen3-0.6b-sft-lora-20260113_063547-003 at: https://wandb.ai/vinokuteam/aimo3-exp-1-train/runs/0pkyx3cq
wandb: ⭐️ View project at: https://wandb.ai/vinokuteam/aimo3-exp-1-train
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20260112_223606-0pkyx3cq/logs
Training completed successfully!
[W112 22:37:40.785124394 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator())
[W112 22:37:42.435659668 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator())